Re: Long time outdated jemalloc
- In reply to: Warner Losh : "Re: Long time outdated jemalloc"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 03 Feb 2025 13:32:04 UTC
On 08/12/2024 18:41, Warner Losh wrote: > Great! I'll take a look at that, as well as do the merge the typical > way (which only takes a few minutes now that I've bootstrapped > things). I'll compare the two to see what diffs there might be (to act > as a cross check for both methods). I'll then build a copy of the > Netflix firmware with the change and put it on a couple machines and > see if they can handle the load and if there are any performance > regressions. I don't expect any, since malloc typically doesn't appear > in the flame graphs as "visible", but you never know. > > So, once that's done, and I expect it to be done this week, I'll push > it into main with both the proper vendor branch merge commits as well > as an acknowledgement for this pull request and your work to move it > forward (I'm just verifying the typical process will produce the same > results and the typical process doesn't take a long time, etc). > > Warner > > On Sun, Dec 8, 2024 at 9:03 AM Minsoo Choo <minsoochoo0122@proton.me> > wrote: > > I resolved merge conflict by rebasing main. What's next? > > https://github.com/freebsd/freebsd-src/pull/1337 > On Sunday, December 1st, 2024 at 2:05 AM, Warner Losh > <imp@bsdimp.com> wrote: >> (sorry to follow up to my own email and topposting) >> >> I got the vendor branch bootstrapped: I created vendor/jemalloc >> and tagged vendor/jemalloc/5.2.1, >> and created a merge commit from that branch to main. I had to >> tweak the >> FREEBSD-Xlist a little.I've not updated the other two FREEBSD-* >> files, but the steps were >> documented in the commit messages (the vendor branch one is >> especially long and >> likely should migrate into the developer handbook). >> FREEBSD-update is basically >> a shell script to do the same thing that git subtree merge does, >> though I'm sure some >> tweaks could help the next time (if there is a next time, >> jemalloc upstream seems to >> have slowed way down of late). >> >> Next up,I'll create 5.3.0 import on the vendor branch, do the >> merge and start testing (it will be >> Minsoo's pull request, rebased, with any conflicts resolved and >> merge commit recorded). >> But that will have to be tomorrow or more likely during the work >> week. I'm too tired tonight >> to get it right at the moment. >> >> And a special thanks to emaste for giving bz the right recipe for >> doing the subtree merge >> w/o git subtree for his work on the linux wifi drivers in the tree. >> >> Warner >> >> On Sat, Nov 30, 2024 at 9:38 PM Warner Losh <imp@bsdimp.com> wrote: >> >> Yea, I need to get a copy of jemalloc 5.3.0 and 5.2.1 to try >> to 'bootstrap' the vendor branch. >> Then I need to bootstrap it... >> >> I just did the same with edk2 (which had a vendor branch, but >> hadn't been updated since svn times). >> However, jemalloc doesn't have a vendor branch yet, so I'll >> have to create that, but I'll start with the >> current version rather than doing full history... So I'll >> start there. >> I also just did awk and lua, so once I have things >> bootstrapped, I'll be able to add 5.3.0 and then layer >> Minsoo's on >> top of that and then start testing it somehow. >> >> Malloc makes me nervous to touch, honestly, but I'll give it >> a go and test boot on my system and >> maybe see if we can survive a workload at work w/o >> regressions... But I can't do a full test with lots >> of machines until after the first of the year (though I can >> do a couple for a few days before then). >> >> So my next step is to bootstrap the vendor branch... I'll >> give that a try tonight. >> >> Warner >> >> On Sat, Nov 30, 2024 at 8:26 PM Minsoo Choo >> <minsoochoo0122@proton.me> wrote: >> >> I have already submitted PR on github >> (https://github.com/freebsd/freebsd-src/pull/1337) and >> phabricator (https://reviews.freebsd.org/D41421). I don't >> have access (commit bit) to freebsd git repo, so there is >> nothing I can do at this point since vendor import and >> landing patches requires commit bit. >> On Saturday, November 30th, 2024 at 1:42 PM, cglogic >> <cglogic@protonmail.com> wrote: >>> I see, it happens. >>> Maybe another committer will volunteer to do the update. >>> I hope it will make its way into 15.0 release. >>> >>> Thanks. >>> On Friday, November 29th, 2024 at 9:38 PM, Warner Losh >>> <imp@bsdimp.com> wrote: >>>> I've been swamped. we need to bootstrap the vendor >>>> branch, and the way prior updates were done >>>> isn't so great. >>>> >>>> Warner >>>> >>>> On Mon, Nov 25, 2024 at 2:21 AM cglogic >>>> <cglogic@protonmail.com> wrote: >>>> >>>> Hello guys, >>>> >>>> How the update of jemalloc is going? It's November now. >>>> >>>> Thanks. >>>> On Monday, July 22nd, 2024 at 7:02 PM, Minsoo Choo >>>> <minsoochoo0122@proton.me> wrote: >>>>> First, sorry for late response. >>>>> >>>>> cglogic, thank you for bringing up this issue >>>>> again since I nearly forgot that this issue was >>>>> still open. >>>>> >>>>> Warner, as I can't access to my FreeBSD instance >>>>> until the end of August, but I can still edit and >>>>> push the code through my Arm Mac. This means that >>>>> I can't test the updated code on my machine, but I >>>>> can join the review process and listen to change >>>>> proposals. >>>>> >>>>> I'll open a Github PR in a few hours. (The >>>>> phabricator review will stay opened just in case) >>>>> On Monday, July 22nd, 2024 at 5:08 AM, Warner Losh >>>>> <imp@bsdimp.com> wrote: >>>>>> >>>>>> >>>>>> On Sun, Jul 21, 2024 at 2:03 PM cglogic >>>>>> <cglogic@protonmail.com> wrote: >>>>>> >>>>>> >>>>>> On Sunday, July 21st, 2024 at 6:54 AM, Warner >>>>>> Losh <imp@bsdimp.com> wrote: >>>>>>> >>>>>>> >>>>>>> On Sat, Jul 20, 2024 at 1:59 AM cglogic >>>>>>> <cglogic@protonmail.com> wrote: >>>>>>> >>>>>>> Hello FreeBSD community, >>>>>>> >>>>>>> After Jason Evans stepped aside from >>>>>>> maintaining jemalloc in FreeBSD, it's >>>>>>> not updating in time anymore. >>>>>>> Version 5.3.0 was released May 6, 2022 >>>>>>> and FreeBSD still not imported it into >>>>>>> the tree. >>>>>>> >>>>>>> There is a pending review >>>>>>> https://reviews.freebsd.org/D41421 from >>>>>>> Aug 11, 2023. >>>>>>> I'm successfully running FreeBSD/amd64 >>>>>>> system with D41421 applied for 8 months, >>>>>>> as well as many other people. >>>>>>> >>>>>>> Can it be reviewed and committed to CURRENT? >>>>>>> Or, if there is no committers willing to >>>>>>> do it, can commit bit be given to >>>>>>> submitter or another person willing to >>>>>>> do this? >>>>>>> >>>>>>> It's very disappointing when users spend >>>>>>> their time to fill such gaps and their >>>>>>> efforts just ignored by the developers. >>>>>>> Every year FreeBSD Community Survey >>>>>>> asking about user experience in >>>>>>> contributing to FreeBSD. >>>>>>> Here you can see an example of such >>>>>>> contributing. >>>>>>> >>>>>>> >>>>>>> First, thank you for being persistent and >>>>>>> continuing to bring it up. It's important to >>>>>>> do that to make sure this (and your many >>>>>>> other) contribution doesn't fall on the floor. >>>>>>> >>>>>>> And to be fair, we're only 3 months since >>>>>>> the last update. Still, quite a bit longer >>>>>>> than you should have to wait, but not nearly >>>>>>> the year the original date suggests. >>>>>>> >>>>>>> And this is a perfect storm of "how the >>>>>>> project is bad at accepting contributions": >>>>>>> (1) The original submission was close to the >>>>>>> 14 branch creation time. This meant that we >>>>>>> weren't well prepared to look at it since it >>>>>>> is such an invasive change (at least on its >>>>>>> surface). It also slowed the initial response... >>>>>>> (2) There was a number of back and forth >>>>>>> requests for changes, which took time to >>>>>>> sort out... >>>>>>> (3) The size of this is huge, well beyond >>>>>>> the capacity of Phabricator to review >>>>>>> accurately... >>>>>>> (4) It's a vendor import. That means we >>>>>>> can't just drop the Phabricator review into >>>>>>> the tree... >>>>>>> (5) It's phabricator: this is a great tool >>>>>>> for developers, but we have a terrible track >>>>>>> record of using it for intake from new >>>>>>> contributors. We don't have any oversight at >>>>>>> all over this tool, at there's at best tepid >>>>>>> and luke warm attempts to look for drop balls. >>>>>>> >>>>>>> All of these things are a terrible >>>>>>> experience. I can only apologize. These >>>>>>> days, we might steer this towards github, >>>>>>> but the 'vendor import' means you really >>>>>>> need someone on the inside, or you need to >>>>>>> be on the inside to make that work. >>>>>>> >>>>>>> So, how to move forward? Well, I'd like to >>>>>>> propose the following: >>>>>>> (1) submit all the other Phabricator reviews >>>>>>> you have open (they are mostly good, or >>>>>>> close to good) to github. Github is being >>>>>>> actively managed and will make it faster to >>>>>>> get things it. It's a much better tool for >>>>>>> new contributors (and even frequent >>>>>>> contributors of smallish things). >>>>>>> (2) I should do an vendor import of 5.3.0 >>>>>>> from github, and do the merge to a branch >>>>>>> and push that to github. You can then layer >>>>>>> on your changes and those can be reviewed >>>>>>> more closely as a pull request against the >>>>>>> branch I push. I suspect that most of the >>>>>>> issues are sorted out already >>>>>>> (3) I'll land it via that route... >>>>>>> >>>>>>> And, if the sum of the other pull requests >>>>>>> and this are good (and I suspect they will >>>>>>> be), then we can talk about commit bits and >>>>>>> such. >>>>>>> >>>>>>> It's experiences like this which is why I'm >>>>>>> trying to stand up github pull requests as a >>>>>>> reliable way to get things and and the best >>>>>>> place to send people... >>>>>>> >>>>>>> Thanks again for persisting, and also for >>>>>>> expressing this criticism that we >>>>>>> (hopefully) can use to make it better. >>>>>>> >>>>>>> Warner >>>>>> >>>>>> Hello. >>>>>> >>>>>> I'm not the author of D41421. Just applied >>>>>> the patch to test it 8 months ago. And >>>>>> recently discovered that it's still not >>>>>> committed. >>>>>> I can't copy your message to Phabricator >>>>>> because don't have an account. Please, if you >>>>>> have time, help the author in D41421. >>>>>> >>>>>> >>>>>> Ah yes. I've been in touch with the author for >>>>>> other things, and somehow thought it was you.... >>>>>> I'll reach out to him via other means... >>>>>> >>>>>> Warner >>>>> Is this still in progress? >>>>> I remember that we had an issue with varnish on >>>>> ubuntu where varnish would take way more memory >>>>> than configured, with a cache of 512MB, we did >>>>> consume almost 12GB of memory. The internet seems >>>>> to blame this on the jemalloc version comming with >>>>> ubuntu. I had the change to let the traffic go by >>>>> a FreeBSD machine with varnish installed, i saw >>>>> the same thing happening on FreeBSD also, i did >>>>> not have the time to investigate, but now that i >>>>> read this thread, it seems FreeBSD also uses 5.2.1 >>>>> jemalloc. The same version Ubuntu had. >>>>> >>>>> This update to 5.3.0 fixed it on ubuntu, so i gues >>>>> this would fix the varnish problem on FreeBSD also. >>>>> >>>>> regards >>>>> Johan >>>> >>> >> >