From nobody Sun Dec 07 16:18:56 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dPVg85T9Mz6Kf5q for ; Sun, 07 Dec 2025 16:19:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-24.consmr.mail.gq1.yahoo.com (sonic303-24.consmr.mail.gq1.yahoo.com [98.137.64.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4dPVg82ZGgz3t0V for ; Sun, 07 Dec 2025 16:19:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=laR6aK1v; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765124349; bh=UldIEfVSCgTmmF2wHf3mCmLYkXGYtoc5tUHkVkB6rEA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=laR6aK1vsL9OLWSBtIEZL5buAwbPzMfCrh8l6s3uYmc9XkQzWsKpZaBxDxkwgd9881j3P0iqibHqNeM5NVn606xUnIYUuKSF61RmJ8aven0muNhEdNmSBgOsbbNTYidBRCTQ7ayBIy31g4Rl3+0RBqPkbNDzopQGeFBcdIbD621KPig24d4UEJIdyDpwMesIKIID1n6d+qMzCWPjKN7swNB/avTa5RQ/6bbKW1NJ+3qx6OfhPdVjdJx0Uqq3h10JJ66X6MiyYoanw4T9o0/S18qMhtHG/d4aFNmqH5ezHyERKPe2Qmgxh9OLYEiW5a3Ez2TP6lyQApvoaaRsierXog== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765124349; bh=sUMvZ9vwIY4x1ydWg7Bush7QHPlBTx4B7P7RPKpWF2M=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=eCRAvBmFun+ckrndMPrKAahXB3gFYiXw79OvpfXTPWM4vwZdIubH8tziBIyWkwPTtxk+yin8F9XXtQobfJH/i8V0085I9AHCFGEm2cfes0vGTerlapi0TQmwaBDZ6kIMqe6Bzosv8ID7IcDiXnZBVR7gc5uVr8gk5HPfA1rHeD6raWwnPZvP80PYQZwDScrxDM5c+Y0aAfOpTqdbwiogwu9zQlbH4lQMEKm5TZDr40aCdfnBAFRczFBr51fsUcivM39pUZPXMraB7YE6AddwGx3zzbXx4NOQGecvzbF4KzkbssZ3TLp3rvzQdNsRomTdYOdiLghtCyNypHi+Uhqi2A== X-YMail-OSG: Ks61sjAVM1kq3RdT1iAQ0CjXL3DKYSYvC3X7SDuIh7oOJybxgdCb_2R8pkwVS6M ZN1MobZi0kqJYeJlwkSql6daVGcJCUASWScIGz2S2.LenpEEvABX0asjlu46PaokAJD76_81VLZd AVSn5RVDZ.laW2qT7K7FWpXcD9Q7fxks63Jd7dazgW71u6CPfm7Ym2JoY9snL93sxh2YYUe8VWtz 4zk26noAVjBh92fGefRQx06hg3xedf9m5j8zChtOk6wmxI74psRcwFEPiEwEw_mXznn8hwdUKqB6 gXmly4SxiYfhgi8nkiQRc52prHiYGOeasDX0JpJNbQrBTiJ72UdATDnmA02YK33D3By_llzcR769 oe4E.uM39mg1PzqDkDWmZt_pwMqpf5s9FcI.wdCFI0ohCnKq3iMXSj1_czWeuu2ybMpZCko_0ers BPHCKHCK4Qea85gRNPl.3TF6jQMRMUOLFlWy6U0l7JmUjVpbBgYtsm4RCWe.D2L4nIRKD83Bu72M UkkxeB7X63F8fVDMz43dTGYKBeXGu9tADAGlbt1_1W7_y8nQKczoTMbpPouIpEMV807K5PH3HJ4o dfDikf5a259hV2ymAGyEN0TfELfT4DzaA.eKPFMVmNu3ZYi2Aw6r9iZk3_j6idGPIFuXkydQruf0 SoAvWNOAjCYLaDZJNIwadamnFH.lbDeuw9BhgsNr56yxag7W0gBrU60k9xowCanuf8wTwSzxIcU0 GN7p_sHnIKixXXD.iXK5_OTwW4LvkWW3VpEVAqgrdSvET2g9_HTR2IciDzRaYDdCOWfYTiN_aHjc k_CmC9noeuQ0gNzsFqLcMmWiX6Cv7RRa55C8QU22jqHnYIGPsObggD7UKc4hGCRTbWuoa2uw9fgy AGwd_xvMKkgbSTzpnP8LCZMkDooIXsahOYYoxdf_7LfSUd_nyHZbLOpp6mXlWODvZqFwEwv1pTrC nwBOYFMsRAbKCkf1YU1_D0nYxKnxudHvJ_QWHpW9xjutZZxedlQx8V4owdEaZ9ka7GGkcluZTrDP tzPyCajKU5K8BBb_1hsvY5oLzSfBxKD3N.OSaG31.Ja.dL61jTwh7UEoQ9x.R9GHR8_VZFBJgDgh fGTk95yQrh8YFHF4cVjGilf4a9af5jzlDeNuG46geEE_Vp7EeokkYyoih_s0qFJJzaWPIMM0QbzD BnN1Nqe8mTHEN4EtAhlm2ktz5v1ajiUjdJQkJs4SBg2_F9u7DP.gnQIn2_eLppeo2rhz8jyObbVU 2yVktaNGwQBBOlCYpdQYCQhJM2HNZ3.cWP7JxZS5hAXdZpRSyPPaZHIWIiA0V3HnzY8xevV40RzN ICkRBzt1vEld8xMMHExCOHUverwhJyW9TlYQqjH7bZVkfwI5tmaD19LyPJGYrPiOmc.IMVFb0BTo kHuRuDHKKlbZp4.ihU5TXf1ltVbXI9ShfGirMU67_z2IwV1Q5tP8w0hFQO5ngBoP_sBN8UP7A7qY 5Iw0i35vsiFiPdrmqBxEF2qu9VHIewGwzLDoydCISxe4Sdramu0QExTi3edTdAywqxX9keZdMmQ4 YHnv1BSCn_vusCLXVPfcFG1nYJC99RUOxcJ_ICm9ObBr2nB6KCFbtAIySDAns0xJdkw9Z5DN9yJd puGQyaFwAtzo1sjvq8Wmvf.fu04leEYQN.sA4XwCrbyfiJ1Hi8nk_3vsQVSRyPkaGJsj8mhbzMVP R.oK3X8.f.y8aljHYZ3y4pw_WVGD9W.h7tKLinV7xp6y.6Wufwc7ICEMPwKPMAlAP1srw6rOWfv6 L5mQ4PiljxPJR0SGdd2hjq2hg4bg_cdjQe2QM.f0YD4YAMCha4yFoorpyGuji2R5BVBvQEiWm0ms lhp2SIFpqB77VEm5pCtjJpNFI65lDtBWp3yvKUEvNIXzdudpCplH0Ic4VDMhORnP4ZWGuFeB6zUQ 1kKDe4MuDJSdIBgGFq9H.QPsGxLcI7v6XK7MQVe.sJpypZe97J_7Djjb1_dswBBdus9x0kcxa.Tt mOoc6lL5Q7OlayfLv90FSupZuupukiORi436w1wTmOxNu0CmawybNzA7asmYVBogK8oefgQrAa99 eXaJP7WMjzWNMr664z91NvSI6Nm2xWHeWYniddHjkRq6tHNTXn3SvddJLsHZfTmoNlWIfiXgmp.b jOH76SyjitoGgRs01JVlhsSfFli3DtGZm.hC0DaCkMzerZXhaiUm3A7ye1ZpQxEQGAeu8oEuXbII WB10BxzwhbUUA9mlAVZG3RiBpV3zLxeTSahhfKNh9uxHVPiRX2tiyaISdJwSxYmLlcnZf6bp5x5Q Edpw- X-Sonic-MF: X-Sonic-ID: f757bbae-fcee-422b-8efa-a7e7b0b0034b Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sun, 7 Dec 2025 16:19:09 +0000 Received: by hermes--production-gq1-54bf57fc64-j2cpf (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 75c72eb846e41cb33511c4686f35e6f1; Sun, 07 Dec 2025 16:19:07 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: performance regressions in 15.0 [The Microsoft Dev Kit 2023 buildworld took about 6 minutes less time for jemalloc 5.3.0, not more, for non-debug contexts] From: Mark Millard In-Reply-To: <19A848A6-0042-4873-B70D-AD6805225B92@yahoo.com> Date: Sun, 7 Dec 2025 08:18:56 -0800 Cc: FreeBSD Current , FreeBSD-STABLE Mailing List , Konstantin Belousov , mjguzik@gmail.com Content-Transfer-Encoding: quoted-printable Message-Id: <902C948B-0A4C-48E1-8C6C-1BC7A15209D7@yahoo.com> References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> <19A848A6-0042-4873-B70D-AD6805225B92@yahoo.com> To: Warner Losh X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.205:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.205:from]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4dPVg82ZGgz3t0V On Dec 6, 2025, at 19:03, Mark Millard wrote: > On Dec 6, 2025, at 14:25, Warner Losh wrote: >=20 >> On Sat, Dec 6, 2025, 3:06=E2=80=AFPM Mark Millard = wrote: >>=20 >>> On Dec 6, 2025, at 06:14, Mark Millard wrote: >>>=20 >>>> Mateusz Guzik wrote on >>>> Date: Sat, 06 Dec 2025 10:50:08 UTC : >>>>=20 >>>>> I got pointed at phoronix: = https://www.phoronix.com/review/freebsd-15-amd-epyc >>>>>=20 >>>>> While I don't treat their results as gospel, a FreeBSD vs FreeBSD = test >>>>> showing a slowdown most definitely warrants a closer look. >>>>>=20 >>>>> They observed slowdowns when using iperf over localhost and when = compiling llvm. >>>>>=20 >>>>> I can confirm both problems and more. >>>>>=20 >>>>> I found the profiling tooling for userspace to be broken again so = I >>>>> did not investigate much and I'm not going to dig into it further. >>>>>=20 >>>>> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems >>>>> running as 8 core vms under kvm. >>>>> . . . >>>>=20 >>>>=20 >>>>=20 >>>> Both of the below are from ampere3 (aarch64) instead, its >>>> 2 most recent "bulk -a" runs that completed, elapsed times >>>> shown for qt6-webengine-6.9.3 builds: >>>>=20 >>>> 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46 >>>> 135arm64-default qt6-webengine-6.9.3 38:43:36 >>>>=20 >>>> For reference: >>>>=20 >>>> Host OSVERSION: 1600000 >>>> Jail OSVERSION: 1500068 >>>>=20 >>>> vs. >>>>=20 >>>> Host OSVERSION: 1600000 >>>> Jail OSVERSION: 1305000 >>>>=20 >>>> The difference for the above is in the Jail's world builds, >>>> not in the boot's (kernel+world) builds. >>>>=20 >>>>=20 >>>> For reference: >>>>=20 >>>>=20 >>>> = https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D150releng-a= rm64-quarterly&build=3D88084f9163ae >>>>=20 >>>> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov = 30 05:40:02 -00 2025 >>>> build time: 2D:05:33:52 >>>>=20 >>>>=20 >>>> = https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D135arm64-de= fault&build=3Df5384fe59be6 >>>>=20 >>>> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov = 22 15:33:34 -00 2025 >>>> build time: 1D:14:43:41 >>>=20 >>>=20 >>> Expanding the notes to before and after jemalloc 5.3.0 >>> was merged to main: beefy18 was the main-amd64 builder >>> before and somewhat after the jemalloc 5.3.0 merge from >>> vendor branch: >>>=20 >>> Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 = port-packages, start 05 Aug 2025 01:10:59 GMT >>> ( jemalloc 5.3.0 merge from = vendor branch: 15 Aug 2025) >>> After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 = port-packages, start 19 Aug 2025 01:30:33 GMT >>>=20 >>> (The log files are long gone for port-packages built.) >>>=20 >>> main-15 used a debug jail world but 15.0-RELEASE does not. >>>=20 >>> I'm not aware of such a port-package builder context for a >>> non-debug jail world before and after a jemalloc 5.3.0 merge. >>>=20 >> A few months before I landed the jemalloc patches, i did 4 or 5 from = dirt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to = see maybe a diff with the small sample size, but not enough for ministat = to trigger at 95%. I didn't recall keeping the data for this and can't = find it now. And I'm not even sure, in hindsight, I ran a good = experiment. It might be related, or not, but it would be easy enough for = someone to setup a two jails: one just before and one just after. Build = from scratch the world (same hash) on both. That would test it since = you'd be holding all other variables constant. >>=20 >> When we imported the tip of FreeBSD main at work, we didn't get a cpu = change trigger from our tests that I recall... >=20 >=20 > The range of commits look like: >=20 > =E2=80=A2 git: 9a7c512a6149 - main - ucred groups: restore a useful = comment Eric van Gyzen=20 > =E2=80=A2 git: bf6039f09a30 - main - jemalloc: Unthin = contrib/jemalloc Warner Losh=20 > =E2=80=A2 git: a0dfba697132 - main - jemalloc: Update = jemalloc.xml.in per FreeBSD-diffs Warner Losh=20 > =E2=80=A2 git: 718b13ba6c5d - main - jemalloc: Add FreeBSD's = updates to jemalloc_preamble.h.in Warner Losh=20 > =E2=80=A2 git: 6371645df7b0 - main - jemalloc: Add = JEMALLOC_PRIVATE_NAMESPACE for the libc namespace Warner Losh=20 > =E2=80=A2 git: da260ab23f26 - main - jemalloc: Only replace = _pthread_mutex_init_calloc_cb in private namespace Warner Losh=20 > =E2=80=A2 git: c43cad871720 - main - jemalloc: Merge from jemalloc = 5.3.0 vendor branch Warner Losh=20 > =E2=80=A2 git: 69af14a57c9e - main - jemalloc: Note update in = UPDATING and RELNOTES Warner Losh=20 >=20 > I've started a build of a non-debug 9a7c512a6149 world > to later create a chroot to do a test buildworld in. >=20 > I'll also do a build of a non-debug 69af14a57c9e world > to later create the other chroot to do a test > buildworld in. >=20 > non-debug means my use of: >=20 > WITH_MALLOC_PRODUCTION=3D > WITHOUT_ASSERT_DEBUG=3D > WITHOUT_PTHREADS_ASSERTIONS=3D > WITHOUT_LLVM_ASSERTIONS=3D >=20 > I've used "env WITH_META_MODE=3D" as it cuts down on the > volume and frequency of scrolling output. I'll do the > same later. >=20 > If there is anything you want controlled in a different > way, let me know. >=20 > The Windows Dev Kit 2023 is booted (world and kernel) > with: >=20 > # uname -apKU > FreeBSD aarch64-main-pbase 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n281922-4872b48b175c GENERIC-NODEBUG arm64 aarch64 1600004 1600004 >=20 > which is from an official pkgbase distribution. So the > boot-world is a debug world but the boot-kernel is not. >=20 > The Windows Dev Kit 2023 will take some time for such > -j8 builds and I may end up sleeping in the middle of > the sequence someplace. So it may be a while before > I've any comparison/contrast data to report. >=20 Summary for jemalloc for before vs. at 5.3.0 for *non-debug* contexts doing the buildworld : before 5.3.0: 9754 seconds (about 2.7 hrs) with 5.3.0: 9384 seconds (about 2.6 hrs) So: somewhat less time with 5.3.0 but nearly the same. It does not clarify what is going on for building qt6-webengine-6.9.3 --other than suggesting including looking for alternative sources of issues. Also, it seems that the Mateusz Guzik microbenchmark results do not scale for the specific type of activity for the specific type of platform. Details . . . My two source trees for creating the 2 chroots are: # ~/fbsd-based-on-what-commit.sh -C /usr/src-jemalloc-5p3p0-before/ 9a7c512a6149 (HEAD) ucred groups: restore a useful comment Author: Eric van Gyzen Commit: Eric van Gyzen CommitDate: 2025-08-15 13:29:18 +0000 # ~/fbsd-based-on-what-commit.sh -C /usr/src-jemalloc-5p3p0-at/ 69af14a57c9e (HEAD) jemalloc: Note update in UPDATING and RELNOTES Author: Warner Losh Commit: Warner Losh CommitDate: 2025-08-15 21:57:59 +0000 Both have src.conf : WITH_MALLOC_PRODUCTION=3D WITHOUT_ASSERT_DEBUG=3D WITHOUT_PTHREADS_ASSERTIONS=3D WITHOUT_LLVM_ASSERTIONS=3D since that works for the main 16 in use. (But /etc/src.conf needs to be used in the chroot's.) Having main 16 build /usr/src-jemalloc-5p3p0-before/ : World build completed on Sat Dec 6 21:24:09 PST 2025 World built in 11817 seconds, ncpu: 8, make -j8 Having main 16 build /usr/src-jemalloc-5p3p0-at/ : World build completed on Sun Dec 7 00:46:25 PST 2025 World built in 11996 seconds, ncpu: 8, make -j8 (So: not much difference, as expected.) I then did installation and setup of the two chroot directory trees, creating: # ls -dC1 /usr/obj/DESTDIRs/jemalloc-5p3p0-*/ /usr/obj/DESTDIRs/jemalloc-5p3p0-at/ /usr/obj/DESTDIRs/jemalloc-5p3p0-before/ Both got /etc/src.conf : WITH_MALLOC_PRODUCTION=3D WITHOUT_ASSERT_DEBUG=3D WITHOUT_PTHREADS_ASSERTIONS=3D WITHOUT_LLVM_ASSERTIONS=3D I then created and, via rsync, populated each of: # ls -dC1 /usr/obj/DESTDIRs/jemalloc-5p3p0-*/usr/src-jemalloc-5p3p0-*/ /usr/obj/DESTDIRs/jemalloc-5p3p0-at/usr/src-jemalloc-5p3p0-at/ /usr/obj/DESTDIRs/jemalloc-5p3p0-before/usr/src-jemalloc-5p3p0-at/ I then did: # chroot /usr/obj/DESTDIRs/jemalloc-5p3p0-before/ # cd /usr/src-jemalloc-5p3p0-at/ # env WITH_META_MODE make -j8 buildworld It resulted in: World build completed on Sun Dec 7 12:25:45 UTC 2025 World built in 9754 seconds, ncpu: 8, make -j8 (So definitely less time consuming than main 16's build of the src-jemalloc-5p3p0-at/ source, as expected.) After exiting that chroot, I then did: # chroot /usr/obj/DESTDIRs/jemalloc-5p3p0-at/ # cd /usr/src-jemalloc-5p3p0-at/ # env WITH_META_MODE make -j8 buildworld It resulted in: World build completed on Sun Dec 7 15:36:41 UTC 2025 World built in 9384 seconds, ncpu: 8, make -j8 So, less time than before jemalloc 5.3.0 . Note: the Microsoft Windows Dev Kit 2023 was using a 1.4 TByte Optane U.2 via a USB3 adapter, of all things. =3D=3D=3D Mark Millard marklmi at yahoo.com