From nobody Wed Dec 10 04:10:53 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dR2Ml5BGcz6JrWC for ; Wed, 10 Dec 2025 04:11:15 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4dR2Ml0T0Kz3fvN for ; Wed, 10 Dec 2025 04:11:14 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Li9Ekp7v; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765339868; bh=hhCwMboHYvPMsnv/fbUuCya0BD/p2OZL7YpJNMIN1to=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Li9Ekp7vsIj+aSbGxcsWx9dW4OqxTdwBseQPY8gOx0m80UHJwdMdeH7GkJed20ywKZSDpnI45e44wyeQ5aBd0s+cxuI2SteSYo11zyN/pjqWUbl7lgPRp3FYt/20SYHqRM2EOMJAbyV+cyzQ3MUC36u75b6NY9YYrBXvb28B8E0+YYHQ7Qmh7KB4HMYXqUS9LxtlqIpw5Mg5pc7bFT27TLJs8X7Tfkl8aR1zgDZzwqazcH+dqzaEE4l5GH9jZ1BzGOqZkDlHrOIb7fMsxktRDFoImd/vpRXG9litm4hXUfftCt30uNkAgZ71svtx3wzaU8luiQskUV3xTfLorI/Q1w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765339868; bh=417hp/pH3vUwPpHr1KiYeS1SBu+eQvmEjMbOeyRoYeS=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=BDzGHrQuiSMuNHY/D7BXUgFwHRhSdnrJ6KsKhW6o6oFAvpIGwfLFWWdE5zgXzH1XY9oijTZVmJTkbkM49+77N9AkHjnclGS3G8g1jUKxvNGP7MW6/W4iOn9eD3ZjPTXFosuvlnNmjUBZ30ort9zKULmqcfu+o1Zi/mae6kLOpVY9SZXVVBf3y0d3BypTF68LjqqpnB1i4OZrWK7VPvgdq0HxwpAOWwE1N6hRiF/llFZiQyEl225aMcQ7MHXQAe8KSE/zilAlrSmpwAk1jQxv/8shxfIQlViwhshu4+hKP7Fr/f+KYpnZbaoxsmCOlAWJTO6cC0C72qck3w/Rvuz0bg== X-YMail-OSG: rqkcJRkVM1mPR_OMIyR09g6mLGswuPx.tvJwnKFkxW_.gf78c6uEJWaU36ZfQnI g2ZvJyaiFUkvMtJnVeI7NkXTuy2NmSda39972mr2KWEm.RklH5niK.17m48BOWSKk8vKveYDkiRT vHjXWEMuSms.Fq2UcX3HECCmA3XA1TzhSK3hb5kn7yuNHpVniYOhI._0y7uVXw1kb2kK7TxJBEtS JTz3dAeP9goE_iWSecwb3W6YunT_ofQH2xbWyQKRLSlT1gL1jWHsFMrfTY4faUdfegPmdHKUQg03 eWo2TaSrdlilnKjWa0QyXHNexb1U3PbTRmBbH17oqaeRV2tktWJ4VL8KrQmdwmqCaS9NVKCdaenY mf580R4uyHt9.wYXJopr17WcfRJhYURRz_pXu3I4h1qYg2kJbSyhGs_Lvb0McHRcNgi3JpgV5614 jpk0ASGI_vDEEPUzKJvZyaFjuuzUCtaoQSkcDglK9yh0w3iBD9JaogIFaqd33Qkyx7VjE85WepZL JougMP1FYmr96P9Y6cztCBZ4u5fVeDF_pneq0.y196DTs6PrAQx0eXdBC6CvTrN4CVk2Jd1H7ZUk kYvQ2Bs17Myi53zOfYPQqGaOmBOp8TTi8nruwfoulk062zAyUTCqKfxbzhufiZkkiPMTJA.Yis9q duMdTQbrPbETALcq6DhIHwn5jHLRIaYSZomrlH1EXcpjJZ6FSmvuG8gs4f4aHFMi6CmsFIPcexS3 tj0PWiU2Z.d.uhhBKZoKWcQUQ4nZHL.t78fBVeVzh0b2dNnPlT73s2bIwsdsOAdej3dw7ZH9KPmz 2G9NI0QiNmL5xW..ZT2n37QRZgIZllUyfcNeUAZ26pFSC_ghBZ0L1fwzG5r.LGf.Egi9BYAeZ86S K3BElSK1LZQDY2SBumVyE43IYuiWdCfIGVg0hPu9UOGnv1KT9PrzcnJkKscuouatidkltInY0szc y.E0aW26P3JDqSjOM2Hq9AsYbDq5jPuDOFLLJbux6RUzmf2AgXanOgjkFr63qlVtSJfh7uha0baV WRKFh3M68drJ0AUeZ0jnrYkVKqxU5KKWQvxEzXDKl37rrOh7mk5__PAA1iczsYjOibiuDMmkDoj2 lKagA1pDGO0uY0QkPESiXo9Dy1kfQmmbQCaEdIiTdE6gkevrIamMWEuCrwrb8dNbT6V0R31ESzyS zuN8M.Cg_awwHLH7EhHkBIY5UvbMnqux8gil6oegPhcrbn1_72G3w.X4AA.vTNBBOl_V1K51EzrL jpuw6kMFc_NeW.RpSXL26kJNPaM3Qv2ZfZaVQd75md3dawESZriXKvZFJ.U4V0PgDedrNxA3ebuk CV7fqkDhUk6ITzWCxPnsZ1Y_JrOT5GNsF4Ozp39jg0nxurO6sz9hfxYyK3TQfDWNBM6UpGRBsv5i fRNq37ud5B96eSOXYHuD3mAfC1rLg4Zx4LvvhTLrvzsTbweOG8kCP5EE0FDdmuF67ztdkfcpUCzX gyai4XJMrny9EFeIpy0F6fT__iqGrrO1uwoiHxDOE7cl2rEyvZQt8oFHScbLFjRyXyoR3HU7IPeD 8INByUCqVBJmagUvVTORwBeMMKmVaQqnGSdnXnrsGTvQIaapLX8KHv6tvjFDW_mKVrW1QqcXhH7P xjceU.xJ6gwCyiNE2SKm5mQmdy5xrHVm1T_7_JTd2pRDzvylct6fV4keOA0MHtNT55haSBcBG4m8 1gC1.T4UD69scS5Gmr_oPYCktWWodGTT2tvUIyh17V8lZ24C1IUcb2p.prl7FTr4BAJU122l7QWT sfwNqA.VsUGoXPCxtdXCDSrOSj7KCh47UZxzqNS1udTwhGLP3JnIHIufBo6AjVdnMEig0Q6xFWBy C6iVuj6OTJTAkX1Kcw00TFwtwiziX75jQfYlh0q1P6t6KuOmlpnak33l2R4iHd4g8fYkqqVI6zCP Xi1wCY3p6WXJ4xC.NhgfMGM6OtBrUMzi_Vga5vH8NEJtCg8RtPoiYXbFa7A5_KwA7JXPgXY8nfXa 7zUwvwwyxvAn9fa0fnWJ1XzEb_WqTQezGKVCInKzu06dYRSHvVLMK.Ws.U4IAPAwx9Fy6CXyiggR mwLaaDo51dvZ3qeZxllroKsZAWvIEe.1lZIh2fiDW3x5xqGGGBZ3mhBmlU.G7QgHHqu_xUZWKIyd IAeiJ6ikCQMkWvQSrHn60azbarQjJ20r2cZ7CxHKuUmAJpjYVTOeNQf.jNL7KZdmdwQYoURlK9xl HfLQLToqyzIOcZtJdhbFRndjJnwx8lJMgzi2ggWlYsBYz6HYKkfbm89Jcf9LIOI62FwRi1m4e4vi YXKf4M6IZ8zLvdnqcmpMPITWplmi_q7Q- X-Sonic-MF: X-Sonic-ID: aef9e9e6-71d6-40c7-a210-51e3b641b7d5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Wed, 10 Dec 2025 04:11:08 +0000 Received: by hermes--production-gq1-54bf57fc64-tvxhk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 08073f65c57fc7d952d38da05dee0323; Wed, 10 Dec 2025 04:11:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: performance regressions in 15.0 [14.3-STABLE performs much like 15.0-RELEASE for my aarch64 devel/cmake-core build tests, not like 14.3-RELEASE] From: Mark Millard In-Reply-To: <927C3BBE-BA9E-45A4-A43E-6416CF388FF5@yahoo.com> Date: Tue, 9 Dec 2025 20:10:53 -0800 Cc: Rozhuk Ivan Content-Transfer-Encoding: quoted-printable Message-Id: <577319E0-2051-436A-B9C0-6C1A2DE90618@yahoo.com> References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> <19A848A6-0042-4873-B70D-AD6805225B92@yahoo.com> <902C948B-0A4C-48E1-8C6C-1BC7A15209D7@yahoo.com> <33F5F7DE-8DFA-47E2-A890-E07564825D05@yahoo.com> <20251209172255.6d1a7f4c@rimwks.local> <927C3BBE-BA9E-45A4-A43E-6416CF388FF5@yahoo.com> To: Warner Losh , Konstantin Belousov , Mateusz Guzik , FreeBSD Current , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_CC(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[bsdimp.com,freebsd.org,gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.32:from]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BLOCKLISTDE_FAIL(0.00)[98.137.69.32:server fail]; APPLE_MAILER_COMMON(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.32:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-Rspamd-Queue-Id: 4dR2Ml0T0Kz3fvN On Dec 9, 2025, at 17:15, Mark Millard wrote: > On Dec 9, 2025, at 12:32, Mark Millard wrote: >=20 >> On Dec 9, 2025, at 07:22, Rozhuk Ivan wrote: >>=20 >>> On Mon, 8 Dec 2025 09:23:52 -0800 >>> Mark Millard wrote: >>>=20 >>>> But, as of yet, I've no good evidence for blaming >>>> jemalloc as a major contributor to those timing >>>> ratios --or for blaming any other specific part >>>> of 15.0 . >>>=20 >>> If you want to bench jmalloc - there is another ways to do that = without building something. >>> Try to find some sythetic benchmarks. >>> Also jmalloc can be build without OS rebuild and linked with bench. >>>=20 >>> This 2 things can reduce time to tests, but it will eliminate OS = integation factors. >>> Run same bench on different OS may give more info. >>>=20 >>=20 >> [I've eliminated direct Email to most everyone >> for this reply. There is not even minor new >> technical content.] >>=20 >> At this point I'm more likely to explore if I >> get similar ratios as ampere[13] do for some >> port-package builds that have the large ratios on >> ampere[13]. There are examples that are not as >> overall time consuming for ampere[13] as what I've >> already referenced (but are still non-trivial for >> the time taken). As stands, I do not have a good >> reproduce-the-issue context, much less one with >> build time frames I'd be willing to deal with in >> my environment. >=20 > Time-ratios similar to the ampere[13] ones for > 15.0 vs. 14.3 (or 13.5) were easily repeatable > on the Microsoft Windows Dev Kit 2023 for doing > poudriere builds of the examples that I tried. >=20 > port-package builds tested for below: devel/cmake-core > TMPFS_BLACKLIST empty > ALLOW_MAKE_JOBS=3D in use (no explicit MAKE_JOBS_NUMBER like = restrictions) > UFS context (except for what USE_TMPFS=3Dall does in poudriere) > The below did not update /usr/ports/distfiles/ . >=20 > This does some exploration of USE_TMPFS=3Dno vs. > USE_TMPFS=3Dall as well, starting with > USE_TMPFS=3Dno . >=20 > Listed in the sequence executed, first time > runs shown first: >=20 >=20 > USE_TMPFS=3Dno . . . > (Note: The first times had other port-packages to build first.) >=20 > 15.0 poudriere jail: > [00:37:37] [01] [00:12:30] Finished devel/cmake-core | = cmake-core-3.31.9: Success >=20 > 14.3 poudriere jail: > [00:28:26] [01] [00:09:38] Finished devel/cmake-core | = cmake-core-3.31.9: Success >=20 > Approx. 1.30 time ratio (15.0's 12:30 / 14.3's 9:38) >=20 >=20 > USE_TMPFS=3Dall (no tmpfs black list) . . . >=20 > 14.3 poudriere jail: > [00:09:32] [03] [00:09:24] Finished devel/cmake-core | = cmake-core-3.31.9: Success >=20 > 15.0 poudriere jail: > [00:12:45] [03] [00:12:34] Finished devel/cmake-core | = cmake-core-3.31.9: Success >=20 > Approx. 1.34 time ratio (15.0's/14.3's) >=20 >=20 > The following also prefixed the poudriere bulk -C command > with: time -l >=20 > 15.0 poudriere jail: > [00:12:36] [04] [00:12:25] Finished devel/cmake-core | = cmake-core-3.31.9: Success > . . . > 757.10 real 4613.06 user 251.09 sys > 866580 maximum resident set size > 131 average shared memory size > 27 average unshared data size > 234 average unshared stack size > 31148816 page reclaims > 0 page faults > 0 swaps > 14 block input operations > 36 block output operations > 37061 messages sent > 33671 messages received > 1758 signals received > 143987 voluntary context switches > 167515 involuntary context switches >=20 > 14.3 poudriere jail: > [00:09:23] [01] [00:09:15] Finished devel/cmake-core | = cmake-core-3.31.9: Success > . . . > 564.48 real 3449.89 user 204.14 sys > 822900 maximum resident set size > 64692 average shared memory size > 791 average unshared data size > 235 average unshared stack size > 28153497 page reclaims > 0 page faults > 0 swaps > 9 block input operations > 12 block output operations > 34180 messages sent > 31539 messages received > 1758 signals received > 131899 voluntary context switches > 132775 involuntary context switches >=20 > Approx. 1.34 time ratio (15.0's/14.3's) >=20 >=20 > USE_TMPFS=3Dno . . . (again) >=20 > 15.0 poudriere jail: > [00:13:01] [04] [00:12:27] Finished devel/cmake-core | = cmake-core-3.31.9: Success > . . . > 784.89 real 4596.42 user 257.12 sys > 866600 maximum resident set size > 128 average shared memory size > 25 average unshared data size > 234 average unshared stack size > 31194466 page reclaims > 2371 page faults > 0 swaps > 3573 block input operations > 6687 block output operations > 37643 messages sent > 33840 messages received > 1756 signals received > 241548 voluntary context switches > 304249 involuntary context switches >=20 > 14.3 poudriere jail: > [00:09:49] [04] [00:09:18] Finished devel/cmake-core | = cmake-core-3.31.9: Success > . . . > 592.83 real 3446.18 user 207.61 sys > 823880 maximum resident set size > 64712 average shared memory size > 787 average unshared data size > 236 average unshared stack size > 28176650 page reclaims > 2374 page faults > 0 swaps > 3481 block input operations > 5148 block output operations > 34521 messages sent > 31580 messages received > 1758 signals received > 218881 voluntary context switches > 255193 involuntary context switches >=20 > Approx. 1.34 time ratio (15.0's/14.3's) >=20 >=20 > Only some port-packages have time-ratios > near 1.34. For example, building lang/gcc15 > does not on ampere[13]: closer to 1.1 as > I remember. (For the most part, lang/gcc15 > does most of its own building based on a > smaller amount of clang-built code > to bootstrap.) >=20 >=20 > For reference: >=20 > # poudriere jail -l > JAILNAME VERSION OSVERSION ARCH METHOD = TIMESTAMP PATH > release14-aarch64 14.3-RELEASE-p6 1403000 arm64.aarch64 ftp-archive = 2025-12-09 12:54:06 /usr/local/poudriere/jails/release14-aarch64 > . . . > release-aarch64 15.0-RELEASE 1500068 aarch64 pkgbase = 2025-12-06 11:34:39 /usr/local/poudriere/jails/release-aarch64 > . . . >=20 > # ~/fbsd-based-on-what-commit.sh -C /usr/ports > bb7b77417165 (HEAD -> main, freebsd/main, freebsd/HEAD) www/hurl: = update 7.0.0 -> 7.1.0 > Author: Rodrigo Osorio > Commit: Rodrigo Osorio > CommitDate: 2025-11-28 23:11:52 +0000 > branch: main > merge-base: bb7b774171651eea0dc56376c225fe976231daa5 > merge-base: CommitDate: 2025-11-28 23:11:52 +0000 > n726888 (--first-parent --count for merge-base) >=20 > # uname -apKU > FreeBSD aarch64-main-pbase 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n281922-4872b48b175c GENERIC-NODEBUG arm64 aarch64 1600004 1600004 >=20 > (That last was an official pkgbase distribution.) 14.3-STABLE does not have jemalloc 5.3.0 or libsys but performs like 15.0-RELEASE, not 14.3-RELEASE for the aarch64 devel/cmake-core build tests. But 14.3-STABLE does have: # ldd /usr/local/poudriere/jails/official14-aarch64/usr/bin/cc /usr/local/poudriere/jails/official14-aarch64/usr/bin/cc: libprivateclang.so.19 =3D> /usr/lib/libprivateclang.so.19 = (0x732e0d600000) libprivatellvm.so.19 =3D> /usr/lib/libprivatellvm.so.19 = (0x732e12600000) . . . while 14.3-RELEASE does not. (Another data point is that lang/gcc15 does not have nearly as large of a time-ratio vs. 14.3-RELEASE in the data from ampere[13] .) Details from the Microsoft Dev Kit 2023 experiments . . . I've collected a sequence for a new poudriere jail to compare/contrast with: # poudriere jail -l JAILNAME VERSION OSVERSION ARCH METHOD = TIMESTAMP PATH . . . official14-aarch64 14.3-STABLE 1403506 arm64.aarch64 freebsdci = 2025-12-09 18:24:20 /usr/local/poudriere/jails/official14-aarch64 . . . (ampere[13] do not have examples of recent 14.3-STABLE builds at this = point.) USE_TMPFS=3Dno . . . (Note: The first times had other port-packages to build first. But the system still has the cached the file system data.) stable/14 poudriere jail: [00:36:29] [01] [00:12:31] Finished devel/cmake-core | = cmake-core-3.31.9: Success So: 12:31 is far more like 15.0-RELEASE USE_TMPFS=3Dall . . . stable/14 poudriere jail: [00:12:21] [07] [00:12:10] Finished devel/cmake-core | = cmake-core-3.31.9: Success . . . 742.70 real 4586.53 user 248.37 sys 864996 maximum resident set size 133 average shared memory size 24 average unshared data size 235 average unshared stack size 30958626 page reclaims 0 page faults 0 swaps 456 block input operations 80 block output operations 35920 messages sent 33223 messages received 1760 signals received 140580 voluntary context switches 164112 involuntary context switches So: 12:10 is far more like 15.0-RELEASE stable/14 poudriere jail (again): [00:12:30] [08] [00:12:19] Finished devel/cmake-core | = cmake-core-3.31.9: Success . . . 751.98 real 4604.85 user 251.40 sys 866056 maximum resident set size 125 average shared memory size 21 average unshared data size 235 average unshared stack size 30976603 page reclaims 0 page faults 0 swaps 20 block input operations 11 block output operations 36297 messages sent 33327 messages received 1761 signals received 144213 voluntary context switches 166975 involuntary context switches So: 12:19 is far more like 15.0-RELEASE USE_TMPFS=3Dno . . . (Note: The first times had other port-packages to build first.) stable/14 poudriere jail: [00:13:16] [05] [00:12:49] Finished devel/cmake-core | = cmake-core-3.31.9: Success . . . 799.95 real 4626.06 user 261.49 sys 865940 maximum resident set size 134 average shared memory size 24 average unshared data size 235 average unshared stack size 31110419 page reclaims 2380 page faults 0 swaps 3577 block input operations 6262 block output operations 37253 messages sent 33801 messages received 1758 signals received 236161 voluntary context switches 312615 involuntary context switches So: 12:49 is far more like 15.0-RELEASE (Nice to have a known repeatable context to try variations with.) =3D=3D=3D Mark Millard marklmi at yahoo.com