From nobody Sun Dec 07 03:03:43 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dP91d4lvNz6JdWt for ; Sun, 07 Dec 2025 03:04:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4dP91d1360z3RkW for ; Sun, 07 Dec 2025 03:04:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765076637; bh=c9CUEZtfWZ4n8x7rbYegKZiu3lQ/S/zXVTj3T+Y7poY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=dTVcbJpBBgJXPYkAnQEGFqIdfdzAWXUIyRS6qdeNzsuV3Uyci4zk94RiBaZImyPL9/g4NHThOTNNqyIWxI2p/9GFF/VWbRAeuRcOUTZ4kWF1Ns/RG1MLWUFCvhYGDFY3X4Xf9eWvzxk6ksTxDiVck8v2tX2D3cJ+TPWx//fzdGscYwwZbJScngDLagQp5DYuGSmkoQK5xWmNdgjBMOR3I3h+IfiMhwTAEWQM/pSk6Q36CYXA1hj+vkccxyIg2ouJe0ZHtwLaLesOGuAMX1CQIzAGZ8umQN03d2QVyYnfSARI8yzIlJTTo1UC2COcSASYl/wLR0MNNR1ggQ5nuGyYiQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1765076637; bh=5K6LafJUgn/r4be33Nu337F1hNmtCvLajRJBi6zXZvV=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=mcen/T2lCRQvAaSdqjbkpNSfGPa0HLxHYwIagHZJP92noAjh1By5cJB74baQVFOqIp0KBeAi5tpt7ZVBk3DqbN3evMpI11ZOQcLSbxJfXFsH9e7Nhu7lLae+D2uCzZtfALsN2wiWZ0b6ImqbQTurmPuT+e0ud01TC/IhKbgQrLKkSqSuqYdQbWvhNsxGFylHZF6hREE8VOfegEYrK54DEQT9qgyCIWMnjRGB+U/OeerC1nAUdZuxt5hHJHNqLQuhv2qa3KCRHqw1Zr2t68Nr6hw8cJuhzwS9SlB/UELQ9n61m7F2msbVOiF9VGAAcZFXppVHr5e+HBlq8erHDgqLjw== X-YMail-OSG: fPqlas4VM1k__YIoBNjlz_iV44bKygNwxuKVH99NVst9T0bVXqKQpZ.r.EFIyb1 t3oM4M_n_PVPZmH66_yMYPOpiay_0PshGI7wxwANx6tCzDxHOTjwmXtTrZOk08p._PRjrMvIlzL1 x_vPkTxxkhlcAVWFG76u9Nug.CsFhMtb49miyV09qYf6l24Gf_Wy02nMbB8XQk7Cgqv75RIWxJ3X guCwtUHNySmjSOf_JHlps28kGy_oF1ip5NsP4kSNTcBVxaqRLZIp32Cxvn_sVMfrFiUTkGyRw5UW MzUrbVjXhSIyD_RysHKZCDeYIv8ZIEfpVsw67yhINrbz3mMG2ivCz3fc0.i_w8GGsfyRAezJoxE6 nGFOLgUcxupqHj2P.LaRWetY65HNRsDrKnXoMeSyGGCqgCGsxegR2Do_VJT8SgnYicL2d.UZYQKe RV1RPwCROj.q1aoWFK8_q12PmVNwa559rAuDCJsCneXoLg13SoSVYMnXCLLbdeHXm95HciVS45bi 8bL4YOOlcVqpaF4uBZbWzbxKxcb3uoMsCZArORK_eHU8kHHU99lhBZ0qO8egAK0bfHgjsIVnqiYl ykdixErUlH3a_hB7zOF2aRerU1zrrWv1E2wOotEQ14Mlvd40aP6nKC9pgy8nxEN7Rw17aXDw8Pzj c8iN6nW_Fz6lex388BqTZqHQ42mowS0aFYpgM9bvjDju_U9HQUSKn_MgQRTSvMyx6RWKL8Zu8Dws 7mkn8RGuSgBavGqfpjjzmedsEzJv0Q3hwiuBS0f57bhxxTussTYjZpKzEmszr.vVAfE_VY7FMHDi K3pVvI2vbnbH5HwXpBeh5QqPyyb2PlyRuCYKDDblcEnzMUweW0kg4smaj56FRKAD2sd3Xn_xlhHp 9pdCETtRw1imomF110yWnK04RYvOE3AMfz9xUJ1DP8nDMknuMb7PbgDSUJ0lE7e7pi_jBcCTg5Bu pscsBL0yVDeGr5U.dQfJwz7HujCaAcpGCa05fUDe2bcQe0VxigqqPjQ3yhAyE1AvuG0Ia1YsIMRj KCxwfcBHgr2REU47YwUnHY7dIlSnyLC_Fb7pKg311WPinT7N0gzDhI372CiNGe6BAkKhDZ6hpRb9 UuBm0oYTw.xKlSAuZ59SFBp09vQmZGyh8m3T2KOIcGmhe5NWv06bxZomlfPPpxh.5qkDQ3MWK4Ts mvUqG38fcrju1o.752inFX6mrt9vQ4j7LZIzITSCV.Pub4iYFVpz7OxFIE5qwL9qOGVqqhidybte jpzPKqb.5jE.k6pbXO6Kk5_Iho9petMVPxRZwhihybVw7RCvuI513QDwHb9ACzkcdvJ3SecFYHx3 uT7nzYvKckUIAHrO74n0dviSWV8siJEVitV3Zy9ttbGtXGRp6ZZUW8dCBcsoVQdKFCe6q8S1bKYa hiFTJwfEK43fbUs.6FHx8hSrIomxqagHz9KwKfqV0yZrW1zHcQMRZtzOZ.eLIXdVBOZkyYQlKZaE VCfh0wuJ5g83dlh7jxuo1wm.UoVCiUKGhYZFqqcBc3a1JGv.OLTaKYz87JwqqZ.kLi8wKgN_cPF0 twi_dBDkEQPVmrroFiC_u.goFPdkBmsirGTGM_7dyKZpOyizn0h.yiEs0kJRTMqzbuOtCT.jxVMl AeidFWDhgM3TvF9Xs6HmkaUW237QzbwchKz14HGaLeXHq.D9hSYKLN1m2sLS_O5ufaAykraXgwVw fjsTXCvCX28ivUOChmuwaIQtOjNPJ0pTlBv9GitOZcEwDX_5pe6Wcoy1n6NbTI4m1acW9_JjqD4j qnm2.1F83HExxR_P3Sq83jmTLMQlY0v5zDqB66lJNsLJOHQm1CcJf6Udw7MZkF1W.Slnhv2TDumf OnPJRhHdJ4aY1wcXub2_uE1skTst1il88lM5VsVz2_atXmF_1gsR5J3hO_xqz1yEJ1V8q0H3eFKF c2nd.TScaNXEK7HNVWPbbGwxj0a1nBj_RDBkgcpExI0FrWww.qMkDsR0VROcfNjL72vazjR0CTIw b1WikB2aCTx5OR.9MwEbImosVJeM0Wkzv4ym_MnLm1rNfiUkqlNpsMw9wrhhZblPU7vX1lzqmA4T sMYfXXcDkqgoHexkvZ_CeqxygxF5j5UXZxsCeGa6JG4e318x1GnFir.nXd7vHlK.oF_GbuBRVRPu 3KKvN5QqnbB.lyDk367nLLot3HGbVaYe2Faf5NTAdWROjovPRovLy__bLSYTDPuCscK5stCUiWRJ NzdUKvmumjAbffa9YVwwLlqnNScj3chi2MdophrNAtVGcC7Olb5_1HBqAMNBLLMdjTQeIGUKNKcD UITE- X-Sonic-MF: X-Sonic-ID: d1afce47-859e-4a03-940d-e5303235ed81 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sun, 7 Dec 2025 03:03:57 +0000 Received: by hermes--production-gq1-54bf57fc64-fqp47 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 0a68949ec4045250f901a14013482005; Sun, 07 Dec 2025 03:03:53 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: performance regressions in 15.0 From: Mark Millard In-Reply-To: Date: Sat, 6 Dec 2025 19:03:43 -0800 Cc: FreeBSD Current , FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <19A848A6-0042-4873-B70D-AD6805225B92@yahoo.com> References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> To: Warner Losh X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dP91d1360z3RkW On Dec 6, 2025, at 14:25, Warner Losh wrote: > On Sat, Dec 6, 2025, 3:06=E2=80=AFPM Mark Millard = wrote: >=20 >> On Dec 6, 2025, at 06:14, Mark Millard wrote: >>=20 >> > Mateusz Guzik wrote on >> > Date: Sat, 06 Dec 2025 10:50:08 UTC : >> >=20 >> >> I got pointed at phoronix: = https://www.phoronix.com/review/freebsd-15-amd-epyc >> >>=20 >> >> While I don't treat their results as gospel, a FreeBSD vs FreeBSD = test >> >> showing a slowdown most definitely warrants a closer look. >> >>=20 >> >> They observed slowdowns when using iperf over localhost and when = compiling llvm. >> >>=20 >> >> I can confirm both problems and more. >> >>=20 >> >> I found the profiling tooling for userspace to be broken again so = I >> >> did not investigate much and I'm not going to dig into it further. >> >>=20 >> >> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems >> >> running as 8 core vms under kvm. >> >> . . . >> >=20 >> >=20 >> >=20 >> > Both of the below are from ampere3 (aarch64) instead, its >> > 2 most recent "bulk -a" runs that completed, elapsed times >> > shown for qt6-webengine-6.9.3 builds: >> >=20 >> > 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46 >> > 135arm64-default qt6-webengine-6.9.3 38:43:36 >> >=20 >> > For reference: >> >=20 >> > Host OSVERSION: 1600000 >> > Jail OSVERSION: 1500068 >> >=20 >> > vs. >> >=20 >> > Host OSVERSION: 1600000 >> > Jail OSVERSION: 1305000 >> >=20 >> > The difference for the above is in the Jail's world builds, >> > not in the boot's (kernel+world) builds. >> >=20 >> >=20 >> > For reference: >> >=20 >> >=20 >> > = https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D150releng-a= rm64-quarterly&build=3D88084f9163ae >> >=20 >> > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov = 30 05:40:02 -00 2025 >> > build time: 2D:05:33:52 >> >=20 >> >=20 >> > = https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D135arm64-de= fault&build=3Df5384fe59be6 >> >=20 >> > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov = 22 15:33:34 -00 2025 >> > build time: 1D:14:43:41 >>=20 >>=20 >> Expanding the notes to before and after jemalloc 5.3.0 >> was merged to main: beefy18 was the main-amd64 builder >> before and somewhat after the jemalloc 5.3.0 merge from >> vendor branch: >>=20 >> Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 = port-packages, start 05 Aug 2025 01:10:59 GMT >> ( jemalloc 5.3.0 merge from = vendor branch: 15 Aug 2025) >> After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 = port-packages, start 19 Aug 2025 01:30:33 GMT >>=20 >> (The log files are long gone for port-packages built.) >>=20 >> main-15 used a debug jail world but 15.0-RELEASE does not. >>=20 >> I'm not aware of such a port-package builder context for a >> non-debug jail world before and after a jemalloc 5.3.0 merge. >>=20 > A few months before I landed the jemalloc patches, i did 4 or 5 from = dirt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to = see maybe a diff with the small sample size, but not enough for ministat = to trigger at 95%. I didn't recall keeping the data for this and can't = find it now. And I'm not even sure, in hindsight, I ran a good = experiment. It might be related, or not, but it would be easy enough for = someone to setup a two jails: one just before and one just after. Build = from scratch the world (same hash) on both. That would test it since = you'd be holding all other variables constant. >=20 > When we imported the tip of FreeBSD main at work, we didn't get a cpu = change trigger from our tests that I recall... The range of commits look like: =E2=80=A2 git: 9a7c512a6149 - main - ucred groups: restore a useful = comment Eric van Gyzen=20 =E2=80=A2 git: bf6039f09a30 - main - jemalloc: Unthin = contrib/jemalloc Warner Losh=20 =E2=80=A2 git: a0dfba697132 - main - jemalloc: Update = jemalloc.xml.in per FreeBSD-diffs Warner Losh=20 =E2=80=A2 git: 718b13ba6c5d - main - jemalloc: Add FreeBSD's updates = to jemalloc_preamble.h.in Warner Losh=20 =E2=80=A2 git: 6371645df7b0 - main - jemalloc: Add = JEMALLOC_PRIVATE_NAMESPACE for the libc namespace Warner Losh=20 =E2=80=A2 git: da260ab23f26 - main - jemalloc: Only replace = _pthread_mutex_init_calloc_cb in private namespace Warner Losh=20 =E2=80=A2 git: c43cad871720 - main - jemalloc: Merge from jemalloc = 5.3.0 vendor branch Warner Losh=20 =E2=80=A2 git: 69af14a57c9e - main - jemalloc: Note update in = UPDATING and RELNOTES Warner Losh=20 I've started a build of a non-debug 9a7c512a6149 world to later create a chroot to do a test buildworld in. I'll also do a build of a non-debug 69af14a57c9e world to later create the other chroot to do a test buildworld in. non-debug means my use of: WITH_MALLOC_PRODUCTION=3D WITHOUT_ASSERT_DEBUG=3D WITHOUT_PTHREADS_ASSERTIONS=3D WITHOUT_LLVM_ASSERTIONS=3D I've used "env WITH_META_MODE=3D" as it cuts down on the volume and frequency of scrolling output. I'll do the same later. If there is anything you want controlled in a different way, let me know. The Windows Dev Kit 2023 is booted (world and kernel) with: # uname -apKU FreeBSD aarch64-main-pbase 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n281922-4872b48b175c GENERIC-NODEBUG arm64 aarch64 1600004 1600004 which is from an official pkgbase distribution. So the boot-world is a debug world but the boot-kernel is not. The Windows Dev Kit 2023 will take some time for such -j8 builds and I may end up sleeping in the middle of the sequence someplace. So it may be a while before I've any comparison/contrast data to report. =3D=3D=3D Mark Millard marklmi at yahoo.com