From nobody Mon Apr 07 20:15:42 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZWgSy3XH4z5s6M0 for ; Mon, 07 Apr 2025 20:16:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZWgSy1VDsz44Vg for ; Mon, 07 Apr 2025 20:16:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=qdzB2SoX; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744056955; bh=W9QOVogsxHtkt7GvaOzwWUts4IX/vfLLkdo3X+0nUjU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=qdzB2SoXFQbT/PrpITTiWeFZjC51Q/1BJ6WJdsR+avqtg34QLboc04qmkY37licKpvtXACrlXFrHXoLDfPL7RCvA/41o5Spdb18ltxDCJ5Ek3IF+R9VXjBckzcfHhShWu1n2a7p/EijlHNgknoI8cm2i5Cf86WAKc8BabG0frNkNrWoLQQwuOCR2+wzEYp+TaVH2yI8xQiJ+GXUCP3PLsp6FHY/YS65YtRQqDgV0V+BDzxibyOnMFCgxjcawJ2PrOHCI8xLMI0jF5WzaDySM3tVcqmkMEB7GRH7iZ9evJgaoG5/XCBpqYODJc8BsbnuGhiSDVkn2Uc7cFeYsdSbIzA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744056955; bh=ZGEyxkN+Rz3cNLUV1OLHM/dQmOMHI7fCEvlW1S6L4Ot=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=DfeYFGMFzBA5iD8rFIq7ujmgiDu5YDci93gtVohjZ71NbKKCuFqwON2UtEzTmkNDNt8FB+fBVMPF5rVPMMOItVLChCIy4KBf/85jaTojs/6aAVzQhaCBXBiRzPlA4VKDJ7+9gyHpPyImPqc9mG4N1HtmpzJHUZ1E3EkQJFvWoeYjZxlaeckMsaK4px00OKJLHeLXeFRkeNCpfiIf3/ntxgdpg0rTIbLHnp6uG2WHGWo0KGzxaC9eLRwkdwGMcjmGp8S4XCTp+sptAmFjyCoSRTSTi0E/avAn1p86gzG4olrcj4P7CLhglkhGGDLV6IdkdatIf8WDdRu0gxpv/xZdWg== X-YMail-OSG: 5VX6Tl4VM1lYlpjmiTjBHPiF4w.k8WVDewar.LmpP9DjGHa2WlegYlx5.wmc615 nH7MzjWWKcq5inh4tNA.qq5WkHSME62IooBqkdOvzu2NSDVfdpWzZeKg8I5_rYTjdfMSiX9IC_pr iESwwtu0FfeXVjgiyj7JVai84inTTPtr2ncIHRlAqaIMn7XWe0.8MAAp7DufyVVsdYHnjtRayLHr m4rwOFZESQnC1Yiq30GFeDY2yD39aCcFWw1h_Za2EYJ.ikYz.3jAQcArQTO2ZTdW49d9NYFj3Osz 5YWCmdtV9pCZYIUn1suv1WGnu45boioBK.nLU8szWAhm6dXV3o6lzvTbYbRPwDLjscWXaP6yUPjg ..9E5KdB3n38MiEK3nLQo.fNmcYqIiOjplo7By3e3O.Dqr2T5FwGrCnVYQ6rtjIa7h3WyVniVKGn r6bnheVyqv4nDg9vE91D8KGeU_.MFS6MTWCoJ1Ww9jwbeKImyOcYGRP6VR_uJwaS4R4LqqQR2PDX rEiTVVractBsDp2Li6IOfzoX1nKD0ak6rADzIeoyIxsCkfOs0TlIGOrgIl5zr8lIfRkuGIPeMHzF 0jMOcxfyx9EeCE1DNsNDO_U.0kn2FtD6D1g3nLxZRTqOGGiI3pD.45W0HiUNRg4.KVosTVYirEwS KhrmfHqUGq7OkJaO9ViKGDLJYRNooui_WwkEwSsandzZLEqUWG.A23iucRaAPRZM7pz8oI05_C0K paRmfalPRrAHj4y1HWnOt4y6GOKaawut8TV07q5CeheV23cb9EAyWwrNeE_uCdDRVM.6_dGuimY2 0XFiK5pwyxTpm9TD5f2sywSHqyyC5fLXKdxFDONfMxIARhwr5I6gSFgzZJrQyWVF7j26bad_GJhU oj7ruGdY6SHofSLKfgIZgtdNisWzwr.JRfOqMjlneLNdKpslZW7ruSmljhpK2_edlFdzv5Mg6rrZ GsIKzgRUZh7Jig65B5ucBN4u6MlbK3RHRtDrWvoop4gjib1ZUFCo7C2gEBzUzcWap4aVtgXfABq0 L0X4GPMZEzJ.Mzm5nTCLhGGV6pB42WKk_diCm9Y9vg04yr1m6MWs8fEtGt9i7BFCwA9OBWse1Onx r_Uq.je9DZL7brlTIri5qYaQSYZM848_pHQT7464lCtKFyYgGFDczMWQwjqSc37rAS2A07kZCjXi lCidd6E2jkNbCGzQ8CLDx3YwHt8uGP_qjUrA6oy9QPtZNhiKT8bDSEoDrpAXgJK.FlQbaXcgWN_f gT5s_LGayrm7rV1Rwd_vJC_KotGashew_Dj6p.xPQ7SiODpnrGR_KzBeom1zLa_5yaEIR4XQUyY8 QbZtrvA3VXQq..PjU51IC.3CjexK.jj7IVchrmmiRNz.8MBFmRb9Mx7OgpLh3Q.NsAkMg4Y_T8kz NI6B.PN8CJMKp8R.ZMZbgsH5nvyon2v00K_pFPsuR_JYB_0aCuxMwecx_gZVW.9r7sQTo9EA1AMZ zrtwwYovr3IKWBJv3DJAb.El721OELniy.DgCPygsv_S_4Q32UGZEE8WXgo.1kfNwlpCsySS0uCe DjWmB1Q5VjovRKgWS6ms_i7BrgwhIutfDG9cDTxoz7tID7INZ7KpUbrRvLZGg87NubWLk.68NIWi Q6Af5pnI8pq86AR7eWpbOsJpApzHATS9YH0wvcRm_k_rIqRpEoVYsNkBsJS62ZKcU2954TuCBXTP kAv9tG23bai4BE6_RDxVOX6eQnt3TAgVepF2B7.nFYW0CawqG4IoSEIsNdt7C1tVrKO0o1cp_.XB vjqCrQSDOcem61hGRnTA66HMY6.0jfkH6ZWJFaROQLrgKdTPHV_jNCxqnqWD8AuQLm69BqekAppi KzIdBXQua309s5fLJoDvGfxGJaz5WKg31KLqqMJtBSBD8ORXRrKCzaDKEBMVdKpoQdSJtE6TXu2C UUHEZkozTK2Ta3YGSmhWSepJ5ifL1eJ_YL9PAbmWLyAliwd4.d_Wy77UB8V_scyuCiGFncR0kibs a4dmkZLi.6gdAyoB.HUUs3gJR419Gt1rTesjola869VREcdj3bkDZeHfo.xcxFGufAsmbpPFtFb4 ITIeFIRaaH3IL4CJrVRUFciGgB.QI0Apjc4ylriSGkDA0QxxOICBqklpLDmpwTe2baXBkFc87Bfj W5pDmR93tSMk7o0safDRgR0RrCigIJbGdYVOeVrfwbyETW9J7z.EoD2xbQ85bmJHpKGKvSgrSYkE hnl177BhtM3lpgfFsN9O.CxGHOnUfdTRhpdEY856msW5KneT31EAWj0IXF_E.nuQtEBpJLWyR0w6 y4mY9.38- X-Sonic-MF: X-Sonic-ID: 7bfaea4d-da0b-4d89-8b16-861af5758c3d Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 7 Apr 2025 20:15:55 +0000 Received: by hermes--production-gq1-5c477bf655-pmr4h (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 06735517be0eb9e79ee2bd30e7713c81; Mon, 07 Apr 2025 20:15:53 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.500.181.1.5\)) Subject: Re: UPDATE: pkg 2.1.0 looks to be making official bulk builds of packages take much longer [a specific package with large time factor] From: Mark Millard In-Reply-To: <05128806-8F20-43A6-946B-A3780259F61A@yahoo.com> Date: Mon, 7 Apr 2025 13:15:42 -0700 Cc: Gleb Popov <6yearold@gmail.com>, FreeBSD Current , FreeBSD Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <178AD8CD-57F2-4ADB-AE82-6B0C7A4D2C01@yahoo.com> References: <8E2FBAD3-EF6F-4D99-A340-21F8FD19AE0F@yahoo.com> <84FBBAF8-025E-4B9D-9797-51735567A8DB@yahoo.com> <366E27FD-FA5B-4BF8-B6C4-6C495DB289C5@yahoo.com> <7ziazrj7szuqhov3oppjbh3jyu3f2p2owntv4oxprelrdjzc6u@hkuf5szf3zwy> <0414EBFB-B63A-4738-ADB0-38B6CF3725DA@yahoo.com> <05128806-8F20-43A6-946B-A3780259F61A@yahoo.com> To: Baptiste Daroussin X-Mailer: Apple Mail (2.3826.500.181.1.5) X-Spamd-Result: default: False [-4.50 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[98.137.69.205:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-Rspamd-Queue-Id: 4ZWgSy1VDsz44Vg X-Spamd-Bar: ---- On Apr 7, 2025, at 10:15, Mark Millard wrote: > On Apr 7, 2025, at 09:44, Mark Millard wrote: >=20 >> On Apr 7, 2025, at 08:14, Baptiste Daroussin = wrote: >>=20 >>> On Mon 07 Apr 08:07, Mark Millard wrote: >>>> . . . >>>=20 >>> Listing like this is clearly not any useful, the problem we have is = the >>> performance changes depending on what is happening in parallel on = the machines. >>=20 >> I've been exploring looking for an example that reproduces the >> timing issue via commands like: >>=20 >> # poudriere bulk -jrelease-aarch64 -v -p default -c www/gitlab@ee >> vs. >> # poudriere bulk -jrelease-aarch64 -v -p alt -c www/gitlab@ee >>=20 >> so that prior builds are not involved in creating such a context. >> Also, when www/gitlab@ee itself is building, no other builder will >> be active. >>=20 >> I've started such a build based on a pkg 2.0.6 /usr/ports/ context >> and will try one based on a pkg 2.1.0 /usr/ports-alt/ context. >>=20 >> I'm trying www/gitlab@ee because, on beefy17, it went from: >>=20 >> 00:09:01 (pre pkg 2.1.0 example) >=20 > I must have clicked on the wrong thing for the above. Looking again: >=20 > build of www/gitlab@ee | gitlab-ee-17.10.0 ended at Wed Mar 26 = 10:27:22 UTC 2025 > build time: 00:13:50 >=20 >> to: >> 05:35:01 (pkg 23.1.0 example) >>=20 >> (so somewhat over 37 times longer) Make that "somewhat over 24 times". >> and when I looked it up >> it has a huge number of dependencies: >>=20 >> # pkg rquery -U -r FreeBSD "%#d : %n %o" www/gitlab@ee >> 298 : gitlab-ee www/gitlab >>=20 >> The factor of 37 24 instead >> is large enough to be unlikely to have only >> load averages on beefy17 as a major contributor. It is still unlikely that beefy17 had load averages of anywhere in the ballpark of 20 times the number of FreeBSD cpus. >> Given the >> evidence about the count of dependencies, I will see. what >> I get. >>=20 >> The test environment is a Apple Silicon M4 MAX system with >> FreeBSD running under Parallels in macOS. >>=20 >> [00:00:07] Building 943 packages using up to 14 builders >>=20 >>=20 >> OOPS (via checking ampere2 logs): >>=20 >> Looks like aarch64 might end up blocked for a >> rubygem-redis-actionpack-rails70 "Phase: stage" failure. I >> may have to set up a amd64 context for such experiments. >=20 > The above looks to be another example of looking at > that wrong thing. >=20 > But, as I'm using 2 different vintages of ports tree, > there could be an issue for 1 even if there is not for > the other. >=20 >>> which makes the performance issues invisible on local poudriere if = you want to >>> test it on port A or port B, if we want to reduce the performance = penalty we >>> need to be able to make a reproducible case which can then be = profiled, to know >>> where to optimize if needed. >>>=20 >>> I have tried to reproduce each individual case which happen in the = ports tree >>> and I am not able to reproduce them, so impossible to know where to = look at >>> exactly. >>=20 >> I'm hoping to supply reproducible steps. >>=20 >>> I know what is new and what causes the performance penalty, but not >>> which part is causing the super extra penalty on the cluster. >=20 I'll note that both beefy17/18 and ampere2 get the time multipler effect. It it likely more noticable on ampere2 because ampere2 is likely a slower system generally. As for reproduction of the issue (in a much faster context), I also got more like about 26 sec in build-depends instead of about 12 sec. The overall time for building the 943 packages about about 10 min longer. The pkg 2.0.6 vintage /usr/ports/ context got: (Note: There no "Allowing MAKE_JOBS for" notice for www/gitlab@ee .) [01:29:05] [01] [00:00:00] Building www/gitlab@ee | gitlab-ee-17.9.2_1 [01:29:06] [01] [00:00:01] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: check-sanity [01:29:06] [01] [00:00:01] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: pkg-depends [01:29:06] [01] [00:00:01] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: fetch-depends [01:29:06] [01] [00:00:01] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: fetch [01:29:46] [01] [00:00:41] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: checksum [01:29:46] [01] [00:00:41] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: extract-depends [01:29:47] [01] [00:00:42] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: extract [01:29:56] [01] [00:00:51] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: patch-depends [01:29:56] [01] [00:00:51] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: patch [01:29:56] [01] [00:00:51] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: build-depends [01:30:08] [01] [00:01:03] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: lib-depends [01:30:08] [01] [00:01:03] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: configure [01:30:09] [01] [00:01:04] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: build [01:30:09] [01] [00:01:04] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: run-depends [01:30:09] [01] [00:01:04] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: stage [01:30:16] [01] [00:01:11] Status www/gitlab@ee | = gitlab-ee-17.9.2_1: package [01:31:47] [01] [00:02:42] Finished www/gitlab@ee | = gitlab-ee-17.9.2_1: Success . . . [01:31:50] [release-aarch64-default] [2025-04-07_09h06m32s] [committing] = Time: 01:31:48 Queued: 943 Inspected: 0 Ignored: 0 Built: 943 Failed: 0 = Skipped: 0 Fetched: 0 Remaining: 0 . . . The pkg 2.1.0 vintage /usr/ports-alt/ context: (Note: There was no "Allowing MAKE_JOBS for" notice for www/gitlab@ee .) [01:39:03] [01] [00:00:00] Building www/gitlab@ee | gitlab-ee-17.10.3 [01:39:04] [01] [00:00:01] Status www/gitlab@ee | gitlab-ee-17.10.3: = check-sanity [01:39:04] [01] [00:00:01] Status www/gitlab@ee | gitlab-ee-17.10.3: = pkg-depends [01:39:04] [01] [00:00:01] Status www/gitlab@ee | gitlab-ee-17.10.3: = fetch-depends [01:39:04] [01] [00:00:01] Status www/gitlab@ee | gitlab-ee-17.10.3: = fetch [01:39:46] [01] [00:00:43] Status www/gitlab@ee | gitlab-ee-17.10.3: = checksum [01:39:46] [01] [00:00:43] Status www/gitlab@ee | gitlab-ee-17.10.3: = extract-depends [01:39:47] [01] [00:00:44] Status www/gitlab@ee | gitlab-ee-17.10.3: = extract [01:39:57] [01] [00:00:54] Status www/gitlab@ee | gitlab-ee-17.10.3: = patch-depends [01:39:57] [01] [00:00:54] Status www/gitlab@ee | gitlab-ee-17.10.3: = patch [01:39:57] [01] [00:00:54] Status www/gitlab@ee | gitlab-ee-17.10.3: = build-depends [01:40:23] [01] [00:01:20] Status www/gitlab@ee | gitlab-ee-17.10.3: = lib-depends [01:40:23] [01] [00:01:20] Status www/gitlab@ee | gitlab-ee-17.10.3: = configure [01:40:23] [01] [00:01:20] Status www/gitlab@ee | gitlab-ee-17.10.3: = build [01:40:23] [01] [00:01:20] Status www/gitlab@ee | gitlab-ee-17.10.3: = run-depends [01:40:24] [01] [00:01:21] Status www/gitlab@ee | gitlab-ee-17.10.3: = stage [01:40:30] [01] [00:01:27] Status www/gitlab@ee | gitlab-ee-17.10.3: = package [01:42:17] [01] [00:03:14] Finished www/gitlab@ee | gitlab-ee-17.10.3: = Success . . . [01:42:20] [release-aarch64-alt] [2025-04-07_10h44m17s] [committing] = Time: 01:42:08 Queued: 943 Inspected: 0 Ignored: 0 Built: 943 Failed: 0 = Skipped: 0 Fetched: 0 Remaining: 0 . . . This sort of reminds me of an old issue I had on the PowerMac's back when I had access to working ones: poudriere had a contention issue with cpdup activity that made for long delays in the initial, parallel cpdup activity. I modified poudriere to stagger the activity in time to avoid it and cut the time greatly. (Not that such is known to be involved here.) Others did not necessarily see the same on other types of hardware. (I did not have access to a aarch64 or armv7 context at the time.) =3D=3D=3D Mark Millard marklmi at yahoo.com