From nobody Fri Apr 11 18:39:37 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZZ58G5XSjz5sTGM for ; Fri, 11 Apr 2025 18:39:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZZ58G3lLKz3YS8 for ; Fri, 11 Apr 2025 18:39:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=gdp9HXg4; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744396792; bh=ghPbfjr8Nv8jezOs9j1TwAoz8BuA0o2ounDO7N7jy/4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=gdp9HXg4qF/6RZrx6KNSZoGiJ43rnrU/ikdOrmwulp1v5FroCgEvHeY7gKtZorch1fo858TzNQGr1Lh02x3A+5O07GwSptZDTa2zJ+vN2slsGsutPgVe3kY2tuYhlSeOISTriXUl9PveIvTbankhs87Qd6+33WVfIgN3TTr+/XlCNvWA//CAK1RKucmMzp2hYKbMPnvEwn0MwLpnWJ2WJHCIbfdvoMjcrkgELLQH4WmDzB7N+WXksdEiE7la6t0gM7Z6hrBdtbB3M1PMF4AwOXhlbFuvckm8BZDFDFiW1lVKV77ahAd3V8oIhfv7PH/5NeqUwf6+8UtGjmZpdQWDTQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744396792; bh=gjyW+zJTMoZwl0wYF4H4lF0oDN15Ws/0JlJGcSvlddC=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=c7y6sBGoW4KvKtb1yiglxVbwLx0iMLFGXpYYlxMiDOs2MW6eUnxmHXnIpynvb/uKk15QPpL2BXBOo8YUqZLBk7smBjJQFq+YpGQM8PWsTmcQLu27KC9F/kQJO/HhfP9e+WwiHNvgtgVYJ2pJn3dKxwZA0do2WGVc+F3WG/STQrCDV4Biyj/77CTqUFCX2KPKw1B65dFx64zcNlR12hMTmciYuuAvyEqo9ubHHaIoZ8zkhy9hHKXtVbqTqNI35A02Gs95pqZpl6cMifmwEK9odbMaFxBvGlnyY+tPwnUOJY68h4On8nXyKYdJmqc22G6GetLkKdXPskKd/xZQfHBW8Q== X-YMail-OSG: TPNwFtAVM1mqnz2yqQ2QLCbd0J1xYYWIhBeVc5_oK8I8nBeE6BFJwDpoofvYy_p wqbc8ZAId1Qo4bOHLByPGNqG1uLD0xvxV1KbYkeoMpqt5SeAjg.lUEChiNP9TeIx0EJMX6.6J8rv EA4JzdKyICEo1Aa3YDeLHRMrhG6tk5Ombn8c1U_PDKyuDY2sThqpuA.R5QdBilOFEVNxuTySeHTH loWyFGvnDkjOqBrqpDy_OfCd5FnnP5zAH8LCQ1p1uQ0Azxbj4s2w2YHWWT2PGrk1kJ3LE4brKlXd J5.CdV.MjP3Oxor_sOgflCqUqLmLQOn9KNLDMXJ6FiRHLwuYpfSb63LOfZk.uHga9N1oeMyQeXIG BaU4STt54j6b.vebUrtRdicslQWFTwJl2EIkqhJCiPyQvAY3uWiq_0YUX9ML9203DD_l88Eoq6xX Ri.odTEeevJhq6VB3jjGE6BH_0EHC2gDdVjglDyNlSZnstJrieJK921wSWSjActJJdeuwmLh459M t9pUPr8FGKwM2Ym8Va02bLoWlHa7WFon0ZN8clv.QcFxEjRLx1IRcRRoWYS9WV1VKJ1b6foQXaKw n1O2_5pnkyBuwk1EhKwd5sf1Mk.h6JPdEYt1n.DF11UumIpcyc.Cql77X.xdPEKj9T_BpQUDJc5W chygdpV3pwzPQlgAY.cxF4BboA_8eUbMAPCctR9W6UtdEtQFUJ9xjYLVvco9F.12_V_4Vl9uBIIy J_SkM_YOcN.dtVY7iEK8D3clD0gL5l8V6QUzeS1EXl918I1slZUG8MrmNYhuZDY02QzrA.fOoqNB 5xhpG.kJiQV7iYlv2zboUQ4eDMBzF8p3CPiZ.8q4D0sDBeHqBF7GkRLnTjd1cFnfkR9eM2UsAhfr YXkLRkpvBF1j.5FglxAfuh5HQjIgprkuTfO42Vwglczp6ZQcAlmkkivjhOxLfN0124ZIZt_vTKFp DbE.mnoQwfhywYytoZTAk0PfoU5rPYMlldmMX0CS.Hsxp4n9Lyvq_Ce2E4QIDiQpfj_gyzrEfkpB LE7tdnWjuEcNmzIqdrtTygjONJT0zMNyQVF.wcyuXIr04qf6BuLE9Xpn0BFt1jF9DMMbikiKOlr9 7uJdaS0p30nF9ubA5BlRIT5eJw4c..aFa8GV86wv__apbZNc1SZrEHam9u7MhOI4hlWjIkQcW3ba _BkLFRYQZCFV3KTqHa46OVXlCeXFAYLwTJEesIFSEgpxWl.Zz_UhZFSkzAktqyHPVerH.8yGSVzT q5AsKHAb9mYtxWz5p6s.v3FOboIY_qprNM93AumxoZFIPq8AjiUVGsznLzclYm0gsu4a9J4OF8f3 vE3U5rRmcYbq52mh7r7foYVAoMae5rP9daUPiUnF2sE5c8vM39NTTxNguvSQ_2qDdluKiXj_c.C. kytpMuwMnDxdqwJTxHNXJx9CM1fQYbW30I.t_uE7OMyZXKdvSAB_lqb0GtT_h3CY2r.RDRuq0Ago Y5kkUtOreHtp212Zfu5yLvnQqMsmSKD5PVsuLr7l2nJRecBIu1vqDVmpE3LLrzXkzYXwWrvClvon F..Xq7.nyWOueOM.K_UTuukbhfsETi0JvmpLz1MJ1WTfyCdW3dlsdiJYSxwXopYBobh1LY6rNFMb LvLs64KGqDFm7yWgEhsXjMO6LV_ViWxrJN2j8dBgJaCgvN3Rk4j70Z7HgIqh_IhlF4GEJjdRnCYb HsRBY31DeeVWju_RE.SRkl7Kf8impfrbkaeyrs_3w.rxZsEPlbqQiQawCnZid2zaL54HwcQpx8N0 x8PJZAc3Oc5VaFpLFi2TKnT5Z9ZORHt.RLbgXZgPzQCl5V8P5xFDGS.kptM9BAis_6bxyE5FvxZ5 rZTFhRoVxncP6cINEDKLVGp.y_QKPLPVnxknILUhq81zXaqECsedGvdOqWyHB76i7QnkvOEatxN6 P0gVVAjE9qWkMlctaeI3yvmWFk32uLzBonBTgEAV.aL1w3ffdlSvy.2ERCnRnvILvUNOhSNfZ9X6 wOgGWHyFpwjpSxYbbjp2JsHHft.RbdIsEm0GUXRYlBJ5KQVyzRFmVKo2IINxCyZNfjjLk6TXHNGt 1jnitHQyYr4YOvTeDYPoLc5xL3WDUlJ0HYgwDbbyOcIUZ3COr1BZABNz7hJcSF5CIRbotziPoJuk oCZhVZvEeq1mIux20DdYtmWwsJ8iv_vG_boF4AHJ9z8VzEKX6nxv4uHsNc_Nh_G7tz59gq4Qz7UW FjLmdphApP4cwlk1IXx0CKx3EZ21t1BF2tm5isi4h8Z9A7aF_tEmFy9sWdSrkkllTRuveXGWCYMO Ayf1p X-Sonic-MF: X-Sonic-ID: 73b568cb-f11d-467c-a998-70413635e9e1 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Fri, 11 Apr 2025 18:39:52 +0000 Received: by hermes--production-gq1-74d64bb7d7-x6m69 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e95cbde4a1c1567c2f13ee599d18f96a; Fri, 11 Apr 2025 18:39:48 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.500.181.1.5\)) Subject: Re: UPDATE: pkg 2.1.0 looks to be making official bulk builds of packages take much longer [reproducible examples notes] From: Mark Millard In-Reply-To: <7ziazrj7szuqhov3oppjbh3jyu3f2p2owntv4oxprelrdjzc6u@hkuf5szf3zwy> Date: Fri, 11 Apr 2025 11:39:37 -0700 Cc: Gleb Popov <6yearold@gmail.com>, FreeBSD Current , FreeBSD Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <8E2FBAD3-EF6F-4D99-A340-21F8FD19AE0F@yahoo.com> <84FBBAF8-025E-4B9D-9797-51735567A8DB@yahoo.com> <366E27FD-FA5B-4BF8-B6C4-6C495DB289C5@yahoo.com> <7ziazrj7szuqhov3oppjbh3jyu3f2p2owntv4oxprelrdjzc6u@hkuf5szf3zwy> To: Baptiste Daroussin X-Mailer: Apple Mail (2.3826.500.181.1.5) X-Spamd-Result: default: False [-4.50 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[98.137.64.148:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.148:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from] X-Rspamd-Queue-Id: 4ZZ58G3lLKz3YS8 X-Spamd-Bar: ---- On Apr 7, 2025, at 08:14, Baptiste Daroussin wrote: > . . . > the problem we have is the > performance changes depending on what is happening in parallel on the = machines. In separate list messages I've provided multiple examples of the time-taking issue that do not depend on what is running in parallel on the machines, no parallel builds involved. Part of the issue is that there are thousands of examples of "small build-step time" packages for which the build-depends, lib-depends, run-depends combination, takes notable time, given that the total time contribution across those thousands of package builds is notable overall. As stands, mostly it is the early part of "bulk -c -a" avoids the issue via building packages that have no or few dependencies. Later "small build-step time" packages tend to have various dependencies, greatly changing the time scale for their builds. Few builds are of "large build-step time" packages (relative to there being 30000+ packages). That=20 has implications for there being 30000+ packages to build for "bulk -c -a" or other builds with large numbers of packages to try to build. > which makes the performance issues invisible on local poudriere if you = want to > test it on port A or port B, I've provided counter examples to that that only involve the one builder, after the prerequisites have already been built (same or prior bulk run). > if we want to reduce the performance penalty we > need to be able to make a reproducible case which can then be = profiled, to know > where to optimize if needed. I've provided examples of such . . . (time intervals shown are from the aarch64 Windows Dev Kit 2023 with just the one builder active) www/rt50 build-depends: 00:00:27->00:08:46 devel/py-inline-snapshot@py311=20 build-depends: 00:00:01->00:00:55 run-depends: 00:00:56->00:01:47 mail/mailest@nox build-depends: 00:00:01->00:00:28 run-depends: 00:00:30->00:00:59 devel/dwarves build-depends: 00:00:05->00:02:23 lib-depends: 00:02:23->00:02:42 The timings are from the column next to the Building/Status/Finished column from using bulk -v , not from the column for the overall bulk run. > I have tried to reproduce each individual case which happen in the = ports tree > and I am not able to reproduce them, so impossible to know where to = look at > exactly. Try some of the examples that I've provided? There are more examples that I could check and report non-parallel timings on if you want. I just picked to report on only a few initially. An example that you might want is my providing more examples of lib-depends with non-parallel timings. > I know what is new and what causes the performance penalty, but not > which part is causing the super extra penalty on the cluster. Various examples reproduce the timing issues outside the cluster and without the parallel builds. =3D=3D=3D Mark Millard marklmi at yahoo.com