From nobody Fri Apr 11 21:04:49 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZZ8Mh2jJ4z5sgPV for ; Fri, 11 Apr 2025 21:05:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZZ8Mg13vrz3r7h for ; Fri, 11 Apr 2025 21:05:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=I0U9lGo0; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744405496; bh=zX9t+ewjCbo04maeOYBXBh7KlF1tXeGWDK3XaE3XVTM=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=I0U9lGo0rmUyW4X5aWcSRMXs9SbZdFM35XyIqAzyGUF0gSgRcS68Yf+dcXfQhupcPnytE/TuTzmMXo9TRsIrVAaNqalC0XiFvBtnuSh8KYx9psEwqMmFqPN/+w2LkQFFHHRBcNJKkHNRtVUprFvIBnIH4O1j7AWZ6CtCJvrYeu57Si3jud2Oq3cQEmwNblhOS264B4stExY9XxG1p3WAXlH8pgPjzVbei65JdXIVo5t/yNwgg6B6NdYlOLDmO+2gy+eGTAp/q9HimYVgH/huOuZLzvNHSwpAlaSr1xWtM1Q44xZFzNnhSE76C0ur6nFsSd9lIodeVDVKJFdbeHc+hQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1744405496; bh=vYCb+h+VJNAVzM6Sf4EzZ3vaX+IhHQi92i7uB2xiOJx=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=hu9GisHot6qYULl6rOXvy87HVWMp/sWfA46YLs5xTg+i0L6o63vq5MwAXKq8cT3dWT1701mxHeoFAOAixdLb2mchZOoOw5tjXwuMMvGIn89kJsn8Qf2soV6fbQzhzvy3yOH35q4EKh7DuTiPacM2uLTqcord1/F+4N0A1Dz/iwxjgiLMTvGlCTaDoGsdrcyA4dB8aaCuN1Jxaib96VP1EnNEfVYNSSM0sJ+haaChS10jTdLBoh16jjVSHEaWxKpy01PYguanVn/ucYphT4wtZ7Naw728L9qzrAHLmmJG1Z9iuiOrQiAN9JSgAN6tD0F8Uy0Ut1rWj9bhrnrEOazQWA== X-YMail-OSG: TK.OvNMVM1nsac_Vdp2ILaVMe6Y5tPpNdi4yB7U4gcBbnJSH53FL6X6HPFf3Bt1 b96LQWE9Vf1U70r0EmFpi6yopxGzYdVvOduhSHR7Mp_4uCm7s3FbO0pdb8SoPYhEFazDMwMzZm9G tCtqymn9MEAzS5fwAmjH5QcMymI7rLyIRI3egZYmco.wwRvsbFPtVjNCSURw6yBlTrTEWNHl5rMx 2mWF.sgmYt5XWAqJ78N2Opm1dJOfSWLTMKEaxa6EOnDo8nn4IL40yug.VIlTLoWXnNsyYyfDaYge 1wlhXaNcexwa7zac2M0L42KCaJicvhYMvUSSi3UIS4lXstChEpAm0VCSmcv3.1otIc3qsVA4rljc SJ5II6ncM6Vg3ft6HzVJt1o_1ojAQwpqG665ofkIaMuhGa5Z3sO.V41rBjheuFThES.6eXnFhL9l bWQN6OJ_978_MP.aZzf2mLgcmf3bqFrSM5VsLUy48g9Mtzn4ERicmmM0gUgPVpnGfxyl5dcaU.tj 2oW_kuh1af2YPt.DQMSufzxu4cNd.ivYkltYWKPXU40CgJrRNiLSLUIPe1bvVlvwHo2IXSnH_a18 3.gGv4_3CuUi2ql.kGWtbYd7rBIcGVTOzqrHbK_YqY_gL0Uvjr6tJREHwjgEsMrzdl.TVhTjH10v WkgVRgYbvPEluevKoZdG9Qq0vWAtn2TtNbvObGwvG6789NtRuZTXdzIcyGxP7UzEkGYKH9GB2YLC N6hG7m5ZvN4yy1RSkfMqv0ADair6lqfWpBBGNw.RF7aaRMPxKDfuAJ_lN7Ou_.o3NMBNeXmnm5Ur sCSSJFKlFSq7FWuXExk.epMHfHlhUdJE8xFGMvgikr10KXKWGaZIk.wdcSP2cEKZX7foVWRRFonr pa3WWJnahuHFj2LOD_4lXSkJnTiZVSw9ISxxPu0ARP9PJO2Jtk00uYU53DkuFKesP0wuHhuj7V9e zrR0_gk1VwdFJrxWhQcddJjarwC0v4FrB.MJ0Tj4bdHfxIf.eH1uV37Yl0C1MLo10G8XgHn5BBJ7 7_5NfVLMX3S5OvcFl3Ie1kSMXEA76qmVO2xspXRlGT0VR3EhZIhDYnA.P3NOQ0TE19rGl4ZNwyx_ Ay.k6u5WD_aIYL9EeVHfrH7.Z_sJJ907dt7AtMTURaH1aIC17oSjqyMsAb6LHLrxLMv7MUNJDwmc p1eFnrrsGgprV70Omtljal0uw3A1uA9ey2SVrMDSkJpsnogS9ZodM.KqMpJp6hW_mgvwag6BjTu6 jPYFBOsgcreoG.lN2XJlwAFjSjpAaXISE7LbtXKa1LAAitP10ViRv5DKnp8A1pjF9xrtvfVjiIrc CNEGSp7CQJdphAyrJOaj9cOHwOy0WKLuNE_LeVEngIAF.bYyfCn2QjO_AVIbexOhVBGfckuvBNLj 2c5sCZMEcx.61lYldmK35xt0nbl0IlPc7WfhQK_vKddOAOuMkIH63f6lBkC2j2QJwh57T7WJYkZz Biq7WKCG2JaLVSfz0ZuS9ruU8ISfuZCfgeSH6qFHxSXtLc6bbhu0_B8IkXzI_SIYcBSAQ36NgsIN v23xOibh_RiTg5STCN172XTeSZ5_7Jp3diYKcf3fpYU8rIBlwnavyc6R28jgCoKHzQTVqKoKeKDx yp.BlG9k1egtt1FELuVNcmC9VwiASCLG05Yr4NDc_2KlavmD2WU5YOAu5hky4QANF8V2aMjT7qT8 l2b58l3ltoknYffBH4G8vvQpKvzec.YJoPMOZPhmZr5pQpbOOmP_8ShXVNlbr5Yxg9YnPqWOF1Az DI2RojksALWdm9PPGkdnubskddCAJGXkXOAVpXtD6JkVIMUmkmKXt6VLP2tBdOga6gSejR.KC0yx 1Q92okiCDP_AYxDa7VP7Ag5E0zLqwGiSucSTPH0MHR5TJGhZ5uTkUznzBFzOQvJXCUmdcL1akD7P inDC3Xju2Fp.Fl.CepOT0lR7J0iDey.KPEWmk1nvYw2zhNtZO2RrzJgqyik.75TUl.mYQeQPuMdg 0liKrzvcU6fvl4u14dQi8SMXRIDkWuFTQso7f4mForb44llwXZ0_PXabYsFjpkclUQ89nqsZQfKh NPkw3H7Kf67uMtye8toM4c8XTEn7daG1y92ZdYCDVjwL4Iq7fJHylSSeS5BsrjQAId6JzdUq0HI5 gkyNwoN_T4a4RDSNeVM6ReQuFVaQwy72YJ.WSymNFhdz00BssshN4Qzrp_XaxFztBDs7IF3rVhxB JXMTYxRhLFo67eme3XSJK7XPcPed4jqXC1iqu1x8Kyv6T2lnrUv4cRO37vnqgcrQiaGlRFvy4dIX 7St8- X-Sonic-MF: X-Sonic-ID: 3fce1e46-c52c-4f99-ba2a-535d5e50e195 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Fri, 11 Apr 2025 21:04:56 +0000 Received: by hermes--production-gq1-74d64bb7d7-mh87r (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID f7da9cf3db31fc16995632eb8a7c9048; Fri, 11 Apr 2025 21:04:52 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.500.181.1.5\)) Subject: Re: UPDATE: pkg 2.1.0 looks to be making official bulk builds of packages take much longer [reproducible examples notes] From: Mark Millard In-Reply-To: Date: Fri, 11 Apr 2025 14:04:49 -0700 Cc: Gleb Popov <6yearold@gmail.com>, FreeBSD Current , FreeBSD Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <8B070D1D-0524-4DA4-A5C2-EF2CF98C5E15@yahoo.com> References: <8E2FBAD3-EF6F-4D99-A340-21F8FD19AE0F@yahoo.com> <84FBBAF8-025E-4B9D-9797-51735567A8DB@yahoo.com> <366E27FD-FA5B-4BF8-B6C4-6C495DB289C5@yahoo.com> <7ziazrj7szuqhov3oppjbh3jyu3f2p2owntv4oxprelrdjzc6u@hkuf5szf3zwy> To: Baptiste Daroussin X-Mailer: Apple Mail (2.3826.500.181.1.5) X-Spamd-Result: default: False [-4.50 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[98.137.66.147:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from] X-Rspamd-Queue-Id: 4ZZ8Mg13vrz3r7h X-Spamd-Bar: ---- On Apr 11, 2025, at 11:39, Mark Millard wrote: > On Apr 7, 2025, at 08:14, Baptiste Daroussin wrote: >=20 >> . . . >> the problem we have is the >> performance changes depending on what is happening in parallel on the = machines. >=20 > In separate list messages I've provided multiple examples > of the time-taking issue that do not depend on what is > running in parallel on the machines, no parallel builds > involved. >=20 > Part of the issue is that there are thousands of examples of > "small build-step time" packages for which the build-depends, > lib-depends, run-depends combination, takes notable time, > given that the total time contribution across those thousands > of package builds is notable overall. >=20 > As stands, mostly it is the early part of "bulk -c -a" avoids > the issue via building packages that have no or few > dependencies. Later "small build-step time" packages tend to > have various dependencies, greatly changing the time scale > for their builds. Few builds are of "large build-step > time" packages (relative to there being 30000+ packages). That=20 > has implications for there being 30000+ packages to build for > "bulk -c -a" or other builds with large numbers of packages > to try to build. >=20 >> which makes the performance issues invisible on local poudriere if = you want to >> test it on port A or port B, >=20 > I've provided counter examples to that that only involve the > one builder, after the prerequisites have already been built > (same or prior bulk run). >=20 >> if we want to reduce the performance penalty we >> need to be able to make a reproducible case which can then be = profiled, to know >> where to optimize if needed. >=20 > I've provided examples of such . . . > (time intervals shown are from the aarch64 > Windows Dev Kit 2023 with just the one > builder active) >=20 > www/rt50 > build-depends: 00:00:27->00:08:46 >=20 > devel/py-inline-snapshot@py311=20 > build-depends: 00:00:01->00:00:55 > run-depends: 00:00:56->00:01:47 >=20 > mail/mailest@nox > build-depends: 00:00:01->00:00:28 > run-depends: 00:00:30->00:00:59 >=20 > devel/dwarves > build-depends: 00:00:05->00:02:23 > lib-depends: 00:02:23->00:02:42 net-mgmt/fastnetmon build-depends: 00:00:03->00:00:42 lib-depends: 00:00:42->00:01:29 (See later below.) > The timings are from the column next to > the Building/Status/Finished column from > using bulk -v , not from the column for > the overall bulk run. >=20 >> I have tried to reproduce each individual case which happen in the = ports tree >> and I am not able to reproduce them, so impossible to know where to = look at >> exactly. >=20 > Try some of the examples that I've provided? >=20 > There are more examples that I could check > and report non-parallel timings on if you > want. I just picked to report on only a few > initially. >=20 > An example that you might want is my > providing more examples of lib-depends > with non-parallel timings. I took a quick look and quickly ran into: (aarch64 Windows Dev Kit 2023 no-parallel-builders timing again, after having built the prerequisites that had not previously been built) [00:11:37] [01] [00:00:00] Building net-mgmt/fastnetmon | = fastnetmon-1.2.8 [00:11:39] [01] [00:00:02] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: check-sanity [00:11:39] [01] [00:00:02] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: pkg-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: fetch-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: fetch [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: checksum [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: extract-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: extract [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: patch-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: patch [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: build-depends [00:12:19] [01] [00:00:42] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: lib-depends [00:13:06] [01] [00:01:29] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: configure [00:13:09] [01] [00:01:32] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: build [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: run-depends [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: stage [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | = fastnetmon-1.2.8: package [00:14:22] [01] [00:02:45] Finished net-mgmt/fastnetmon | = fastnetmon-1.2.8: Success (I still have thousands of packages that have not built in the bulk -v -a build activity. The M4 MAX is in use for that.) >> I know what is new and what causes the performance penalty, but not >> which part is causing the super extra penalty on the cluster. >=20 > Various examples reproduce the timing issues > outside the cluster and without the parallel > builds. =3D=3D=3D Mark Millard marklmi at yahoo.com