A recent time problem for the ports-package builders for main-amd64 [beefy18] and main-* generally

[Hopefully my classification of the below as a
freebsd-arch issue is reasonable.]

My notes below are also on discord in the project
infrastruture area --but with a plot that has
essential information. The message sequence starts
with:

https://discord.com/channels/727023752348434432/1250652091702050826/1392189829839847495


The text without the plot:

For poudriere bulk runs that rebuild most everything,
pkg 2.2.1's involvement is a small time improvement
for the likes of main-amd64 (and likely all th
main-*) compared to pkg 2.1.2 --but is vastly worse
compared to pkg 2.0.6 or before. It is a much bigger
improvement for the release-based build machines.
My guess is that the debug vs. non-debug kernel
usage is heavily involved in the distinctions:
pkg 2.1+ is doing something that is rather time
expensive in the debug kernels is my guess.

PLOT OF beefy18 main-aamd64 EXAMPLES GOES HERE:
Elapsed Hrs vs. pkgs built

I would claim that the concave-up structure can not
survive long term as more and more package builds
are made in future years.

A basic problem is that packages that have any
build/lib/run dependencies and were built toward
the right of that graph have a
build-depends/lib-depends/run-depends stage effort
that involves all the previously built packages (not
just the ones it is actually dependent on). The
thousands of not large packages spend lots of (new)
time in the depend stages that install packages,
especially ones that have lots of dependencies to
install in the builder. On beefy18 for the above,
the last package to finish, gitlab-ce-18.1.1,
spent over 3 hrs in the build-depends stage.

When beefy17 stops being used for main-i386, I
wonder if the debug-kernel based testing of
building main-amd64 packages and the
for-distribution building of main-amd64's packages
(with a non-debug kernel in use) could be done
separately on separate machines.

This would not help for the slower environment:
ampere2 for main-arm64 and main-armv7. My guess
is that ampere2 will take about a month to cycle
through both main-arm64 and main-armv7 to start
over (when both are similar in scale to full
builds of all the buildable packages). That is so
long between updates for, say, main-arm64, that
each update is likely rather large and may take a
similar sort of time frame. (pkg 2.1.* being
involved lead to over a month between, say,
main-arm64 updates.)

===
Mark Millard
marklmi at yahoo.com