[analysis] Re: 13.2 BETA2: how do debug META_MODE?

It appears as my issue goes away by itself at the fourth subsequent
build.

># cd /usr/src/
># env WITH_META_MODE=yes make buildworld
># env WITH_META_MODE=yes make installworld
># env WITH_META_MODE=yes make buildworld (again #0)
>## no more rebuilds below?
># env WITH_META_MODE=yes make buildworld (again #1)
># env WITH_META_MODE=yes make buildworld (again #2)
>
>"again #0" will rebuild llvm/clang. The other two "again"s
>will not.

Strangely, I observed rather the opposite: the issue was with those
nodes that do *not* get installed. Those that do get installed, they
behaved as expected.

Details:

When starting a build, there are some programs in
/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ from the last build.
These might be suitable for the new build (or they might not, who
knows) - but then, very early in the buildworld, many of these files
are copied from the running base system, with their mtime preserved.
So now these files do not have an mtime of the last build, but of the
last *install* of this running instance. (And that may be too late.)
This is done here - and I have no idea how these files are selected:

> --------------------------------------------------------------
> >>> Rebuilding the temporary build tree
> --------------------------------------------------------------
[...]
> Building /usr/obj/usr/src/amd64.amd64/tools/build/host-symlinks
> Linking host tools into /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin

Over all, /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ tends to become
a zoo: some files are copied from the base system, some are installed
during the build, some are not replaced at all.

I tried to figure what exactly is happening, with the example of
the make-roken + roken.h.
I start the upgrade to 13.2 with an empty obj tree (because after
switching and rebasing branches, my mtimes are recreated from the
latest commit - and they will certainly not align with the old ones).

In the first build the binaries in the obj tree have to be created.
But since the running system is still 13.1, these binaries get all
stamped for 13.1.

In the second build -now running 13.2- META_MODE decides that the
mtimes are now fine and does not rebuild.
META_MODE has no notion of the currently running OS version, it only
considers mtimes and cannot detect that these dependencies are stamped
with an old OS version.

In the third build META_MODE finds some mtimes obsolete again, and
does rebuild toolchain binaries. This would not have any effect
because "install" would not copy them to the bin-dir if the same
binaries are already there. But in this scenario the old binaries
are from 13.1, so they are different, and "install" copies the new
ones in - and now everything that somehow depends on them will also
rebuild.

Finally in the fourth build everything appears to be fine.

Conclusion:
For a safe upgrade (specifically for a major version change) it is
not so much necessary to delete the obj tree before the first build,
but rather *after* the first build, i.e. after the base system has
been upgraded.

Some timings:

The base gets installed into a clean DESTDIR and used for the next
pass. The obj-trees are individually kept.

Initial (obj trees deleted)
( 4 vcore)
230217231308.base.pass1.sst:     real 9339.64	base w/ kernels
230217231308.base.pass2.sst:     real 7982.69
230217231308.admn.pass1.jail.sst:real 9346.90	jail w/ compiler
230217231308.admn.pass2.jail.sst:real 5460.16
230217231308.data.pass1.jail.sst:real 4094.04	jail w/o compiler
230217231308.data.pass2.jail.sst:real 143.39
230217231308.iamk.pass1.jail.sst:real 8050.27	jail w/ compiler
230217231308.iamk.pass2.jail.sst:real 5226.32
230217231308.oper.pass1.jail.sst:real 2910.28	jail w/o compiler
230217231308.oper.pass2.jail.sst:real 92.05
230217231308.rail.pass1.jail.sst:real 3236.29	jail w/o compiler
230217231308.rail.pass2.jail.sst:real 99.49
230217231308.tele.pass1.jail.sst:real 3170.34	jail w/o compiler
230217231308.tele.pass2.jail.sst:real 180.65

pass3
(10 vcore)
230222000242.base.std.sst:     real 1162.80	base w/ kernels
230222000242.admn.std.jail.sst:real 1759.15	jail w/ compiler
230222000242.data.std.jail.sst:real 155.54	jail w/o compiler
230222000242.iamk.std.jail.sst:real 1715.07	jail w/ compiler
230222000242.oper.std.jail.sst:real 149.51	jail w/o compiler
230222000242.rail.std.jail.sst:real 151.73	jail w/o compiler
230222000242.tele.std.jail.sst:real 150.52	jail w/o compiler

pass4
(10 vcore)
230222021535.edge.std.sst:     real 1018.79	base w/ kernels
230222021535.admn.std.jail.sst:real 101.61	jail w/ compiler
230222021535.data.std.jail.sst:real 67.47	jail w/o compiler
230222021535.iamk.std.jail.sst:real 100.91	jail w/ compiler
230222021535.oper.std.jail.sst:real 66.52	jail w/o compiler
230222021535.rail.std.jail.sst:real 68.00	jail w/o compiler
230222021535.tele.std.jail.sst:real 66.54	jail w/o compiler