Re: pkgbase and customised builds via ${SRC}/release/release.sh

On 2026-02-28 19:55, Lexi Winter wrote:

Hey Lexi,

> Alastair Hogge wrote in <bff85ee4a5836465e4199e41ec1b543e@riseup.net>:
>> The biggest problem I had come across, was the "tags=package={foo}"
>> declarations in ${SRC}/etc/mtree/*.dist. I had to remove a couple of
>> these from a 2 or 3 mtree dists. What is going on in this part of the
>> build infrastructure? There is a history of similar commits[1][2][3], to
>> the changes I now have to keep locally, yet, des@ recently added[4] more
>> tags.
>  
> short answer: as of last week, des's commit is correct and the previous
> commits removing the tags should be reverted, i just didn't get around
> to doing it yet.
> 
> the problem is that previously, tags=package=foo would cause the 'foo'
> package to be unconditionally created, even if it was otherwise empty.
> this first came to light when i introduced several of these in commit
> 436618a427b4[0], which created several of these empty packages in
> specific circumstances i hadn't tested (mostly involving non-amd64
> builds and certain src.conf options).
> 
> [0] https://cgit.freebsd.org/src/commit/?id=436618a427b4baaf42d8221ef07d14e3ba787d3a
>     "etc/mtree: Add package tags for /usr/include"
> 
> in at least a couple of cases this broke the build or install because
> we'd end up with a 'foo-dev' package having a depend on a 'foo' package
> that doesn't exist.  the post-436618a427b4 reverts are to fix specific
> instances of this without having to revert the entire initial commit
> which was broadly correct.
> 
> this was on my to-do list for a long time until it came up again in
> D55408[1], the review for des's commit that you mentioned, at which
> point i fixed the problem properly in 7965c93e4d41[2] by not creating
> packages which only contain directories.  this also fixes all the
> previous examples of this problem, which is why the workarounds can
> now be reverted.
> 
> [1] https://reviews.freebsd.org/D55408
>     "build: Move all of lp under LPR option"
> 
> [2] https://cgit.freebsd.org/src/commit/?id=7965c93e4d4103ba6ed7ac1e5f1599c93cbcdbf7
>     "packages: Don't create empty packages"
> 
> if you're on stable/15 (or 15.0), i didn't yet MFC 7965c93e4d41, but
> it's fairly self-contained and should be fine to cherry-pick.  if
> you're rather wait, it will land in 15.1.
> 
>> From what I understand when investigating this, the build infra uses
>> both the mtree dists, and ${SRC}/release/packages/generate-set-ucl.lua
>> to generate the pkgbase sets? Is that correct? Does that mean there is
>> duplicated work here, and one of these paths is redundant?
> 
> pkgbase *sets* are generated purely based on annotations (a type of
> metadata) in the packages we previously built, i.e., we do the full
> pkgbase build, then scan that for 'set' annotations and use that to
> generate the set packages.  this is driven by
> release/packages/create-sets.sh, which invokes generate-set-ucl.lua
> as part of its work.  but note that this is only about set packages
> (i.e., "FreeBSD-set-*"), which never contain any files or directories
> and should never be referenced in mtree.
> 
> mtree is involved in the staging process we do prior to the initial
> (non-set) pkgbase build.  that process is driven by Makefile.inc1, 
> and looks roughly like this:
> 
> * a 'make stageworld' is done to populate <objdir>/worldstage.
>   stageworld is essentially the same thing as installworld, with
>   some minor tweaks to prevent the host system state from affecting
>   the installed world.
> 
>   part of this process involves running mtree to create the directory
>   structure.
> 
>   this stageworld is done with NO_ROOT and METALOG set, which means
>   a complete log of everything installed (including by mtree) is
>   written to a special file called METALOG in the worldstage/
>   directory.
> 
> * then we run release/scripts/mtree-to-plist.awk on the METALOG
>   file to generate the individual *.plist files for each package.
>   this looks at the 'package' tag for each file or directory in the
>   METALOG to determine which package the file belongs to.
> 
>   despite the name, this script is never run on mtree files from src.
>   it's called mtree-to-plist because the METALOG is in mtree format.
> 
> * based on the plist files, we run release/packages/generate-ucl.sh
>   to generate a UCL file containing pkg-create(8) metadata for
>   each package.
> 
> * we build the packages and copy them to ${REPODIR}.

Thanks for the explaining that, I feel it might take sometime to sink
in.

> i find this process both overcomplicated and somewhat fragile,
> so i'd like to improve it at some point, but there are more urgent
> things to fix like release media builds and some kernel-related
> issues (e.g. D54542[4] and D54282[5], if you're interested).

Same, however, ${SRC}/release/release.sh is a powerful device, enabling
users to build customised appliances/hosts in isolation.

> [4] https://reviews.freebsd.org/D54542
>     "release: Build the release media from packages"
> 
> [5] https://reviews.freebsd.org/D54282
>     "packages: Always install kernel as /boot/kernel.NAME"

Something I have wanted for decades, thanks, this is most excellent, and
going to be a great help.

I am behind on 16-CURRENT at the moment, when I get around to sycn'ing,
I will include any remaining Reviews you have shared, and report there.

>> I also noted that a few WITHOUT_foo options had no effect on the World
>> building stage 1.2, bootstrap tools. Here, when WITHOUT_KERBEROS is
>> defined, it is still built. I even still have a Kerberos KDC pkgbase
>> component.
> 
> i am not sure what's going on here since i'm not very familiar with how
> the src bootstrap works.   if no one else has an idea, you could ask on
> hackers@ (or in a new thread, since people might ignore pkgbase threads.)

Will do.