From nobody Sun Mar 08 05:13:49 2026 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fT7bP0J40z6V2Yn for ; Sun, 08 Mar 2026 05:13:53 +0000 (UTC) (envelope-from agh@riseup.net) Received: from mx0.riseup.net (mx0.riseup.net [198.252.153.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx0.riseup.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fT7bM0qNwz3xwZ for ; Sun, 08 Mar 2026 05:13:51 +0000 (UTC) (envelope-from agh@riseup.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=riseup.net header.s=squak header.b=sq3yDD2X; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (mx1.freebsd.org: domain of agh@riseup.net designates 198.252.153.6 as permitted sender) smtp.mailfrom=agh@riseup.net Received: from fews03-sea.riseup.net (fews03-sea-pn.riseup.net [10.0.1.153]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx0.riseup.net (Postfix) with ESMTPS id 4fT7bK3S5Dz9vBN for ; Sun, 8 Mar 2026 05:13:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=riseup.net; s=squak; t=1772946829; bh=dxMPFKjtepcSDwCg2PK56x7xVOK1N9qkCaMSBooUPAs=; h=Date:From:To:Subject:In-Reply-To:References:From; b=sq3yDD2Xs82f+dmaEduri4Gn8UOxAzTSDO+0YyWENnF3xOoZa9Dxy3osUq2NOO8lk llKzRCnmYIi9nlcOZoA2IIWFPbmeqXVO/XG/j9pLsHz9o9ilV6KmAzJjw3lKJkLrvA MfCEmPLm0RbdBP6958EEnu139V734yolTP4hf2Jk= X-Riseup-User-ID: F4C3DFF297EE230DB0E2972FE409BED45054EEE34D786A88BA58A5EACEF5CED7 Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews03-sea.riseup.net (Postfix) with ESMTPSA id 4fT7bK1ylLz24qx for ; Sun, 8 Mar 2026 05:13:49 +0000 (UTC) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Date: Sun, 08 Mar 2026 05:13:49 +0000 From: Alastair Hogge To: freebsd-current@freebsd.org Subject: Re: pkgbase and customised builds via ${SRC}/release/release.sh In-Reply-To: References: <20fb908c0954e62977dffdcd4a883678@riseup.net> Message-ID: Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.10 / 15.00]; SUBJECT_HAS_CURRENCY(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[riseup.net,none]; R_DKIM_ALLOW(-0.20)[riseup.net:s=squak]; R_SPF_ALLOW(-0.20)[+a:mx0.riseup.net]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[198.252.153.6:from]; DKIM_TRACE(0.00)[riseup.net:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RECEIVED_HELO_LOCALHOST(0.00)[]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:16652, ipnet:198.252.153.0/24, country:US]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[riseup.net:dkim] X-Rspamd-Queue-Id: 4fT7bM0qNwz3xwZ X-Spamd-Bar: --- On 2026-02-28 19:55, Lexi Winter wrote: Hey Lexi, > Alastair Hogge wrote in : >> The biggest problem I had come across, was the "tags=package={foo}" >> declarations in ${SRC}/etc/mtree/*.dist. I had to remove a couple of >> these from a 2 or 3 mtree dists. What is going on in this part of the >> build infrastructure? There is a history of similar commits[1][2][3], to >> the changes I now have to keep locally, yet, des@ recently added[4] more >> tags. > > short answer: as of last week, des's commit is correct and the previous > commits removing the tags should be reverted, i just didn't get around > to doing it yet. > > the problem is that previously, tags=package=foo would cause the 'foo' > package to be unconditionally created, even if it was otherwise empty. > this first came to light when i introduced several of these in commit > 436618a427b4[0], which created several of these empty packages in > specific circumstances i hadn't tested (mostly involving non-amd64 > builds and certain src.conf options). > > [0] https://cgit.freebsd.org/src/commit/?id=436618a427b4baaf42d8221ef07d14e3ba787d3a > "etc/mtree: Add package tags for /usr/include" > > in at least a couple of cases this broke the build or install because > we'd end up with a 'foo-dev' package having a depend on a 'foo' package > that doesn't exist. the post-436618a427b4 reverts are to fix specific > instances of this without having to revert the entire initial commit > which was broadly correct. > > this was on my to-do list for a long time until it came up again in > D55408[1], the review for des's commit that you mentioned, at which > point i fixed the problem properly in 7965c93e4d41[2] by not creating > packages which only contain directories. this also fixes all the > previous examples of this problem, which is why the workarounds can > now be reverted. > > [1] https://reviews.freebsd.org/D55408 > "build: Move all of lp under LPR option" > > [2] https://cgit.freebsd.org/src/commit/?id=7965c93e4d4103ba6ed7ac1e5f1599c93cbcdbf7 > "packages: Don't create empty packages" > > if you're on stable/15 (or 15.0), i didn't yet MFC 7965c93e4d41, but > it's fairly self-contained and should be fine to cherry-pick. if > you're rather wait, it will land in 15.1. > >> From what I understand when investigating this, the build infra uses >> both the mtree dists, and ${SRC}/release/packages/generate-set-ucl.lua >> to generate the pkgbase sets? Is that correct? Does that mean there is >> duplicated work here, and one of these paths is redundant? > > pkgbase *sets* are generated purely based on annotations (a type of > metadata) in the packages we previously built, i.e., we do the full > pkgbase build, then scan that for 'set' annotations and use that to > generate the set packages. this is driven by > release/packages/create-sets.sh, which invokes generate-set-ucl.lua > as part of its work. but note that this is only about set packages > (i.e., "FreeBSD-set-*"), which never contain any files or directories > and should never be referenced in mtree. > > mtree is involved in the staging process we do prior to the initial > (non-set) pkgbase build. that process is driven by Makefile.inc1, > and looks roughly like this: > > * a 'make stageworld' is done to populate /worldstage. > stageworld is essentially the same thing as installworld, with > some minor tweaks to prevent the host system state from affecting > the installed world. > > part of this process involves running mtree to create the directory > structure. > > this stageworld is done with NO_ROOT and METALOG set, which means > a complete log of everything installed (including by mtree) is > written to a special file called METALOG in the worldstage/ > directory. > > * then we run release/scripts/mtree-to-plist.awk on the METALOG > file to generate the individual *.plist files for each package. > this looks at the 'package' tag for each file or directory in the > METALOG to determine which package the file belongs to. > > despite the name, this script is never run on mtree files from src. > it's called mtree-to-plist because the METALOG is in mtree format. > > * based on the plist files, we run release/packages/generate-ucl.sh > to generate a UCL file containing pkg-create(8) metadata for > each package. > > * we build the packages and copy them to ${REPODIR}. Thanks for the explaining that, I feel it might take sometime to sink in. > i find this process both overcomplicated and somewhat fragile, > so i'd like to improve it at some point, but there are more urgent > things to fix like release media builds and some kernel-related > issues (e.g. D54542[4] and D54282[5], if you're interested). Same, however, ${SRC}/release/release.sh is a powerful device, enabling users to build customised appliances/hosts in isolation. > [4] https://reviews.freebsd.org/D54542 > "release: Build the release media from packages" > > [5] https://reviews.freebsd.org/D54282 > "packages: Always install kernel as /boot/kernel.NAME" Something I have wanted for decades, thanks, this is most excellent, and going to be a great help. I am behind on 16-CURRENT at the moment, when I get around to sycn'ing, I will include any remaining Reviews you have shared, and report there. >> I also noted that a few WITHOUT_foo options had no effect on the World >> building stage 1.2, bootstrap tools. Here, when WITHOUT_KERBEROS is >> defined, it is still built. I even still have a Kerberos KDC pkgbase >> component. > > i am not sure what's going on here since i'm not very familiar with how > the src bootstrap works. if no one else has an idea, you could ask on > hackers@ (or in a new thread, since people might ignore pkgbase threads.) Will do.