From nobody Wed Sep 29 08:42:46 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4BA461788DFB for ; Wed, 29 Sep 2021 08:42:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HK8yj1W5fz4mrn for ; Wed, 29 Sep 2021 08:42:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 1319227D3D for ; Wed, 29 Sep 2021 08:42:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from [192.168.1.202] (host86-148-130-214.range86-148.btcentralplus.com [86.148.130.214]) by smtp.theravensnest.org (Postfix) with ESMTPSA id 94C362CDE2 for ; Wed, 29 Sep 2021 09:42:51 +0100 (BST) Subject: Re: FreeBSD base pkg (packaging) and critical ports build alongside To: freebsd-current@freebsd.org References: <20210929102805.1ac3a59d@freyja> From: David Chisnall Message-ID: <74afb0a9-be7f-c797-bd15-a96022188d9a@FreeBSD.org> Date: Wed, 29 Sep 2021 09:42:46 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 In-Reply-To: <20210929102805.1ac3a59d@freyja> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Hi, I think your best option would be to do the opposite of what you suggest. Poudriere can build pkgbase sets from a source tree and populate a jail from them. The flow that I'd suggest is: - Poudriere jail to build a jail from an existing source tree. - If there are kernel changes, install the packages on the package builder and reboot. - Poudriere bulk in the new jail to build the new package set. Note: You can *normally* skip the second step (drm ports, for example, will be built against the new kernel sources in the jail, though they might not be loadable on the host) but there's no guarantee that you can run a newer userland on an older kernel so things may break. If you enable reproduceable builds in the src.conf that you use for building the jail then you should be able to just diff the kernel binary to see if anything has changed. If you have bhyve or are running on a cloud platform then you can replace the second step with a poudriere image invocation to build a VM image containing poudriere and your newly-built base system and deploy this to build the packages. I'm planning on working on some tooling to do this in Azure with GitHub Actions. Note that poudriere uses packages installed on the host system to build a jail. If you have, for example, installed llvm12 then you can put a line in your src-env.conf for the jail to tell it to use that as an external toolchain and skip the toolchain-bootstrap phase of the build. This means that the base-build is fairly fast even on quite modest hardware (it still builds clang, but at least it does it only once). David On 29/09/2021 09:28, FreeBSD User wrote: > Hello, > > I use FreeBSD-base packages built on self hosted systems to update 13-STABLE > and CURRENT hosts. I run into the problem, that the packages of the FreeBSD > base, built via the FreeBSD framework and from most recent 13-STABLE sources, > are often oit of synchronisation with our poudriere packaging builders, that is > especially true for critical ports with kernel modules, like i915 drm, > virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE sources > and probably the API changes more rapidly than those of the appropriate builder > hosts for poudriere and since it takes a bunch of days to build a whole > poudriere packages repository, there is often a gap between the revision of the > kernel and the port containing kernel modules. > > So, the question is: how can I add ports to the building process of the FreeBSD > sources tree in the way they get build every time I build the FreeBSD-base > packages alongside the OS? > > Thanks in advance, > > oh >