From nobody Thu Oct 16 20:08:18 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cnfCQ5Mkyz6D4hL; Thu, 16 Oct 2025 20:08:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cnfCQ4rTWz3V3x; Thu, 16 Oct 2025 20:08:18 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760645298; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Nf/rB1nYCF6Qmv7O6VaazoRcxH+Pb4JBRQaLL1FAM4U=; b=m/OGXEa7hPjnMso1mwmP2/g/3SJGb+8vKN2GeyKBNqe8wES5eMpwn5LMbUi6jp+NVaR5j7 ThX5lAGxGXoF3iIj2A0wzhi3tB7t0JnmKrkmCR6jgBqdwc1bieNGqZlMKPO410zAfO8SJF bgcYtHcb+Z0bnn22csw7GHJEdT64v1X9EaG0Wta5ZEUjobcXltiVCS+pM5Yza+ObVNw7vl uCKFRgr/fIiYeKj8dZYH0z/FSLvxhdEl9fjzVBhGfpgLSY1Zsi2t3cs8hKbbCNDqLtsr3E dL3NlXcCZUV/0VqNqXJxeS0TCLWcFW6GsXT7wkUme2IEkAWJOoJaiLaOWSE7bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760645298; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Nf/rB1nYCF6Qmv7O6VaazoRcxH+Pb4JBRQaLL1FAM4U=; b=P5uRnsTqydV+qmcpJ3WStqQJZeNl5k1UMs1HTKOJNp2p3NYZtR15Y2Mj70Y/SbsI12ppbN Qrwma9M0vSL0A9XadmvhnKSiGQPv/LZKwQVcnFRmwzdhj6LtqhbM40kfMPKQinnL/a5GxN bp7TgwCZeJXGjh1O8raEh5KSFbfzX7CCuH251KnpoXa0emYbGsIDQQ4npaKQ/odjfvLYy5 dMHa2V2r9w+diqJpnxCeRyIm4hidzJ/li9cfA72ebo4GVNoQuDkCTFz4KdRHx3wPGeP+oe oE4wc7q98woeHD6vbe4jf3HYNx0S6bvW+24DoeXjWDkYXJKufWQvrNIBIqfrVA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1760645298; a=rsa-sha256; cv=none; b=SRIsdaiHRtaxuvNHWiEgfS8NmB5jFlRutMTm9mvBfszg2s8z0SUy8Jt2saddAt57OWdDKo XaJHaBdbTbSaIb/m1OccIFvqM6LA0z6TOj58Y/Tv01j4aC/fs4LMC513lr6PPqMjyzIa5a uOd2/mxyP1hirPTw0AvXsBDPiqW8ssHvXY39TY27RORhcYVu0q27/c8kFnyfraCCCXpD2N fddMftsikfRs2QasLKYpxUuEHyyYtj96hUlwGW7qE0N/6WM+rxpschUR5N7ZUCzTGF0Q5B 4QiPl+EtfqpVBt8+M2c6CzJsdU3nds9BlaNM/oYMWcjJ8FnuT6NY93314DzDew== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cnfCQ4Gv8z71G; Thu, 16 Oct 2025 20:08:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 59GK8IoU018308; Thu, 16 Oct 2025 20:08:18 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 59GK8IdF018305; Thu, 16 Oct 2025 20:08:18 GMT (envelope-from git) Date: Thu, 16 Oct 2025 20:08:18 GMT Message-Id: <202510162008.59GK8IdF018305@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Colin Percival Subject: git: 8f5eb720e2e6 - releng/15.0 - packages: Compress in parallel List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: cperciva X-Git-Repository: src X-Git-Refname: refs/heads/releng/15.0 X-Git-Reftype: branch X-Git-Commit: 8f5eb720e2e67446fc9c10ab61d0069c356cd73b Auto-Submitted: auto-generated The branch releng/15.0 has been updated by cperciva: URL: https://cgit.FreeBSD.org/src/commit/?id=8f5eb720e2e67446fc9c10ab61d0069c356cd73b commit 8f5eb720e2e67446fc9c10ab61d0069c356cd73b Author: Colin Percival AuthorDate: 2025-10-12 21:44:14 +0000 Commit: Colin Percival CommitDate: 2025-10-16 20:08:05 +0000 packages: Compress in parallel The FreeBSD-base pkg repository is currrently dominated by a handful of large packages: Out of a compressed repository size of ~960 MB, about 160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by the FreeBSD-src-sys package. Consequently, running 'make packages -jN' provides less benefit than one might hope, as most of the packages finish building quickly, ultimately leaving the FreeBSD-src package building by itself for a couple minutes while all the other CPUs are idle. Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default of -T0) in order to instruct pkg's zstd compression to use multiple threads. Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the time taken by 'make packages' from 6m17s to 1m39s; package creation time itself (excluding the initial process of installing into world and kernel staging directories) dropped from 5m37s to 59s. Approved by: re (cperciva) Reviewed by: ivy MFC after: 3 days Sponsored by: https://www.patreon.com/cperciva Differential Revision: https://reviews.freebsd.org/D53053 (cherry picked from commit 16155cccc4bf42a5a76806128e7ef637f9ae61fe) (cherry picked from commit 3d8ec3273dbb9edf9080ded97e071edff59dd708) --- Makefile.inc1 | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/Makefile.inc1 b/Makefile.inc1 index d530ca4e9584..21adf215c0aa 100644 --- a/Makefile.inc1 +++ b/Makefile.inc1 @@ -1977,6 +1977,7 @@ REPODIR?= ${OBJROOT}repo PKG_FORMAT?= tzst PKG_LEVEL?= -1 PKG_CLEVEL?= ${"${PKG_FORMAT:Mtar}" != "":?:-l ${PKG_LEVEL}} +PKG_CTHREADS?= 0 PKG_REPO_SIGNING_KEY?= # empty PKG_OUTPUT_DIR?= ${PKG_VERSION} PKG_ABI_FILE?= ${WSTAGEDIR}/usr/bin/uname @@ -2157,7 +2158,7 @@ create-source-src-package: _pkgbootstrap .PHONY ${SSTAGEDIR}/src.ucl ${PKG_CMD} -o ABI=${PKG_ABI} \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${SSTAGEDIR}/src.ucl \ -p ${SSTAGEDIR}/src.plist \ -r ${SRCDIR} \ @@ -2183,7 +2184,7 @@ create-source-src-sys-package: _pkgbootstrap .PHONY ${SSTAGEDIR}/src-sys.ucl ${PKG_CMD} -o ABI=${PKG_ABI} \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${SSTAGEDIR}/src-sys.ucl \ -p ${SSTAGEDIR}/src-sys.plist \ -r ${SRCDIR} \ @@ -2226,7 +2227,7 @@ create-world-package-${pkgname}: .PHONY fi ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${WSTAGEDIR}/${pkgname}.ucl \ -p ${WSTAGEDIR}/${pkgname}.plist \ -r ${WSTAGEDIR} \ @@ -2245,7 +2246,7 @@ create-sets-packages: .PHONY @for manifest in ${WSTAGEDIR}/set-*.ucl; do \ echo "--> Processing manifest: $$manifest"; \ ${PKG_CMD} -o ABI=${PKG_ABI} -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M $$manifest \ -o "${REPODIR}/${PKG_ABI}/${PKG_OUTPUT_DIR}" \ || exit 1; \ @@ -2275,7 +2276,7 @@ create-dtb-package: .PHONY ${KSTAGEDIR}/${DISTDIR}/dtb.ucl ; \ ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${KSTAGEDIR}/${DISTDIR}/dtb.ucl \ -p ${KSTAGEDIR}/${DISTDIR}/dtb.plist \ -r ${KSTAGEDIR}/${DISTDIR} \ @@ -2312,7 +2313,7 @@ create-kernel-packages-flavor${flavor:C,^""$,${_default_flavor},}: _pkgbootstrap ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl ; \ ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl \ -p ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.plist \ -r ${KSTAGEDIR}/${DISTDIR} \ @@ -2352,7 +2353,7 @@ create-kernel-packages-extra-flavor${flavor:C,^""$,${_default_flavor},}-${_kerne ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl ; \ ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ -o OSVERSION="${SRCRELDATE}" \ - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ -M ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl \ -p ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.plist \ -r ${KSTAGEDIR}/kernel.${_kernel} \