Re: git: 53c1ad8c4add - main - bsd.port.mk: run pkg-create with MAKE_JOB_NUMBER threads
Date: Wed, 04 Mar 2026 20:00:34 UTC
On Wed Mar 4, 2026 at 36:00 UTC, Vladimir Druzenko <vvd@freebsd.org> wrote: > 04.03.2026 21:53, Daniel Engberg пишет: >> On 2026-03-04 18:57, Adam Weinberger wrote: >>> On Wed, Mar 4, 2026 at 9:40 AM Siva Mahadevan <siva@freebsd.org> wrote: >>>> The branch main has been updated by siva: >>>> >>>> URL: >>>> https://cgit.FreeBSD.org/ports/commit/?id=53c1ad8c4add9656b4ee3e0d1a1c1b643d617f84 >>>> >>>> commit 53c1ad8c4add9656b4ee3e0d1a1c1b643d617f84 >>>> Author: Siva Mahadevan <siva@FreeBSD.org> >>>> AuthorDate: 2026-03-04 14:39:04 +0000 >>>> Commit: Siva Mahadevan <siva@FreeBSD.org> >>>> CommitDate: 2026-03-04 14:39:30 +0000 >>>> >>>> bsd.port.mk: run pkg-create with MAKE_JOB_NUMBER threads >>>> >>>> Following the same as https://reviews.freebsd.org/D53053 >>>> in the src tree. >>>> >>>> Reviewed by: portmgr (bapt) >>>> Approved by: lwhsu (mentor, implicitly) >>>> Differential Revision: https://reviews.freebsd.org/D55232 >>> Thank you for this! The difference is incredible. On the aging >>> E3-1270v2 in my rented box, it brought packaging llvm from 20 minutes >>> down to 10. >> >> There are some caveats to take into considering though which should've >> been documented. >> https://github.com/facebook/zstd/issues/2079 >> For xz this breaks reproducible builds, >> https://www.mail-archive.com/xz-devel@tukaani.org/msg00375.html >> >> Best regards, >> Daniel > > And the compression ratio is also reduced. From https://github.com/facebook/zstd/issues/2079, > The situation is also different when using libzstd's API, as in this > case, it is single-threaded by default, so the outcome will be > similar to --single-thread. > [...] > Single-thread and multi-thread modes produce different outputs, but > both are fully deterministic, and the multi-thread mode doesn't care > about the nb of threads specified. As I understand from this, libzstd (which is used by libarchive) will produce deterministic output for any number of n threads (even 1) as long as it is explicitly specified. Since MAKE_JOB_NUMBER always resolves to >=1, this will explicitly pass in >=1 to the threads option. From https://facebook.github.io/zstd/zstd_manual.html, > When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() > [...] > Default value is `0`, aka "single-threaded mode" We will always go into the asynchronous mode path as described in issue 2079. Thus, the way we have it for PKG_COMPRESSION_FORMAT=zstd, this will not break reproducible builds. >> For xz this breaks reproducible builds, >> https://www.mail-archive.com/xz-devel@tukaani.org/msg00375.html I think this is reasonable though and I missed this initially. I will post a follow-up review that conditionally sets -T only for allowed PKG_COMPRESSION_FORMAT values (zstd only to start). > And the compression ratio is also reduced. How much is it reduced by on average? I think this may be relatively negligible compared to the greatly reduced compression time. And one can always change the PKG_COMPRESSION_LEVEL if their usecase requires it.