Re: git: 53c1ad8c4add - main - bsd.port.mk: run pkg-create with MAKE_JOB_NUMBER threads
Date: Wed, 04 Mar 2026 20:18:26 UTC
On 2026-03-04 21:00, Siva Mahadevan wrote: > On Wed Mar 4, 2026 at 36:00 UTC, Vladimir Druzenko <vvd@freebsd.org> wrote: >> 04.03.2026 21:53, Daniel Engberg пишет: >>> On 2026-03-04 18:57, Adam Weinberger wrote: >>>> On Wed, Mar 4, 2026 at 9:40 AM Siva Mahadevan <siva@freebsd.org> wrote: >>>>> The branch main has been updated by siva: >>>>> >>>>> URL: >>>>> https://cgit.FreeBSD.org/ports/commit/?id=53c1ad8c4add9656b4ee3e0d1a1c1b643d617f84 >>>>> >>>>> commit 53c1ad8c4add9656b4ee3e0d1a1c1b643d617f84 >>>>> Author: Siva Mahadevan <siva@FreeBSD.org> >>>>> AuthorDate: 2026-03-04 14:39:04 +0000 >>>>> Commit: Siva Mahadevan <siva@FreeBSD.org> >>>>> CommitDate: 2026-03-04 14:39:30 +0000 >>>>> >>>>> bsd.port.mk: run pkg-create with MAKE_JOB_NUMBER threads >>>>> >>>>> Following the same as https://reviews.freebsd.org/D53053 >>>>> in the src tree. >>>>> >>>>> Reviewed by: portmgr (bapt) >>>>> Approved by: lwhsu (mentor, implicitly) >>>>> Differential Revision: https://reviews.freebsd.org/D55232 >>>> Thank you for this! The difference is incredible. On the aging >>>> E3-1270v2 in my rented box, it brought packaging llvm from 20 minutes >>>> down to 10. >>> >>> There are some caveats to take into considering though which should've >>> been documented. >>> https://github.com/facebook/zstd/issues/2079 >>> For xz this breaks reproducible builds, >>> https://www.mail-archive.com/xz-devel@tukaani.org/msg00375.html >>> >>> Best regards, >>> Daniel >> >> And the compression ratio is also reduced. > > From https://github.com/facebook/zstd/issues/2079, > >> The situation is also different when using libzstd's API, as in this >> case, it is single-threaded by default, so the outcome will be >> similar to --single-thread. >> [...] >> Single-thread and multi-thread modes produce different outputs, but >> both are fully deterministic, and the multi-thread mode doesn't care >> about the nb of threads specified. > > As I understand from this, libzstd (which is used by libarchive) will > produce deterministic output for any number of n threads (even 1) as > long as it is explicitly specified. Since MAKE_JOB_NUMBER always > resolves to >=1, this will explicitly pass in >=1 to the threads option. > > From https://facebook.github.io/zstd/zstd_manual.html, > >> When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() >> [...] >> Default value is `0`, aka "single-threaded mode" > > We will always go into the asynchronous mode path as described in issue > 2079. Thus, the way we have it for PKG_COMPRESSION_FORMAT=zstd, this > will not break reproducible builds. I haven't looked into it further yet but the issue above suggests that moving between single and multi within the same version also breaks reproducible builds. > >>> For xz this breaks reproducible builds, >>> https://www.mail-archive.com/xz-devel@tukaani.org/msg00375.html > > I think this is reasonable though and I missed this initially. I will > post a follow-up review that conditionally sets -T only for allowed > PKG_COMPRESSION_FORMAT values (zstd only to start). > >> And the compression ratio is also reduced. > > How much is it reduced by on average? I think this may be relatively > negligible compared to the greatly reduced compression time. And one can > always change the PKG_COMPRESSION_LEVEL if their usecase requires it. It's usually a relatively small difference. Best regards, Daniel