Re: Switching release media dist sets to .tzst (tar + zstd)?

From: Shawn Webb <shawn.webb_at_hardenedbsd.org>
Date: Fri, 13 Dec 2024 21:47:53 UTC
Hey Ed,

Thanks for providing the opportunity to discuss this before landing
it.

On Fri, Dec 13, 2024 at 04:15:45PM -0500, Ed Maste wrote:
> I have been reviewing parts of the release artifact build process,
> including ISO and memstick images, and came across the distribution
> sets (e.g., base.txz, src.txz) used by the installer to populate new
> file systems. I’d like to discuss switching these to .tzst (tar +
> zstd) compression.
> 
> While I haven’t yet conducted detailed benchmarks comparing zstd and
> xz specifically for this use case, here are some initial
> considerations:
> 
> Pros of zstd:
> - Faster compression and decompression speeds.
> - Aligns with the compression method used for FreeBSD packages.
> 
> Cons of zstd:
> - Somewhat larger compressed file sizes.
> - Requires updates to tools that interact with distribution sets.
> - May have limited availability on some other operating systems (?).

The tool for updating HardenedBSD installs (and the tool used to build
the update artifacts) would be impacted. It wouldn't be too difficult
to update the tools (hbsd-update and hbsd-update-build). However, if
the switch zstd is not done at the same time for all supported
branches (main and stable/14), we would need to have hbsd-update
reference different archives between different branches--zstd for
main and xz for stable/14. I would prefer not to have to include
branch-specific code in a generic system updater utility.

> 
> I have a review open to demonstrate the extent of the change in the
> build system & installer: https://reviews.freebsd.org/D48042

One thought might be to make the choice of compression method dynamic.
Folks could then choose what makes sense for them. FreeBSD could make
the switch to zstd while downstreams could still use xz (should they
so choose.) HardenedBSD would likely stay on xz until it makes sense
to follow its upstream.

> 
> It might be that this is not worth pursuing, as dist sets will most
> likely go away with the migration to pkgbase, but I would like to
> discuss and make an explicit decision. We can separately consider
> compression on the release media images themselves.
> 
> Feedback Requested:
> 
> Is there support for this idea? Are there objections to pursuing this?
> Are there other factors I should consider, especially compatibility concerns?

For reference, hbsd-update can be found at [1] and hbsd-update-build
can be found at [2].

[1]:
https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current/master/usr.sbin/hbsd-update/hbsd-update?ref_type=heads
[2]:
https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current/master/usr.sbin/hbsd-update/hbsd-update-build?ref_type=heads

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc