Re: Switching release media dist sets to .tzst (tar + zstd)?
Date: Fri, 13 Dec 2024 21:47:53 UTC
Hey Ed, Thanks for providing the opportunity to discuss this before landing it. On Fri, Dec 13, 2024 at 04:15:45PM -0500, Ed Maste wrote: > I have been reviewing parts of the release artifact build process, > including ISO and memstick images, and came across the distribution > sets (e.g., base.txz, src.txz) used by the installer to populate new > file systems. I’d like to discuss switching these to .tzst (tar + > zstd) compression. > > While I haven’t yet conducted detailed benchmarks comparing zstd and > xz specifically for this use case, here are some initial > considerations: > > Pros of zstd: > - Faster compression and decompression speeds. > - Aligns with the compression method used for FreeBSD packages. > > Cons of zstd: > - Somewhat larger compressed file sizes. > - Requires updates to tools that interact with distribution sets. > - May have limited availability on some other operating systems (?). The tool for updating HardenedBSD installs (and the tool used to build the update artifacts) would be impacted. It wouldn't be too difficult to update the tools (hbsd-update and hbsd-update-build). However, if the switch zstd is not done at the same time for all supported branches (main and stable/14), we would need to have hbsd-update reference different archives between different branches--zstd for main and xz for stable/14. I would prefer not to have to include branch-specific code in a generic system updater utility. > > I have a review open to demonstrate the extent of the change in the > build system & installer: https://reviews.freebsd.org/D48042 One thought might be to make the choice of compression method dynamic. Folks could then choose what makes sense for them. FreeBSD could make the switch to zstd while downstreams could still use xz (should they so choose.) HardenedBSD would likely stay on xz until it makes sense to follow its upstream. > > It might be that this is not worth pursuing, as dist sets will most > likely go away with the migration to pkgbase, but I would like to > discuss and make an explicit decision. We can separately consider > compression on the release media images themselves. > > Feedback Requested: > > Is there support for this idea? Are there objections to pursuing this? > Are there other factors I should consider, especially compatibility concerns? For reference, hbsd-update can be found at [1] and hbsd-update-build can be found at [2]. [1]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current/master/usr.sbin/hbsd-update/hbsd-update?ref_type=heads [2]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current/master/usr.sbin/hbsd-update/hbsd-update-build?ref_type=heads Thanks, -- Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50 https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc