From nobody Fri Dec 13 21:47:53 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y92y51pQHz5gBQK for ; Fri, 13 Dec 2024 21:47:57 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y92y46BCQz4g1T for ; Fri, 13 Dec 2024 21:47:56 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-il1-x129.google.com with SMTP id e9e14a558f8ab-3a81a0277d3so6850725ab.3 for ; Fri, 13 Dec 2024 13:47:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; t=1734126475; x=1734731275; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wHWQuT6exLBsYrwBQHRBbp+ZWWOo58z0SRWPW3Wrq2U=; b=EDp1A0dPrv/7z1RY3P62IUBWctRS+/v5Y9kc3lCa65Jgrvd/yfoGwupJ0ETd9afhEg j0jkyqqMrJHMu4jPmG+Wokt1Gu0JNEaa3ioIQe1ZmKJLqmriKI4vMu+1O+SDAelZY4Xq MI5S6seGE0ITAPeQhV4WDjbNNRJnq8xSZ7sDlyO8W7CuPyPbbWEf30FsjqHEnQrJnZb8 7z0rt65xz2nQ684rM5rTSc4c0AZbFury/Rl3HK58dK2GqIAYeIiC+wJ4LxwvxfWx0SGm mjtFpipq5wvQB4E/TXXKyrId8kqvPBY32HcJ5WTLYDwrtUAT0qYOV5qJAcjnoJmtLiRK iRig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734126475; x=1734731275; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=wHWQuT6exLBsYrwBQHRBbp+ZWWOo58z0SRWPW3Wrq2U=; b=Qf6GrnuVWyMzdAl/3QGDvx2Nc9tAH6XyLxLU0keuN4EWv4DQ7VzZ9+m7s6yN2srzsj yTPO1MShHY4fMuxblUe5TQzlA2Ut+KHbG0ttxLnfo61+iTKEkJn0nrGyp3ImxjFE5SBy Qc1zR2aDMPyirf3PhhClXIgNr8SAFJnfAAFbi8Jid0+UjK63GNaOXjaLB68sZg6jp++G pQ8K99O7tv+m3TtuorZc8EM+SooJO4Uon7HTstPQVutbQpytTOt7JrqPajdWGz/kEvKU Rx6jg+oGpRLz/qiOyLaUJF8v8Iq/oNc7R3GDFmZ31XoGgx7CtPB4yqNhuBIkzyjnwWjb yaOQ== X-Gm-Message-State: AOJu0YzMdcydPTHtmjGVngImHs6bP/9i8beGqBXE2PMR1+DgcNSNfSLn KqhO/YPnzx9cgy8wMjHPaWkj/gUiJr7Yz6VdsJR9pBpwUDsfmuKYtTAy7YEl2rXlgieGvZseD3J Exeg= X-Gm-Gg: ASbGncvh+XqZ42lfVD4MQQV0dq0SxilshSijYv9Zvse71e74316MTdGrVAZmrbOURvB yzAHG9n9TiRQcUlwmRUPzCC8kXt4Hve3S2GXRKdGYw8LoAfIhR/jYbnwZWzd2PFE7d+N1BATCdi 30w7zdMTwj4yPU2GDoYSG3+9REth/YqvNdekZcIQUY8xH1H+J/mWKQZ96WFEHVD7TvcGmaWW+jk NwTJzy3R+NqDS28E367AKQ9zOZIztjIC19p+Fg= X-Google-Smtp-Source: AGHT+IFIJoI6zW3mNj5g1/3vw6S+qPXKgsQbX7kanrnl8TTtp3QzBhAB/dTSuxuftBG+610CYiUsiQ== X-Received: by 2002:a92:c266:0:b0:3a8:1195:f1f9 with SMTP id e9e14a558f8ab-3aff50b324fmr54547905ab.6.1734126475381; Fri, 13 Dec 2024 13:47:55 -0800 (PST) Received: from mutt-hbsd ([2001:470:4001:1::95]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4e5e0a3de83sm73992173.57.2024.12.13.13.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Dec 2024 13:47:54 -0800 (PST) Date: Fri, 13 Dec 2024 21:47:53 +0000 From: Shawn Webb To: Ed Maste Cc: FreeBSD Current Subject: Re: Switching release media dist sets to .tzst (tar + zstd)? Message-ID: X-Operating-System: FreeBSD mutt-hbsd 14.2-STABLE-HBSD FreeBSD 14.2-STABLE-HBSD HARDENEDBSD-14-STABLE amd64 X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="jruhgstu4ull3b45" Content-Disposition: inline In-Reply-To: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Y92y46BCQz4g1T X-Spamd-Bar: ---- --jruhgstu4ull3b45 Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Subject: Re: Switching release media dist sets to .tzst (tar + zstd)? MIME-Version: 1.0 Hey Ed, Thanks for providing the opportunity to discuss this before landing it. On Fri, Dec 13, 2024 at 04:15:45PM -0500, Ed Maste wrote: > I have been reviewing parts of the release artifact build process, > including ISO and memstick images, and came across the distribution > sets (e.g., base.txz, src.txz) used by the installer to populate new > file systems. I=E2=80=99d like to discuss switching these to .tzst (tar + > zstd) compression. >=20 > While I haven=E2=80=99t yet conducted detailed benchmarks comparing zstd = and > xz specifically for this use case, here are some initial > considerations: >=20 > Pros of zstd: > - Faster compression and decompression speeds. > - Aligns with the compression method used for FreeBSD packages. >=20 > Cons of zstd: > - Somewhat larger compressed file sizes. > - Requires updates to tools that interact with distribution sets. > - May have limited availability on some other operating systems (?). The tool for updating HardenedBSD installs (and the tool used to build the update artifacts) would be impacted. It wouldn't be too difficult to update the tools (hbsd-update and hbsd-update-build). However, if the switch zstd is not done at the same time for all supported branches (main and stable/14), we would need to have hbsd-update reference different archives between different branches--zstd for main and xz for stable/14. I would prefer not to have to include branch-specific code in a generic system updater utility. >=20 > I have a review open to demonstrate the extent of the change in the > build system & installer: https://reviews.freebsd.org/D48042 One thought might be to make the choice of compression method dynamic. Folks could then choose what makes sense for them. FreeBSD could make the switch to zstd while downstreams could still use xz (should they so choose.) HardenedBSD would likely stay on xz until it makes sense to follow its upstream. >=20 > It might be that this is not worth pursuing, as dist sets will most > likely go away with the migration to pkgbase, but I would like to > discuss and make an explicit decision. We can separately consider > compression on the release media images themselves. >=20 > Feedback Requested: >=20 > Is there support for this idea? Are there objections to pursuing this? > Are there other factors I should consider, especially compatibility conce= rns? For reference, hbsd-update can be found at [1] and hbsd-update-build can be found at [2]. [1]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current= /master/usr.sbin/hbsd-update/hbsd-update?ref_type=3Dheads [2]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/blob/hardened/current= /master/usr.sbin/hbsd-update/hbsd-update-build?ref_type=3Dheads Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50 https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --jruhgstu4ull3b45 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmdcq4EACgkQ/y5nonf4 4fps4hAAgkCULKlwU/+oOoN0qQOzTYrYzyQEizkYR3ZWCiKMfjgqgEeCG9ihhv1+ tHqkRCaO5rJDiFJviHG6UfhCaZvD2UhpAnDHwJu2ZPXRi746EU5VCHQndk+y19pL eeoZelqqb0KCDIfHZw+PNVozhec57ngP2VjCvRXtTjdXHNYufTkyIXe+R8VLmyEX 2Qh54Yy9cWKXrfcQNiCgJYWg2ZPZKghrZDEavTD/ss0Dwh3eETfwysb6x1YoxJfT XClYfNZvqng9TLi1PRabSSRinfw6sJJKTupN9kw1KHAR5CBxlf0cwDMp1d7mSQMt 3b4hAeVjLRtlUqqT/tnu1AohpP3XuIUWVyvLiLhWYinHcusm+QZJ5Z7oAMfIZnga NIsqeITbaHt4vMUxZ7aQht8Co4ZNDuI/KvZzmOAFIEyfvSI7+GoumwpdJAr8gkzI pYMEfbHaFCMGjrDhtkj6pyYFfvCiU9eOv0e6lItqmBKEgSUTHE2C6WayzsTlkp5A Y5/GiJfVycz522GFwoLXyzeKIn4Cojrd3vVQMihwTzsTaSltg8pl0KD+EAw7Ge2i Q2aUCa0jVABvD+5TQde+QOlLZpSYTv4f/0g8EkxuGO1EZNnulXp5udbWT3HVoLeX jggbK8y/9OGfL4gr0ujxp1gL+HSTvd19tHbf/dbpLKExPdkT2VU= =iIAH -----END PGP SIGNATURE----- --jruhgstu4ull3b45--