From nobody Tue Oct 03 17:56:11 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0QVf1RCDz4vnL6 for ; Tue, 3 Oct 2023 17:56:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0QVd6XJYz4NsR for ; Tue, 3 Oct 2023 17:56:25 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-5333fb34be3so2147502a12.1 for ; Tue, 03 Oct 2023 10:56:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1696355784; x=1696960584; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=6bc29Yj4cHcNmRqn0pZ6ZjWSK1HqaaDkk6m/+PrpauI=; b=g4ZgrPT2VaQeFYEhtCVrAmHymxkv1XJ+OYpLRgGlA4TZoc6Ra/FiqUucycU7TDBK8r QX9dmd4YdYkP030QgWqS5CnZAzF8SE3ugAVpZlKpZM9CdAbUreNLZSp8Ang+Kgdh8tOL LqKqpabS0hjv+/2IcmNp52W/hWLdrDhk1sJ1CgRQYbnBFqzv6C+TSgLCixwboo5hWxwS 0glAmO3tA3xdPMxyanHc+ixq/a7vlzGZxPArZK0AFshTwwJ/vzHWHrm0paYgODt3BnKM l43YbDrN1nrGnQs+DKHFRygA6aujnM752AaelzFxchb2AioxwUAx9B8cjU/sMWoYheiI RGtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696355784; x=1696960584; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6bc29Yj4cHcNmRqn0pZ6ZjWSK1HqaaDkk6m/+PrpauI=; b=M5YJDtN2lC4fAysgTXxkYpKbvpsIez/wCFgT7SeROKhBaxlsKSFoDvYuWUVjrPUnIi wX7Kr4ECa+2h/Hy+369+dyTezMslToPYCehI0gV0F7hIKJOAtyhYCnlE4O0XtAQSmnRy xcbF6Etu3gEieKJ7X6WfcgKvErl/6GGAlyVO1yNrXhrMJo/i/xp9RocFvgoRVuyZ5Gm4 bipNCo8UHfLG4yGpXnBfRk1WMac6mji80PYuRICzcX/duxGEYWHffD5V5RBLxYH5yNO7 7DAA/q+1nkJtmajF6h3WCGRQAygDIMzBC7L/tL92zJbi6oSZbfINc4fq83eaO3mAfOAp vzYw== X-Gm-Message-State: AOJu0YwzTMdBaajcDnhVbqlTykuLfRSRTb2p56nVshWCJtGSP9KHhb3w 6RXSGadXOdUaE0Jkz6HfdsAb2gV1/6/ILnMpUnAvtg== X-Google-Smtp-Source: AGHT+IFwyShD5Gg4R5i9TSXJUX3jMjJtsgTSOR0X9OGedafNRRl8uNj9JWiEP/MlFG3QzttAeeRDzkWDs2+eAcXjans= X-Received: by 2002:aa7:d492:0:b0:522:2782:537 with SMTP id b18-20020aa7d492000000b0052227820537mr13117522edr.15.1696355783322; Tue, 03 Oct 2023 10:56:23 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <86y1gjhdx7.fsf@ltc.des.no> In-Reply-To: <86y1gjhdx7.fsf@ltc.des.no> From: Warner Losh Date: Tue, 3 Oct 2023 11:56:11 -0600 Message-ID: Subject: Re: something magic about the size of a ports tree To: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= Cc: FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000e0c1b10606d39b60" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4S0QVd6XJYz4NsR --000000000000e0c1b10606d39b60 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 3, 2023, 10:24 AM Dag-Erling Sm=C3=B8rgrav wr= ote: > Matthias Apitz writes: > > I have on my poudriere build host a ports tree and wanted to move it to > > the host where the resulting packages are installed: > > > > root@jet:/usr/local/poudriere/ports # du -sh ports20230806 > > 397M ports20230806 > > root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806 > > root@jet:/usr/local/poudriere/ports # ls -lh p.tar > > -rw-r--r-- 1 root wheel 672M Oct 3 18:00 p.tar > > > > already the size of the tar file is somewhat magic; but if you un-tar i= t > > on the other host I will get: > > > > [guru@c720-1400094 ~]$ ls -lh p.tar > > -rw-r--r-- 1 guru wheel 672M 3 oct. 18:00 p.tar > > [guru@c720-1400094 ~]$ tar xf p.tar > > [guru@c720-1400094 ~]$ du -sh ports20230806 > > 1,2G ports20230806 > > > > How this is possible? > > Most files in the ports tree are very small. On disk, each file gets > rounded up to the nearest multiple of the filesystem block size, which > could be as small as 512 bytes or as large as 8 kB (or even more in > pathological cases). In a tarball, they get rounded up to the nearest > multiple of 512 bytes plus an additional 512 bytes per file for > metadata. > > For instance, your average distinfo file (of which there are 30k in the > ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an > FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem. > > Note that if the target system is FreeBSD 14 or newer, you can simply > mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`). > Do we support any compression on top of that? Has support for poudriere been added for it? Aldo I want a pony.... I'm mostly curious... I have no immediate plans here (though aligning with the boot loader and supporting this on a block device to support rootfs would be cool). Maybe some or all of these wishes would make good GSOC projects? Warner DES > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > > --000000000000e0c1b10606d39b60 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Oct 3, 2023, 10:24 AM Dag-Erling Sm=C3=B8rgrav= <des@freebsd.org> wrote:
<= /div>
Matthias Apitz <guru@unixarea.de&g= t; writes:
> I have on my poudriere build host a ports tree and wanted to move it t= o
> the host where the resulting packages are installed:
>
> root@jet:/usr/local/poudriere/ports # du -sh ports20230806
> 397M=C2=A0 =C2=A0 ports20230806
> root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806
> root@jet:/usr/local/poudriere/ports # ls -lh p.tar
> -rw-r--r--=C2=A0 1 root wheel=C2=A0 672M Oct=C2=A0 3 18:00 p.tar
>
> already the size of the tar file is somewhat magic; but if you un-tar = it
> on the other host I will get:
>
> [guru@c720-1400094 ~]$ ls -lh p.tar
> -rw-r--r--=C2=A0 1 guru wheel=C2=A0 672M=C2=A0 3 oct.=C2=A0 18:00 p.ta= r
> [guru@c720-1400094 ~]$ tar xf p.tar
> [guru@c720-1400094 ~]$ du -sh ports20230806
> 1,2G=C2=A0 ports20230806
>
> How this is possible?

Most files in the ports tree are very small.=C2=A0 On disk, each file gets<= br> rounded up to the nearest multiple of the filesystem block size, which
could be as small as 512 bytes or as large as 8 kB (or even more in
pathological cases).=C2=A0 In a tarball, they get rounded up to the nearest=
multiple of 512 bytes plus an additional 512 bytes per file for
metadata.

For instance, your average distinfo file (of which there are 30k in the
ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an
FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem.
Note that if the target system is FreeBSD 14 or newer, you can simply
mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`).

Do we support a= ny compression on top of that? Has support for poudriere been added for it?=

Aldo I want a pony.... = I'm mostly curious... I have no immediate plans here (though aligning w= ith the boot loader and supporting this on a block device to support rootfs= would be cool). Maybe some or all of these wishes would make good GSOC pro= jects?

Warner

DES
--
Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org

--000000000000e0c1b10606d39b60--