From nobody Mon Mar 25 23:42:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3Ty22t4nz5FYds for ; Mon, 25 Mar 2024 23:42:50 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-vk1-xa35.google.com (mail-vk1-xa35.google.com [IPv6:2607:f8b0:4864:20::a35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3Ty13Fx2z4K5c for ; Mon, 25 Mar 2024 23:42:49 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=sHaT0Cwz; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::a35 as permitted sender) smtp.mailfrom=nonesuch@longcount.org Received: by mail-vk1-xa35.google.com with SMTP id 71dfb90a1353d-4d435a60217so1404181e0c.0 for ; Mon, 25 Mar 2024 16:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1711410168; x=1712014968; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=DN4E7jcj6IQ3jM85YpbpdH5uQoJSEi9XnaZUmgqxXGM=; b=sHaT0CwzN7FzpCHIaC8aEMz6FEPd9PvNsotB5Jx2mIbc6TeBq0JfuS42LBdrwUPZlE +HsWo7hP5SUhG2Xwr7z72VVRgVsekkXyV1idIxgbUcy7YlOygJlfazYzWTPvPrOwp5xJ N4gTlL84M0X4SmlsSu3/p5oPejcNv6+psgDxAr93Hshqg2eI4tmt545xor2jAQ1XkGrI 4KDwXvv39DodIpywYoKYrVZhr7++7i3Nzq8xe5aOfCKFSVx0XZdJO6AED9cJCqEqmC6w dbePeyGNxA/v7a3Jlz0X5KvTpx9J41RhyNwYFaZQVr2kgHkHCvp1EuL6u4Wpkj0gwPsi sncg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410168; x=1712014968; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DN4E7jcj6IQ3jM85YpbpdH5uQoJSEi9XnaZUmgqxXGM=; b=drgb/UQSdRbvNZVo2kQXEyU/+UR+T2RMnjL8sSbgA1nHRKL3PMzcN4DNh6v+REOEU7 NmH/5faEcKjaJfQ2htDMPXw22RcuqUN7352ra+TkdAjSwBfwmaGRFL/K0Vm27Q6g4fev dpqqWNpZeAwI+Q+PH54pke7sVa6xRv/KeAbC8J+jOnyA/iRPAqHF8eigImmQQBZThHxQ JgZTF1m53wHsgr9mGjmI/xBsyx4X4KNXPflke1DLyD0XRnF5KbV5HBVvnAFKiNS5VMCa ceTrCH8kfMrmhrH2mZ5bhOwkA7jhdYIwUy6qESZ6kMU0w5f8KGmxConaj/II7DX60iBQ A1ww== X-Gm-Message-State: AOJu0Yy9u5tpqTwOgxHRT75HLlrHh1EuHcgqRzxIXsoK2HpoVU+yXGmD uaHcvMvabiAKK3dR75nb514jjw5cQY/XtQqCjzE2zGc60cgfU4vcKXzRhHS7E0n3MzIqjTz7l5P 7ZvnORz3DMpEqIGoskXCtEPdeMDtrKQISkULJDfI/+lT4kr5P2BW13w== X-Google-Smtp-Source: AGHT+IEUgswXQferwM4/x46iHJkJ4ULbH9EAefNpdZzFexAPmJ79GdAPejuS10epxV6mbOE344dfM7pTP2Nl+cz5pvk= X-Received: by 2002:a05:6122:1682:b0:4d3:3adc:b639 with SMTP id 2-20020a056122168200b004d33adcb639mr6254511vkl.6.1711410166250; Mon, 25 Mar 2024 16:42:46 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <86edbyh0od.fsf@ltc.des.dev> In-Reply-To: <86edbyh0od.fsf@ltc.des.dev> From: Mark Saad Date: Mon, 25 Mar 2024 19:42:35 -0400 Message-ID: Subject: Re: TarFS To: FreeBSD Hackers Content-Type: multipart/alternative; boundary="000000000000065f0a061484bb93" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[longcount.org]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a35:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[longcount.org:+] X-Rspamd-Queue-Id: 4V3Ty13Fx2z4K5c --000000000000065f0a061484bb93 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Mar 25, 2024 at 9:59=E2=80=AFAM Dag-Erling Sm=C3=B8rgrav wrote: > Mark Saad writes: > > I was wondering if anyone has started to play with tarfs in FreeBSD 14? > > I wrote it, does that count? > > > It appears to puke with larger tarballs. > > On the contrary, one of the use cases I tested was buildworld from a > tarball of the source tree, which it handled just fine. > > Dag let me just say , great work. I really like this. > > For example, if I fetch a copy of ports or pkgsrc and decompress it to > > just a posix tar archive I can't get it to mount. > > > > root@mono:/home/nonesuch # file ports.tar > > ports.tar: POSIX tar archive > > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports > > mount: ./ports.tar: Inappropriate file type or format > > % fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst > ports.tar.zst 47 MB 5431 kBps > 09s > % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt > mount: /home/des/ports.tar.zst: Inappropriate file type or format > % dmesg | grep tarfs > tarfs_alloc_one: unsupported global extended header at 0 > % zcat ports.tar.zst | hexdump -C | head -3 > 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 > |pax_global_heade| > 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |r...............| > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > > This is a non-standard extension header emitted by `git archive` which > contains metadata about the git tree from which the archive was created. > I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k > skip=3D1 >ports.tar), but you'll run into issues with very long paths in > devel/electron*. I will take a closer look when I find the time. > > Note that tarfs can mount tarballs compressed with zstd, but performance > will be poor unless you create a multi-frame archive (try using --zstd > --options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536). > > DES > So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but I wonder if it's a by-product of netbsd using pax . NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format. root@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00 |pkgsrc/.........| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * root@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade| 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| root@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3 00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * Two other questions ( I have not tried to read the code , please forgive my assumptions ) behind the scenes are the tarballs extracted over some fs, or mounted directly and read ? Second are there any contradictions with tarfs ; can they be nfs exported, mounted on a tmpfs etc . Again good work. C an we expect to see a libarchive-fs next :) > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > --=20 mark saad | nonesuch@longcount.org --000000000000065f0a061484bb93 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Mar 25, 2024 at 9:59=E2=80=AF= AM Dag-Erling Sm=C3=B8rgrav <des@free= bsd.org> wrote:
Mark Saad <nonesuch@longcount.org> writes:
> I was wondering if anyone has started to play with tarfs in FreeBSD 14= ?

I wrote it, does that count?

> It appears to puke with larger tarballs.

On the contrary, one of the use cases I tested was buildworld from a
tarball of the source tree, which it handled just fine.


Dag let me just say , great work. I re= ally like this.
=C2=A0
> For example, if I fetch a copy of ports or pkgsrc and decompress it to=
> just a posix tar archive I can't get it to mount.
>
> root@mono:/home/nonesuch # file ports.tar
> ports.tar: POSIX tar archive
> root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports
> mount: ./ports.tar: Inappropriate file type or format

% fetch ftp://ftp.freebsd.org/pub/FreeBSD= /ports/ports/ports.tar.zst
ports.tar.zst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A047 MB 5431 kBps=C2=A0 =C2=A0 09s
% sudo mount -rt tarfs $PWD/ports.tar.zst /mnt
mount: /home/des/ports.tar.zst: Inappropriate file type or format
% dmesg | grep tarfs=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0
tarfs_alloc_one: unsupported global extended header at 0
% zcat ports.tar.zst | hexdump -C | head -3
00000000=C2=A0 70 61 78 5f 67 6c 6f 62=C2=A0 61 6c 5f 68 65 61 64 65=C2=A0 = |pax_global_heade|
00000010=C2=A0 72 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |r...............|
00000020=C2=A0 00 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |................|

This is a non-standard extension header emitted by `git archive` which
contains metadata about the git tree from which the archive was created. I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k
skip=3D1 >ports.tar), but you'll run into issues with very long path= s in
devel/electron*.=C2=A0 I will take a closer look when I find the time.

Note that tarfs can mount tarballs compressed with zstd, but performance will be poor unless you create a multi-frame archive (try using --zstd
--options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536).

DES

So I am not sure exactly what tarfs is yel= ling about with pkgsrc.tar , but I wonder if it's a by-product of netbs= d using pax .
NetBSD tar's dont work , OpenBSD stuff doe= s and they use a ustar format.

root@mono:/hom= e/nonesuch # cat pkgsrc.tar |hexdump -C |head -3
00000000 =C2=A070 6b 6= 7 73 72 63 2f 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|pkgsrc/.........|
= 00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0= |................|
*
root@mono:/home/nonesuch # cat ports.tar =C2=A0|= hexdump -C |head -3
00000000 =C2=A070 61 78 5f 67 6c 6f 62 =C2=A061 6c 5= f 68 65 61 64 65 =C2=A0|pax_global_heade|
00000010 =C2=A072 00 00 00 00 = 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|r...............|
00000020= =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|.......= .........|


root@mono:/home/none= such # cat xenocara.tar =C2=A0 |hexdump -C |head -3
00000000 =C2=A02e 00= 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|................|00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2= =A0|................|
*
<= br>

Two other questions ( I have not tried to read= the code , please forgive my assumptions )=C2=A0 behind the scenes are the= tarballs extracted over some fs, or mounted directly and read ?
= Second are there any contradictions with tarfs ; can they be nfs exported, = mounted on a tmpfs etc .

Again good work. C a= n we expect to see a libarchive-fs next :)
=C2=A0
--
Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org


--
mark saad | <= a href=3D"mailto:nonesuch@longcount.org" target=3D"_blank">nonesuch@longcou= nt.org
--000000000000065f0a061484bb93--