From nobody Tue Apr 01 10:39:54 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZRkz74Jflz5sK1P for ; Tue, 01 Apr 2025 10:40:03 +0000 (UTC) (envelope-from SRS0=hXZs=WT=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int-backup.realworks.nl (smtp-relay-int-backup.realworks.nl [87.255.56.188]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZRkz71mdgz3ThT; Tue, 01 Apr 2025 10:40:03 +0000 (UTC) (envelope-from SRS0=hXZs=WT=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; none Received: from smtp-relay-int.realworks.nl (rwvirtual375.colo.realworks.nl [10.0.10.75]) by mailrelayint1.colo2.realworks.nl (Postfix) with ESMTP id 4ZRkyy6qMDz1lq; Tue, 1 Apr 2025 12:39:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1743503995; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=lITHTZ0vZ6HzLd7ON9acwZrPCb+K5Jx8++gmYNdbdVQ=; b=rU6r6Dd1OAlOSUkh+rdYTHFo2Fus8+3i6irk+oM9H4J+cCCUNRrnubVqBsnshMrDSpVrSp dKNk5Qchs61bibG1Rodf0u6+CRWYeXNu3HHrLsVS5vDzyEL+DY1BtXrTlALII+dWtsUnRs AiQCP5qRrXqr//e0AgCoJ5p3CVSt0VUXVKjtIh9G4WZ941/itlQU7gyn9qRSD3enboqNNi 0va5YvL2LBgxyDyVcAKTeQ/MvpYclnG20CNF8bITfeZcGFvlPKByyXh2/3pV0RAFh+O24m IwKdQWvkhaYS3nVJuP3Mm/Q/gRzRQq3yrZy4czIWfKy8S+e3R9vqlsd4tyKcfA== Received: from rwvirtual375.colo.realworks.nl (localhost [127.0.0.1]) by rwvirtual375.colo.realworks.nl (Postfix) with ESMTP id DA57740030; Tue, 1 Apr 2025 12:39:54 +0200 (CEST) Date: Tue, 1 Apr 2025 12:39:54 +0200 (CEST) From: Ronald Klop To: Dave Cottlehuber Cc: current@freebsd.org Message-ID: <2001775390.7235.1743503994892@localhost> In-Reply-To: <7d155dfe-800c-4ba4-b59b-bf710ab45991@app.fastmail.com> References: <7d155dfe-800c-4ba4-b59b-bf710ab45991@app.fastmail.com> Subject: Re: nfs4: some shares have invisible files & directories List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7234_805019400.1743503994890" X-Mailer: Realworks (744.30) X-Originating-Host: from (84-105-120-103.cable.dynamic.v4.ziggo.nl [84.105.120.103]) by rwvirtual375 [10.0.10.75] with HTTP; Tue, 01 Apr 2025 12:39:54 +0200 Importance: Normal X-Priority: 3 (Normal) X-Originating-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:136.0) Gecko/20100101 Firefox/136.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:38930, ipnet:87.255.32.0/19, country:NL] X-Rspamd-Queue-Id: 4ZRkz71mdgz3ThT X-Spamd-Bar: ---- ------=_Part_7234_805019400.1743503994890 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, Do you have some local filesytem mounted over the NFS4 mount? Your remark a= bout 'It was "resolved" by `zfs destroy zroot/usr/{ports,src}`' made me thi= nk in this direction. Did you do the zfs destroy on the NFS client or on th= e NFS server? Regards, Ronald. =20 Van: Dave Cottlehuber Datum: dinsdag, 1 april 2025 12:17 Aan: current@freebsd.org Onderwerp: nfs4: some shares have invisible files & directories >=20 > TLDR I have working nfs4 mounts but in some mounts, none of the > files are visible to ls etc. But if you know the filename, you can > still access it directly by name. Why? >=20 > I should have a very simple setup for nfsv4. What I need is to > have ro exports /usr/{ports,obj,src} available to clients, all > FreeBSD. >=20 > I can mount all exported filesystems, but only /usr/obj has visible > files on the client, and I don't understand why. This behaviour is > consistent across clients. >=20 > during beinstall.sh from clients, they are unable to access files > that definitely exist in /usr/obj/* such as /usr/bin/cc & /bin/rm. >=20 > It was "resolved" by `zfs destroy zroot/usr/{ports,src}` & recreating > them. I don't recall ever using getfacl/setfacl, and I can't think > of anything else that might make the mount work but the files > be invisible. Sadly I did not think to just rename the afflicted > datasets. >=20 > Does anybody know what might cause this? I did find a mention of > this from SUSE linux, but in 2011, and I don't see any interesting > commits related to readdir recently. >=20 > https://forums.opensuse.org/t/11-4-strange-nfs-v4-problem-invisible-files= /63107 >=20 > ## server >=20 > - fast build box running nfsv4 only > - 15-CURRENT, up to date > - providing /usr/{src,obj,ports} to 3 clients > - this is a zfs system, and each are on separate datasets >=20 > ``` > # egrep -hr 'nfs|mount' /etc/rc.conf* >=20 > mountd_enable=3DYES > nfs_server_enable=3DYES > nfsv4_server_enable=3DYES > nfsv4_server_only=3DYES >=20 > # cat /etc/exports > V4: /usr -sec=3Dsys > /usr/src -ro > /usr/obj -ro > /usr/ports -ro >=20 > # ls -AFGhld /usr/{src,obj,ports} > drwxr-xr-x 3 dch wheel 3B Feb 28 23:27 /usr/obj/ > drwxr-xr-x 70 dch wheel 86B Apr 1 06:55 /usr/ports/ > drwxr-xr-x 27 dch wheel 47B Mar 31 13:26 /usr/src/ >=20 > # getfacl /usr/src > # file: /usr/src > # owner: dch > # group: wheel > owner@:rwxp--aARWcCos:-------:allow > group@:rwxp--a-R-c--s:-------:allow > everyone@:r-x---a-R-c--s:-------:allow > # getfacl /usr/src/COPYRIGHT > # file: /usr/src/COPYRIGHT > # owner: dch > # group: wheel > owner@:rw-p--aARWcCos:-------:allow > group@:rw-p--a-R-c--s:-------:allow > everyone@:r-----a-R-c--s:-------:allow > ``` >=20 > # clients >=20 > - either 14.2-RELEASE or also 15-CURRENT > - no daemons >=20 > ``` > # mount_nfs -o vers=3D4,nolockd,retrycnt=3D0,noatime,ro 172.16.1.4:/obj /= usr/obj > # ls /usr/obj/usr/src/amd64.amd64/ > bin/ ... lots more files, very good >=20 > # mount_nfs -o vers=3D4,nolockd,retrycnt=3D0,noatime,ro 172.16.1.4:/src /= usr/src > # mkdir /usr/src/foo > mkdir: /usr/src/foo: Read-only file system # good its clearly mounted >=20 > # ls -AFGhl /usr/src > total 0 >=20 > # head /usr/src/COPYRIGHT > The compilation of software known as FreeBSD is distributed under the > following terms: > .... what how is this file even here? ls shows nothing, tar also fails > ``` >=20 > A+ > Dave > =E2=80=94=E2=80=94=E2=80=94 > O for a muse of fire, that would ascend the brightest heaven of invention= ! > =20 >=20 >=20 >=20 =20 ------=_Part_7234_805019400.1743503994890 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi,

Do you have some local filesytem mounted over the NFS4 mount? Your remark a= bout 'It was "resolved" by `zfs destroy zroot/usr/{ports,src}`' made me thi= nk in this direction. Did you do the zfs destroy on the NFS client or on th= e NFS server?

Regards,
Ronald.

 

Van: Dave Cottlehuber <dch@FreeBSD.org>
Datum: dinsdag, 1 april 2025 12:17
Aan: current@freebsd.org
Onderwerp: nfs4: some shares have invisible files & di= rectories

TLDR I have working nfs4 mounts b= ut in some mounts, none of the
files are visible to ls etc. But if you know the filename, you can
still access it directly by name. Why?

I should have a very simple setup for nfsv4. What I need is to
have ro exports /usr/{ports,obj,src} available to clients, all
FreeBSD.

I can mount all exported filesystems, but only /usr/obj has visible
files on the client, and I don't understand why. This behaviour is
consistent across clients.

during beinstall.sh from clients, they are unable to access files
that definitely exist in /usr/obj/* such as /usr/bin/cc & /bin/rm.

It was "resolved" by `zfs destroy zroot/usr/{ports,src}` & recreating them. I don't recall ever using getfacl/setfacl, and I can't think
of anything else that might make the mount work but the files
be invisible. Sadly I did not think to just rename the afflicted
datasets.

Does anybody know what might cause this? I did find a mention of
this from SUSE linux, but in 2011, and I don't see any interesting
commits related to readdir recently.

https://forums.opensuse.org/t/11-4-strange-nfs-v4-problem-= invisible-files/63107

## server

- fast build box running nfsv4 only
- 15-CURRENT, up to date
- providing /usr/{src,obj,ports} to 3 clients
- this is a zfs system, and each are on separate datasets

```
# egrep -hr 'nfs|mount' /etc/rc.conf*

mountd_enable=3DYES
nfs_server_enable=3DYES
nfsv4_server_enable=3DYES
nfsv4_server_only=3DYES

# cat /etc/exports
V4: /usr -sec=3Dsys
/usr/src -ro
/usr/obj -ro
/usr/ports -ro

# ls -AFGhld /usr/{src,obj,ports}
drwxr-xr-x   3 dch wheel    3B Feb 28 23:27 /usr/o= bj/
drwxr-xr-x  70 dch wheel   86B Apr  1 06:55 /usr/ports/=
drwxr-xr-x  27 dch wheel   47B Mar 31 13:26 /usr/src/

# getfacl /usr/src
# file: /usr/src
# owner: dch
# group: wheel
            own= er@:rwxp--aARWcCos:-------:allow
            gro= up@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c= --s:-------:allow
# getfacl /usr/src/COPYRIGHT
# file: /usr/src/COPYRIGHT
# owner: dch
# group: wheel
            own= er@:rw-p--aARWcCos:-------:allow
            gro= up@:rw-p--a-R-c--s:-------:allow
         everyone@:r-----a-R-c= --s:-------:allow
```

# clients

- either 14.2-RELEASE or also 15-CURRENT
- no daemons

```
# mount_nfs -o vers=3D4,nolockd,retrycnt=3D0,noatime,ro 172.16.1.4:/obj /us= r/obj
# ls /usr/obj/usr/src/amd64.amd64/
bin/  ... lots more files, very good

# mount_nfs -o vers=3D4,nolockd,retrycnt=3D0,noatime,ro 172.16.1.4:/src /us= r/src
# mkdir /usr/src/foo
mkdir: /usr/src/foo: Read-only file system # good its clearly mounted

# ls -AFGhl /usr/src
total 0

# head /usr/src/COPYRIGHT
The compilation of software known as FreeBSD is distributed under the
following terms:
.... what how is this file even here? ls shows nothing, tar also fails
```

A+
Dave
=E2=80=94=E2=80=94=E2=80=94
O for a muse of fire, that would ascend the brightest heaven of invention!<= br>  

  ------=_Part_7234_805019400.1743503994890--