[Bug 251347] NFS hangs on client side when mounted from outside in Jail Tree (BROKEN NFS SERVER OR MIDDLEWARE)

Tue Nov 24 18:24:03 UTC 2020

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251347

            Bug ID: 251347
           Summary: NFS hangs on client side when mounted from outside in
                    Jail Tree (BROKEN NFS SERVER OR MIDDLEWARE)
           Product: Base System
           Version: 12.1-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: arne at Steinkamm.COM

Created attachment 219934
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=219934&action=edit
tar file with all configs and command outputs of server, client and jail

Hi folks,
I ran into an really nasty problem after introducing FreeBSD to a really big
company.

All machines are running FreeBSD 12.1 Release and use 10GBit/s ethernet links
with Cisco Nexus 55xx switches. Additional 1GBit/s networks for admin- und
security related issues exists.

I have this problem on all my NFS client machine running FreeBSD.

NFS Server (FreeBSD 12.1) exports a ZFS dataset with ca. 1,5 TB.

Linux clients mounting this dataset with nfsv3 TCP mounts have no problems.

FreeBSD hosts mount this dataset with nfs3 TCP.

Scenario 1)
FreeBSD client mounts share. No problems.

Scenario 2)
FreeBSD client mounts share and relay it using nullfs mounts in VNET
(if_bridge) Jails.

Scenario 3)
FreeBSD client mounts share twice. One mountpoint is located in his primary
file system tree and the other one is inside the jail's sub tree.

Accessing the share from inside the jail hangs the NFS mount after only a few
minutes of usage.
In Scenario 3 only the mount inside the jails directory sub tree is blocked.
The other one is working. Nevertheless df & friends on the main host is blocked
too because they touch the Jails subtree o9n their traversal down.
Same in Scenario 2. There the only NFS mount is blocked.

There are no error messages at this point of time. No console text about stale
NFS links etc. Nothing in the Jail's console file.

umount of the mount blocks too but the blocking mount line vanished in the
mount(8) output. 
The only way to unfreeze is to stop the jail in which sub tree the NULLFS or
NFS mount is located in.

Stopping the jail immediately frees the hanging df and ls etc. processes.

On the console I can read after umount -f / jail stop:
newnfs: server '192.168.67.38' error: fileid changed. fsid 0:0: expected fileid
0x4, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE)

I read the comments about this error message in
/usr/src/sys/fs/nfsclient/nfs_clport.c. They are not matching this setup. There
is no broken middleware... only cisco switches, a HP Proliant Gen9 FreeBSD NFS
Server and a few Oracle Sunfire-something x86_64 FreeBSD clients.

The NFS Server is running FreeBSD 12.1 and the network is as simple as a 10
GBit/s network can be.

Playing with enforce_statfs seems to have no effect.

What's going wrong here?

Some details:
I use the jib script from /usr/src/share/exampes/jails after adding two lines
to add MTU support to handle the Jumbo 10 Gig network.
Firewall configurations are enabled in the kernel. To understand the problem
here all is disabled or in pass through configurations.
There are no other visible problems. 

I add this files as attachment:

adm01:
total 240
-rw-r--r--  1 arne  ego  21833 Nov 24 18:24 NFS-Client-ADM001
-rw-------  1 arne  ego   3254 Nov 24 00:07 NFS-Client-devfs.rules
-rw-r-----  1 arne  ego  17288 Nov 24 18:57 NFS-Client-df
-rw-r--r--  1 arne  ego  19051 Nov 24 18:24 NFS-Client-dmesg.boot
-rw-r--r--  1 arne  ego    636 Nov 24 14:41 NFS-Client-fstab
-rw-r-----  1 arne  ego  25965 Nov 24 18:24 NFS-Client-ifconfig-a
-rw-r--r--  1 arne  ego   3600 Nov 24 18:33 NFS-Client-jail.conf
-rw-r--r--  1 arne  ego   1239 Nov 23 23:31 NFS-Client-loader.conf
-rw-r-----  1 arne  ego    522 Nov 24 18:58 NFS-Client-nfsstat-m
-rw-r-----  1 arne  ego  76519 Nov 24 03:23 NFS-Client-pf.conf
-rw-r--r--  1 arne  ego  25053 Nov 24 18:21 NFS-Client-rc.conf
-rw-r--r--  1 arne  ego   1084 Nov 23 23:33 NFS-Client-sysctl.conf

fs001:
total 124
-rw-r--r--  1 arne  ego  21767 Nov 24 19:05 Fileserver-FS001
-rw-r--r--  1 arne  ego  27395 Nov 24 19:06 Fileserver-dmesg.boot
-rw-r-----  1 arne  ego  14781 Nov 24 19:03 Fileserver-ifconfig-a
-rw-r--r--  1 arne  ego   1020 Nov 16 21:40 Fileserver-loader.conf
-rw-r--r--  1 arne  ego  14356 Nov 24 19:04 Fileserver-rc.conf
-rw-r--r--  1 arne  ego   1106 Nov 16 21:38 Fileserver-sysctl.conf
-rw-r-----  1 arne  ego   6398 Nov 24 18:14 Fileserver-zfs_properties

jail:
total 28
-rw-r-----  1 arne  ego  1671 Nov 24 19:01 inside-jail-ifconfig-a
-r--------  1 arne  ego     9 Nov 24 00:22 inside-jail-pf.conf
-rw-r--r--  1 arne  ego   636 Nov 24 19:00 inside-jail-rc.conf
-rw-r--r--  1 arne  ego   709 Nov 24 00:17 inside-jail-sysctl.conf

Thanks in advance,
.//. Arne

-- 
You are receiving this mail because:
You are the assignee for the bug.