NFS+ZFS+nullfs on Server random Permission Denied errors on client (nfsv4)

[Rick Macklem]

Bryan Drewery wrote:
> Server:
> 9.1-RELEASE
> mount:
> > tank/distfiles/freebsd on /tank/distfiles/freebsd (zfs, NFS
> > exported, local, noatime, noexec, nfsv4acls)
> /etc/zfs/exports:
> > /tank/distfiles/freebsd -maproot=root -network 10.10.0.0/16
> 
> Clients:
> 
> 8.3-RELEASE
> fstab:
> > tank:/tank/distfiles/freebsd /mnt/distfiles nfs
> > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0
> 
> HEAD r250991:
> > tank:/tank/distfiles/freebsd /mnt/distfiles nfs
> > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0
> 
> 9.1-STABLE r247421:
> > tank:/tank/distfiles/freebsd /mnt/distfiles nfs
> > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0
> 
> Problem:
> 
> For months I constantly get random Permission Denied errors on the
> client side. Just trying the read again can fix the problem. The
> client
> will usually show this in dmesg as the same time:
> > nfsv4 client/server protocol prob err=10020
> 
This error means "no file handle" and I believe it will happen when
the client tries to access the root of the exported tree. (The root
specified by the "V4: ..." line in /etc/exports.)

> Example errors:
> > root at c1100_1:/usr/local/poudriere/data/logs/bulk/91amd64-default/2013-05-29_19:17:45/logs/errors
> > # grep "Permission denied" *
> > dconf-0.12.1_1.log:cannot open gnome3/dconf-0.12.1.tar.xz:
> > Permission denied
> > p5-Pod-Tests-1.19_1.log:pax: Unable to open
> > /mnt/distfiles//Pod-Tests-1.19.tar.gz to read <Permission denied>
> > sofia-sip-1.12.11.log:pax: Unable to open
> > /mnt/distfiles//sofia-sip-1.12.11.tar.gz to read <Permission denied>
> > tex-ptex-3.3_1.log:fetch: texlive-20120701-texmf.tar.xz: Permission
> > denied
> > tkman-2.2_2.log:pax: Unable to open /mnt/distfiles//tkman-2.2.tar.gz
> > to read <Permission denied>
> 
> 
> I have realized that this problem manifests when I nullfs mount the
> /tank/distfiles/freebsd directory on the server and start doing reads
> from it, while also reading from the clients.
> 
> It became so bad that I had to move my mail off of NFS and back onto
> the
> mail server itself since I had questions of where mail had disappeared
> to!
> 
> Has anyone else experienced this? Is is fixed in STABLE already?
> 
> 
> Bryan Drewery
There have been assorted commits to nullfs recently, but I have no idea
if they will have fixed this. If you want the NFS server to work reliably,
I'd recommend you not use nullfs. You should be able to access the ZFS
volumes locally in the server safely, just don't put any nullfs mount
in the mix.

rick