Hanging/stalling mountd on heavily loaded NFS server

[Rick Macklem]

Marc Goroff wrote:

> From: owner-freebsd-fs at freebsd.org <owner-freebsd-fs at freebsd.org> on behalf of Marc Goroff <marc.goroff at quorum.net

> Sent: Wednesday, July 27, 2016 7:04 PM
> To: freebsd-fs at freebsd.org
> Subject: Hanging/stalling mountd on heavily loaded NFS server
>
> We have a large and busy production NFS server running 10.2 that is
> serving approximately 200 ZFS file systems to production VMs. The system
> has been very stable up until last night when we attempted to mount new
> ZFS filesystems on NFS clients. The mountd process hung and client mount
> requests timed out. The NFS server continued to serve traffic to
> existing clients during this time. The mountd was hung in state nfsv4lck:
>
> [root at zfs-west1 ~]# ps -axgl|grep mount
   0 38043     1   0  20  0  63672 17644 nfsv4lck Ds    - 0:00.30
/usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports
>
> It remains in this state for an indeterminate amount of time. I once saw
> it continue on after several minutes, but most of the time it seems to
> stay in this state for 15+ minutes. During this time, it does not
> respond to kill -9 but it will eventually exit after many minutes.
> Restarting mountd will allow the existing NFS clients to continue (they
> hang when mountd exits), but any attempt to perform additional NFS
> mounts will push mountd back into the bad state.
>
> This problem seems to be related to the number of NFS mounts off the
> server. If we unmount some of the clients, we can successfully perform
> the NFS mounts of the new ZFS filesystems. However, when we attempt to
> mount all of the production NFS mounts, mountd will hang as above.
>
Stuff snipped for brevity...
>
> Any suggestion on how to resolve this issue? Since this is a production
> server, my options for intrusive debugging are very limited.
>
I think you should try the patch that is r300254 in stable/10. It is a simple
patch you can apply to your kernel without other changes.

http://svnweb.freebsd.org/base/stable/10/sys/fs/nfsserver/nfs_nfsdkrpc.c?r1=291869&r2=300254

It reverses the lock acquisition priority so that mountd doesn't wait until the
nfsd threads are idle before updating exports.

rick

> Thanks.
>
> Marc
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs<https://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"