ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4
Rick Macklem
rmacklem at uoguelph.ca
Fri Dec 19 02:35:30 UTC 2014
Loic Blot wrote:
> Hi rick,
> i tried to start a LXC container on Debian Squeeze from my freebsd
> ZFS+NFSv4 server and i also have a deadlock on nfsd
> (vfs.lookup_shared=0). Deadlock procs each time i launch a squeeze
> container, it seems (3 tries, 3 fails).
>
> 921 - D 0:00.02 nfsd: server (nfsd)
>
> Here is the procstat -kk
>
Ok, I took a closer look at this and I think you have found a bug
in the NFSv4 server's file locking code. If I am correct, this will
only happen if you have the vfs.nfsd.enable_locallocks=1.
Please try setting:
vfs.nfsd.enable_locallocks=0
(which I thought was the default setting?)
This only needs to be non-zero if non-nfsd threads running in
the NFS server need to see the locks being set by NFS clients.
(If your application does require non-nfsd threads running in
the NFS server to see the file locks, you'll have to wait until
I come up with a patch for this, which could be a week or more.)
In case you are curious, when vfs.nfsd.enable_locallocks=1,
the server vop_unlock()s/vop_lock()s the vnode around a
VOP_ADVLOCK() call. Unfortunately, this introduces a LOR with
the nfsv4_lock() which is held at this time. I think this
resulted in this deadlock.
To fix it, I will have to delay the vn_lock() call until
after the nfsv4_lock() is released.
This probably hasn't been reported before, because most
don't set vfs.nfsd.enable_locallocks=1.
Thanks for reporting this and sorry I didn't spot this
sooner, rick
> PID TID COMM TDNAME KSTACK
> 921 100538 nfsd nfsd: master mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
This is the bad one, which is trying to get a vnode lock while
holding the nfsv4_lock(), which is a LOR (the code should always
acquire the nfsv4_lock() after having the vnode locked).
Since another thread has the vnode locked and is trying to acquire
the nfsv4_lock()--> deadlock.
> 921 100572 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100573 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100574 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100575 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100576 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100577 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100578 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100579 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100580 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100581 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100582 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100583 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100584 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100585 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100586 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100587 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100588 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100589 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100590 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100591 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100592 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100593 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100594 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100595 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100596 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100597 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100598 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100599 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100600 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100601 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100602 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100603 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100604 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100605 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100606 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100607 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100608 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100609 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100610 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100611 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100612 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100613 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100614 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100615 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100616 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 921 100617 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100618 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 921 100619 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100620 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100621 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100622 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100623 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100624 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100625 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100626 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100627 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100628 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100629 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100630 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100631 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100632 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100633 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100634 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100635 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100636 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100637 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100638 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100639 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100640 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100641 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100642 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100643 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100644 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100645 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100646 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100647 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100648 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100649 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100650 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100651 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100652 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100653 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100654 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100655 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100656 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100657 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100658 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100659 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100660 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100661 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100662 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100663 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100664 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100665 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 921 100666 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 nfsrvd_dorpc+0xc76
> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
>
>
> Regards,
>
> Loïc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>
> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
> > Loic Blot wrote:
> >
> >> For more informations, here is procstat -kk on nfsd, if you need
> >> more
> >> hot datas, tell me.
> >>
> >> Regards, PID TID COMM TDNAME KSTACK
> >> 918 100529 nfsd nfsd: master mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
> >> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >> amd64_syscall+0x351
> >
> > Well, most of the threads are stuck like this one, waiting for a
> > vnode
> > lock in ZFS. All of them appear to be in zfs_fhtovp().
> > I`m not a ZFS guy, so I can`t help much. I`ll try changing the
> > subject line
> > to include ZFS vnode lock, so maybe the ZFS guys will take a look.
> >
> > The only thing I`ve seen suggested is trying:
> > sysctl vfs.lookup_shared=0
> > to disable shared vop_lookup()s. Apparently zfs_lookup() doesn`t
> > obey the vnode locking rules for lookup and rename, according to
> > the posting I saw.
> >
> > I`ve added a couple of comments about the other threads below, but
> > they are all either waiting for an RPC request or waiting for the
> > threads stuck on the ZFS vnode lock to complete.
> >
> > rick
> >
> >> 918 100564 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >
> > Fyi, this thread is just waiting for an RPC to arrive. (Normal)
> >
> >> 918 100565 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100566 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100567 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100568 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100569 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100570 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> >> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
> >> 918 100571 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100572 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8 nfsrvd_dorpc+0xc76
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >
> > This one (and a few others) are waiting for the nfsv4_lock. This
> > happens
> > because other threads are stuck with RPCs in progress. (ie. The
> > ones
> > waiting on the vnode lock in zfs_fhtovp().)
> > For these, the RPC needs to lock out other threads to do the
> > operation,
> > so it waits for the nfsv4_lock() which can exclusively lock the
> > NFSv4
> > data structures once all other nfsd threads complete their RPCs in
> > progress.
> >
> >> 918 100573 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >
> > Same as above.
> >
> >> 918 100574 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100575 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100576 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100577 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100578 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100579 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100580 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100581 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100582 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100583 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100584 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100585 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100586 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100587 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100588 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100589 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100590 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100591 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100592 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100593 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100594 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100595 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100596 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100597 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100598 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100599 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100600 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100601 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100602 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100603 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100604 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100605 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100606 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100607 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >
> > Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
> >
> >> 918 100608 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> >> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100609 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100610 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> >> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
> >> fork_trampoline+0xe
Ouch! I missed this one when I looked at it, so I missed the
critical line that showed the race.
Fortunately you posted a shorter one.
This would the same race as above. Hopefully you can run with:
vfs.nfsd.enable_locallocks=0
rick
> >> 918 100611 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100612 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100613 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100614 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100615 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100616 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100617 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100618 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100619 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100620 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100621 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100622 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100623 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >> 918 100624 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100625 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100626 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100627 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100628 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100629 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100630 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100631 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100632 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100633 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100634 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100635 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100636 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100637 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100638 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100639 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100640 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100641 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100642 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100643 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100644 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100645 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100646 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100647 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100648 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100649 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100650 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100651 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100652 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100653 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100654 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100655 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100656 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100657 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >> 918 100658 nfsd nfsd: service mi_switch+0xe1
> >> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_fhtovp+0x38d
> >> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >> nfssvc_program+0x554 svc_run_internal+0xc77 svc_thread_start+0xb
> >> fork_exit+0x9a fork_trampoline+0xe
> >>
> >> Loïc Blot,
> >> UNIX Systems, Network and Security Engineer
> >> http://www.unix-experience.fr
> >>
> >> 15 décembre 2014 13:29 "Loïc Blot" <loic.blot at unix-experience.fr>
> >> a
> >> écrit:
> >>> Hmmm...
> >>> now i'm experiencing a deadlock.
> >>>
> >>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server (nfsd)
> >>>
> >>> the only issue was to reboot the server, but after rebooting
> >>> deadlock arrives a second time when i
> >>> start my jails over NFS.
> >>>
> >>> Regards,
> >>>
> >>> Loïc Blot,
> >>> UNIX Systems, Network and Security Engineer
> >>> http://www.unix-experience.fr
> >>>
> >>> 15 décembre 2014 10:07 "Loïc Blot" <loic.blot at unix-experience.fr>
> >>> a
> >>> écrit:
> >>>
> >>> Hi Rick,
> >>> after talking with my N+1, NFSv4 is required on our
> >>> infrastructure.
> >>> I tried to upgrade NFSv4+ZFS
> >>> server from 9.3 to 10.1, i hope this will resolve some issues...
> >>>
> >>> Regards,
> >>>
> >>> Loïc Blot,
> >>> UNIX Systems, Network and Security Engineer
> >>> http://www.unix-experience.fr
> >>>
> >>> 10 décembre 2014 15:36 "Loïc Blot" <loic.blot at unix-experience.fr>
> >>> a
> >>> écrit:
> >>>
> >>> Hi Rick,
> >>> thanks for your suggestion.
> >>> For my locking bug, rpc.lockd is stucked in rpcrecv state on the
> >>> server. kill -9 doesn't affect the
> >>> process, it's blocked.... (State: Ds)
> >>>
> >>> for the performances
> >>>
> >>> NFSv3: 60Mbps
> >>> NFSv4: 45Mbps
> >>> Regards,
> >>>
> >>> Loïc Blot,
> >>> UNIX Systems, Network and Security Engineer
> >>> http://www.unix-experience.fr
> >>>
> >>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>> écrit:
> >>>
> >>>> Loic Blot wrote:
> >>>>
> >>>>> Hi Rick,
> >>>>> I'm trying NFSv3.
> >>>>> Some jails are starting very well but now i have an issue with
> >>>>> lockd
> >>>>> after some minutes:
> >>>>>
> >>>>> nfs server 10.10.X.8:/jails: lockd not responding
> >>>>> nfs server 10.10.X.8:/jails lockd is alive again
> >>>>>
> >>>>> I look at mbuf, but i seems there is no problem.
> >>>>
> >>>> Well, if you need locks to be visible across multiple clients,
> >>>> then
> >>>> I'm afraid you are stuck with using NFSv4 and the performance
> >>>> you
> >>>> get
> >>>> from it. (There is no way to do file handle affinity for NFSv4
> >>>> because
> >>>> the read and write ops are buried in the compound RPC and not
> >>>> easily
> >>>> recognized.)
> >>>>
> >>>> If the locks don't need to be visible across multiple clients,
> >>>> I'd
> >>>> suggest trying the "nolockd" option with nfsv3.
> >>>>
> >>>>> Here is my rc.conf on server:
> >>>>>
> >>>>> nfs_server_enable="YES"
> >>>>> nfsv4_server_enable="YES"
> >>>>> nfsuserd_enable="YES"
> >>>>> nfsd_server_flags="-u -t -n 256"
> >>>>> mountd_enable="YES"
> >>>>> mountd_flags="-r"
> >>>>> nfsuserd_flags="-usertimeout 0 -force 20"
> >>>>> rpcbind_enable="YES"
> >>>>> rpc_lockd_enable="YES"
> >>>>> rpc_statd_enable="YES"
> >>>>>
> >>>>> Here is the client:
> >>>>>
> >>>>> nfsuserd_enable="YES"
> >>>>> nfsuserd_flags="-usertimeout 0 -force 20"
> >>>>> nfscbd_enable="YES"
> >>>>> rpc_lockd_enable="YES"
> >>>>> rpc_statd_enable="YES"
> >>>>>
> >>>>> Have you got an idea ?
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>>>> écrit:
> >>>>>> Loic Blot wrote:
> >>>>>>
> >>>>>>> Hi rick,
> >>>>>>>
> >>>>>>> I waited 3 hours (no lag at jail launch) and now I do: sysrc
> >>>>>>> memcached_flags="-v -m 512"
> >>>>>>> Command was very very slow...
> >>>>>>>
> >>>>>>> Here is a dd over NFS:
> >>>>>>>
> >>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
> >>>>>>> bytes/sec)
> >>>>>>
> >>>>>> Can you try the same read using an NFSv3 mount?
> >>>>>> (If it runs much faster, you have probably been bitten by the
> >>>>>> ZFS
> >>>>>> "sequential vs random" read heuristic which I've been told
> >>>>>> things
> >>>>>> NFS is doing "random" reads without file handle affinity. File
> >>>>>> handle affinity is very hard to do for NFSv4, so it isn't
> >>>>>> done.)
> >>>>
> >>>> I was actually suggesting that you try the "dd" over nfsv3 to
> >>>> see
> >>>> how
> >>>> the performance compared with nfsv4. If you do that, please post
> >>>> the
> >>>> comparable results.
> >>>>
> >>>> Someday I would like to try and get ZFS's sequential vs random
> >>>> read
> >>>> heuristic modified and any info on what difference in
> >>>> performance
> >>>> that
> >>>> might make for NFS would be useful.
> >>>>
> >>>> rick
> >>>>
> >>>>>> rick
> >>>>>>
> >>>>>>> This is quite slow...
> >>>>>>>
> >>>>>>> You can found some nfsstat below (command isn't finished yet)
> >>>>>>>
> >>>>>>> nfsstat -c -w 1
> >>>>>>>
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 16 0
> >>>>>>> 2 0 0 0 0 0 17 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 3 0
> >>>>>>> 0 0 0 0 0 0 3 0
> >>>>>>> 37 10 0 8 0 0 14 1
> >>>>>>> 18 16 0 4 1 2 4 0
> >>>>>>> 78 91 0 82 6 12 30 0
> >>>>>>> 19 18 0 2 2 4 2 0
> >>>>>>> 0 0 0 0 2 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 1 0 0 0 0 1 0
> >>>>>>> 4 6 0 0 6 0 3 0
> >>>>>>> 2 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 1 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 1 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 6 108 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 98 54 0 86 11 0 25 0
> >>>>>>> 36 24 0 39 25 0 10 1
> >>>>>>> 67 8 0 63 63 0 41 0
> >>>>>>> 34 0 0 35 34 0 0 0
> >>>>>>> 75 0 0 75 77 0 0 0
> >>>>>>> 34 0 0 35 35 0 0 0
> >>>>>>> 75 0 0 74 76 0 0 0
> >>>>>>> 33 0 0 34 33 0 0 0
> >>>>>>> 0 0 0 0 5 0 0 0
> >>>>>>> 0 0 0 0 0 0 6 0
> >>>>>>> 11 0 0 0 0 0 11 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 17 0 0 0 0 1 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 4 5 0 0 0 0 12 0
> >>>>>>> 2 0 0 0 0 0 26 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 2 0
> >>>>>>> 2 0 0 0 0 0 24 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 0 0 0 0 0 7 0
> >>>>>>> 2 1 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 2 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 6 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 6 0 0 0 0 3 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 2 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 71 0 0 0 0 0 0
> >>>>>>> 0 1 0 0 0 0 0 0
> >>>>>>> 2 36 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 1 0 0 0 0 0 1 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 79 6 0 79 79 0 2 0
> >>>>>>> 25 0 0 25 26 0 6 0
> >>>>>>> 43 18 0 39 46 0 23 0
> >>>>>>> 36 0 0 36 36 0 31 0
> >>>>>>> 68 1 0 66 68 0 0 0
> >>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>>>> 36 0 0 36 36 0 0 0
> >>>>>>> 48 0 0 48 49 0 0 0
> >>>>>>> 20 0 0 20 20 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 3 14 0 1 0 0 11 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 0 4 0 0 0 0 4 0
> >>>>>>> 0 0 0 0 0 0 0 0
> >>>>>>> 4 22 0 0 0 0 16 0
> >>>>>>> 2 0 0 0 0 0 23 0
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Loïc Blot,
> >>>>>>> UNIX Systems, Network and Security Engineer
> >>>>>>> http://www.unix-experience.fr
> >>>>>>>
> >>>>>>> 8 décembre 2014 09:36 "Loïc Blot"
> >>>>>>> <loic.blot at unix-experience.fr> a
> >>>>>>> écrit:
> >>>>>>>> Hi Rick,
> >>>>>>>> I stopped the jails this week-end and started it this
> >>>>>>>> morning,
> >>>>>>>> i'll
> >>>>>>>> give you some stats this week.
> >>>>>>>>
> >>>>>>>> Here is my nfsstat -m output (with your rsize/wsize tweaks)
> >>>
> >>>
> >>
> > nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
> >>>
> >>>>>>>>
> >>>
> >>>
> >>
> > etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
> >>>
> >>> s=2147483647
> >>>
> >>> On server side my disks are on a raid controller which show a
> >>> 512b
> >>> volume and write performances
> >>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
> >>> count=100000000 => 450MBps)
> >>>
> >>> Regards,
> >>>
> >>> Loïc Blot,
> >>> UNIX Systems, Network and Security Engineer
> >>> http://www.unix-experience.fr
> >>>
> >>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>> écrit:
> >>>
> >>>> Loic Blot wrote:
> >>>>
> >>>>> Hi,
> >>>>> i'm trying to create a virtualisation environment based on
> >>>>> jails.
> >>>>> Those jails are stored under a big ZFS pool on a FreeBSD 9.3
> >>>>> which
> >>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a big
> >>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but only 1
> >>>>> was
> >>>>> used at this time).
> >>>>>
> >>>>> The problem is simple, my hypervisors runs 6 jails (used 1% cpu
> >>>>> and
> >>>>> 10GB RAM approximatively and less than 1MB bandwidth) and works
> >>>>> fine at start but the system slows down and after 2-3 days
> >>>>> become
> >>>>> unusable. When i look at top command i see 80-100% on system
> >>>>> and
> >>>>> commands are very very slow. Many process are tagged with
> >>>>> nfs_cl*.
> >>>>
> >>>> To be honest, I would expect the slowness to be because of slow
> >>>> response
> >>>> from the NFSv4 server, but if you do:
> >>>> # ps axHl
> >>>> on a client when it is slow and post that, it would give us some
> >>>> more
> >>>> information on where the client side processes are sitting.
> >>>> If you also do something like:
> >>>> # nfsstat -c -w 1
> >>>> and let it run for a while, that should show you how many RPCs
> >>>> are
> >>>> being done and which ones.
> >>>>
> >>>> # nfsstat -m
> >>>> will show you what your mount is actually using.
> >>>> The only mount option I can suggest trying is
> >>>> "rsize=32768,wsize=32768",
> >>>> since some network environments have difficulties with 64K.
> >>>>
> >>>> There are a few things you can try on the NFSv4 server side, if
> >>>> it
> >>>> appears
> >>>> that the clients are generating a large RPC load.
> >>>> - disabling the DRC cache for TCP by setting vfs.nfsd.cachetcp=0
> >>>> - If the server is seeing a large write RPC load, then
> >>>> "sync=disabled"
> >>>> might help, although it does run a risk of data loss when the
> >>>> server
> >>>> crashes.
> >>>> Then there are a couple of other ZFS related things (I'm not a
> >>>> ZFS
> >>>> guy,
> >>>> but these have shown up on the mailing lists).
> >>>> - make sure your volumes are 4K aligned and ashift=12 (in case a
> >>>> drive
> >>>> that uses 4K sectors is pretending to be 512byte sectored)
> >>>> - never run over 70-80% full if write performance is an issue
> >>>> - use a zil on an SSD with good write performance
> >>>>
> >>>> The only NFSv4 thing I can tell you is that it is known that
> >>>> ZFS's
> >>>> algorithm for determining sequential vs random I/O fails for
> >>>> NFSv4
> >>>> during writing and this can be a performance hit. The only
> >>>> workaround
> >>>> is to use NFSv3 mounts, since file handle affinity apparently
> >>>> fixes
> >>>> the problem and this is only done for NFSv3.
> >>>>
> >>>> rick
> >>>>
> >>>>> I saw that there are TSO issues with igb then i'm trying to
> >>>>> disable
> >>>>> it with sysctl but the situation wasn't solved.
> >>>>>
> >>>>> Someone has got ideas ? I can give you more informations if you
> >>>>> need.
> >>>>>
> >>>>> Thanks in advance.
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
>
More information about the freebsd-fs
mailing list