High Kernel Load with nfsv4
Loïc Blot
loic.blot at unix-experience.fr
Mon Jan 5 16:35:00 UTC 2015
Hi Rick,
nfsstat -e -s don't show usefull datas on server.
Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove
26935254 16911 5755728 302 2334920 3673866 0 328332
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
77980 28 0 0 3 8900 3 1806052
Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf
1 1095 0 0 614377 8172 8 8
Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock
1595299 0 44145 1495 0 0 5197490 635015
LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH
0 614919 1270938 0 0 22688676 0 5
Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
42104 197606 275820 0 143 4578
Server:
Retfailed Faults Clients
0 0 6
OpenOwner Opens LockOwner Locks Delegs
32335 145448 204 181 0
Server Cache Stats:
Inprog Idem Non-idem Misses CacheSize TCPPeak
0 0 1 15082947 60 16522
Only GetAttr and Lookup increase and it's only every 4-5 seconds and only +2 to +5 into theses values.
Now on client, if i take four processes stack i got
PID TID COMM TDNAME KSTACK
63170 102547 mv - mi_switch+0xe1 turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351 Xfast_syscall+0xfb
Another mv:
63140 101738 mv - mi_switch+0xe1 turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_syscall+0xfb
62070 102170 sendmail - mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666 clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72 nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_syscall+0xfb
63200 100930 mv - mi_switch+0xe1 turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65 nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351 Xfast_syscall+0xfb
When client is in this state, server was doing nothing special (procstat -kk)
PID TID COMM TDNAME KSTACK
895 100538 nfsd nfsd: master mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_syscall+0xfb
895 100568 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100569 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100570 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100571 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100572 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100573 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100576 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100577 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100578 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100579 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100580 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100581 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100582 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100583 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100584 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100585 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100586 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100587 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100588 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100589 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100590 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100592 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100593 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100594 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100595 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100596 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100597 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100598 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100599 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100600 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100602 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100603 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100604 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100605 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100606 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100607 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100608 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100609 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100610 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100611 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100612 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100613 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100614 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100615 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100617 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100618 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100619 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100621 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100622 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100623 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100624 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100625 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100626 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100627 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100628 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100629 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100630 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100631 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100632 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100633 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100634 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100635 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100636 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100638 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100639 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100640 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100641 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100642 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100643 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100644 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100645 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100646 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100647 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100648 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100649 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100651 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100652 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100653 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100654 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100655 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100656 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100657 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100658 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100659 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100661 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100662 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100684 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100685 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100686 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100797 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100798 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100799 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100800 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
895 100801 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
I really think it's a client side problem, maybe a lookup problem.
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
5 janvier 2015 14:35 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
> Loic Blot wrote:
>
>> Hi,
>> happy new year Rick and @freebsd-fs.
>>
>> After some days, i looked my NFSv4.1 mount. At server start it was
>> calm, but after 4 days, here is the top stat...
>>
>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0%
>> idle
>>
>> Definitively i think it's a problem on client side. What can i look
>> into running kernel to resolve this issue ?
>
> Well, I'd start with:
> # nfsstat -e -s
> - run repeatedly on the server (once every N seconds in a loop).
> Then look at the output, comparing the counts and see which RPCs
> are being performed by the client(s). You are looking for which
> RPCs are being done a lot. (If one RPC is almost 100% of the load,
> then it might be a client/caching issue for whatever that RPC is
> doing.)
>
> Also look at the Open/Lock counts near the end of the output.
> If the # of Opens/Locks is large, it may be possible to reduce the
> CPU overheads by using larger hash tables.
>
> Then you need to profile the server kernel to see where the CPU
> is being used.
> Hopefully someone else can fill you in on how to do that, because
> I'll admit I don't know how to.
> Basically you are looking to see if the CPU is being used in
> the NFS server code or ZFS.
>
> Good luck with it, rick
>
>> Regards,
>>
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>>
>> 30 décembre 2014 16:16 "Loïc Blot" <loic.blot at unix-experience.fr> a
>> écrit:
>>> Hi Rick,
>>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1
>>> (mountoptions:
>>> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)
>>>
>>> Performance is quite stable but it's slow. Not as slow as before
>>> but slow... services was launched
>>> but no client are using them and system CPU % was 10-50%.
>>>
>>> I don't see anything on NFSv4.1 server, it's perfectly stable and
>>> functionnal.
>>>
>>> Regards,
>>>
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>>
>>> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>> écrit:
>>>
>>>> Loic Blot wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> To clarify because of our exchanges. Here are the current sysctl
>>>>> options for server:
>>>>>
>>>>> vfs.nfsd.enable_nobodycheck=0
>>>>> vfs.nfsd.enable_nogroupcheck=0
>>>>>
>>>>> vfs.nfsd.maxthreads=200
>>>>> vfs.nfsd.tcphighwater=10000
>>>>> vfs.nfsd.tcpcachetimeo=300
>>>>> vfs.nfsd.server_min_nfsvers=4
>>>>>
>>>>> kern.maxvnodes=10000000
>>>>> kern.ipc.maxsockbuf=4194304
>>>>> net.inet.tcp.sendbuf_max=4194304
>>>>> net.inet.tcp.recvbuf_max=4194304
>>>>>
>>>>> vfs.lookup_shared=0
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 22 décembre 2014 09:42 "Loïc Blot" <loic.blot at unix-experience.fr>
>>>>> a
>>>>> écrit:
>>>>>
>>>>> Hi Rick,
>>>>> my 5 jails runs this weekend and now i have some stats on this
>>>>> monday.
>>>>>
>>>>> Hopefully deadlock was fixed, yeah, but everything isn't good :(
>>>>>
>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>>>>>
>>>>> As i can see this is because of nfsd:
>>>>>
>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
>>>>> 273.68% nfsd: server (nfsd)
>>>>>
>>>>> If i look at dmesg i see:
>>>>> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
>>>>
>>>> Well, you have a couple of choices:
>>>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
>>>> (NFSv4.1 avoids use of the DRC and instead uses something
>>>> called sessions. See below.)
>>>> OR
>>>>
>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
>>>>
>>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
>>>> "nfs server cache flooded" messages. (I think Garrett Wollman uses
>>>> 100000. (You may still see quite a bit of CPU overheads.)
>>>>
>>>> OR
>>>>
>>>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid
>>>> of the CPU overheads). However, there is a risk of data corruption
>>>> if you have a client->server network partitioning of a moderate
>>>> duration, because a non-idempotent RPC may get redone, becasue
>>>> the client times out waiting for a reply. If a non-idempotent
>>>> RPC gets done twice on the server, data corruption can happen.
>>>> (The DRC provides improved correctness, but does add overhead.)
>>>>
>>>> If #1 works for you, it is the preferred solution, since Sessions
>>>> in NFSv4.1 solves the correctness problem in a good, space bound
>>>> way. A session basically has N (usually 32 or 64) slots and only
>>>> allows one outstanding RPC/slot. As such, it can cache the
>>>> previous
>>>> reply for each slot (32 or 64 of them) and guarantee "exactly
>>>> once"
>>>> RPC semantics.
>>>>
>>>> rick
>>>>
>>>>> Here is 'nfsstat -s' output:
>>>>>
>>>>> Server Info:
>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>> Remove
>>>>> 12600652 1812 2501097 156 1386423 1983729 123
>>>>> 162067
>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>> Access
>>>>> 36762 9 0 0 0 3147 0
>>>>> 623524
>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>> 0 0 0 0 328117
>>>>> Server Ret-Failed
>>>>> 0
>>>>> Server Faults
>>>>> 0
>>>>> Server Cache Stats:
>>>>> Inprog Idem Non-idem Misses
>>>>> 0 0 0 12635512
>>>>> Server Write Gathering:
>>>>> WriteOps WriteRPC Opsaved
>>>>> 1983729 1983729 0
>>>>>
>>>>> And here is 'procstat -kk' for nfsd (server)
>>>>>
>>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> ---
>>>>>
>>>>> Now if we look at client (FreeBSD 9.3)
>>>>>
>>>>> We see system was very busy and do many and many interrupts
>>>>>
>>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
>>>>> idle
>>>>>
>>>>> A look at process list shows that there are many sendmail process
>>>>> in
>>>>> state nfstry
>>>>>
>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner at 00:30:00 for
>>>>> /var/spool/clientm
>>>>>
>>>>> Here is 'nfsstat -c' output:
>>>>>
>>>>> Client Info:
>>>>> Rpc Counts:
>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>> Remove
>>>>> 1051347 1724 2494481 118 903902 1901285 162676
>>>>> 161899
>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>> Access
>>>>> 36744 2 0 114 40 3131 0
>>>>> 544136
>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>> 9 0 0 0 245821
>>>>> Rpc Info:
>>>>> TimedOut Invalid X Replies Retries Requests
>>>>> 0 0 0 0 8356557
>>>>> Cache Info:
>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
>>>>> Misses
>>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
>>>>> 1871871
>>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
>>>>> Misses
>>>>> 144035 118 53736 2753 27813 1 57238839
>>>>> 544205
>>>>>
>>>>> If you need more things, tell me, i let the PoC in this state.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>>> écrit:
>>>>>
>>>>> Loic Blot wrote:
>>>>>
>>>>>> Hi Rick,
>>>>>> ok, i don't need locallocks, i haven't understand option was for
>>>>>> that
>>>>>> usage, i removed it.
>>>>>> I do more tests on monday.
>>>>>> Thanks for the deadlock fix, for other people :)
>>>>>
>>>>> Good. Please let us know if running with
>>>>> vfs.nfsd.enable_locallocks=0
>>>>> gets rid of the deadlocks? (I think it fixes the one you saw.)
>>>>>
>>>>> On the performance side, you might also want to try different
>>>>> values
>>>>> of
>>>>> readahead, if the Linux client has such a mount option. (With the
>>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what
>>>>> the
>>>>> optimal readahead value would be.)
>>>>>
>>>>> Good luck with it and please let us know how it goes, rick
>>>>> ps: I now have a patch to fix the deadlock when
>>>>> vfs.nfsd.enable_locallocks=1
>>>>> is set. I'll post it for anyone who is interested after I put it
>>>>> through some testing.
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Loïc BLOT,
>>>>> UNIX systems, security and network engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
>>>>>
>>>>> Loic Blot wrote:
>>>>>> Hi rick,
>>>>>> i tried to start a LXC container on Debian Squeeze from my
>>>>>> freebsd
>>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
>>>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
>>>>>> squeeze
>>>>>> container, it seems (3 tries, 3 fails).
>>>>>
>>>>> Well, I`ll take a look at this `procstat -kk`, but the only thing
>>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
>>>>> nullfs. (I have no idea if you are using any nullfs mounts, but
>>>>> if so, try getting rid of them.)
>>>>>
>>>>> Here`s a high level post about the ZFS and vnode locking problem,
>>>>> but there is no patch available, as far as I know.
>>>>>
>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>>>>>
>>>>> rick
>>>>>
>>>>> 921 - D 0:00.02 nfsd: server (nfsd)
>>>>>
>>>>> Here is the procstat -kk
>>>>>
>>>>> PID TID COMM TDNAME KSTACK
>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>> _cv_wait_sig+0x16a
>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>> nfsrvd_dorpc+0xc76
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>>> écrit:
>>>>>
>>>>> Loic Blot wrote:
>>>>>
>>>>>> For more informations, here is procstat -kk on nfsd, if you
>>>>>> need
>>>>>> more
>>>>>> hot datas, tell me.
>>>>>>
>>>>>> Regards, PID TID COMM TDNAME KSTACK
>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>> amd64_syscall+0x351
>>>>>
>>>>> Well, most of the threads are stuck like this one, waiting for
>>>>> a
>>>>> vnode
>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
>>>>> subject line
>>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
>>>>> look.
>>>>>
>>>>> The only thing I`ve seen suggested is trying:
>>>>> sysctl vfs.lookup_shared=0
>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
>>>>> doesn`t
>>>>> obey the vnode locking rules for lookup and rename, according
>>>>> to
>>>>> the posting I saw.
>>>>>
>>>>> I`ve added a couple of comments about the other threads below,
>>>>> but
>>>>> they are all either waiting for an RPC request or waiting for
>>>>> the
>>>>> threads stuck on the ZFS vnode lock to complete.
>>>>>
>>>>> rick
>>>>>
>>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>
>>>>> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
>>>>>
>>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>> _cv_wait_sig+0x16a
>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>> fork_trampoline+0xe
>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>>> nfsrvd_dorpc+0xc76
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>
>>>>> This one (and a few others) are waiting for the nfsv4_lock.
>>>>> This
>>>>> happens
>>>>> because other threads are stuck with RPCs in progress. (ie. The
>>>>> ones
>>>>> waiting on the vnode lock in zfs_fhtovp().)
>>>>> For these, the RPC needs to lock out other threads to do the
>>>>> operation,
>>>>> so it waits for the nfsv4_lock() which can exclusively lock the
>>>>> NFSv4
>>>>> data structures once all other nfsd threads complete their RPCs
>>>>> in
>>>>> progress.
>>>>>
>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>
>>>>> Same as above.
>>>>>
>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>> zfs_fhtovp+0x38d
>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>> svc_thread_start+0xb
>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>
>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>>>>>
>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
>>>>> fork_trampoline+0xe
>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>> zfs_fhtovp+0x38d
>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>> svc_thread_start+0xb
>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 15 décembre 2014 13:29 "Loïc Blot"
>>>>> <loic.blot at unix-experience.fr>
>>>>> a
>>>>> écrit:
>>>>>
>>>>> Hmmm...
>>>>> now i'm experiencing a deadlock.
>>>>>
>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
>>>>> (nfsd)
>>>>>
>>>>> the only issue was to reboot the server, but after rebooting
>>>>> deadlock arrives a second time when i
>>>>> start my jails over NFS.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 15 décembre 2014 10:07 "Loïc Blot"
>>>>> <loic.blot at unix-experience.fr>
>>>>> a
>>>>> écrit:
>>>>>
>>>>> Hi Rick,
>>>>> after talking with my N+1, NFSv4 is required on our
>>>>> infrastructure.
>>>>> I tried to upgrade NFSv4+ZFS
>>>>> server from 9.3 to 10.1, i hope this will resolve some
>>>>> issues...
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 10 décembre 2014 15:36 "Loïc Blot"
>>>>> <loic.blot at unix-experience.fr>
>>>>> a
>>>>> écrit:
>>>>>
>>>>> Hi Rick,
>>>>> thanks for your suggestion.
>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
>>>>> the
>>>>> server. kill -9 doesn't affect the
>>>>> process, it's blocked.... (State: Ds)
>>>>>
>>>>> for the performances
>>>>>
>>>>> NFSv3: 60Mbps
>>>>> NFSv4: 45Mbps
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>> a
>>>>> écrit:
>>>>>
>>>>> Loic Blot wrote:
>>>>>
>>>>>> Hi Rick,
>>>>>> I'm trying NFSv3.
>>>>>> Some jails are starting very well but now i have an issue
>>>>>> with
>>>>>> lockd
>>>>>> after some minutes:
>>>>>>
>>>>>> nfs server 10.10.X.8:/jails: lockd not responding
>>>>>> nfs server 10.10.X.8:/jails lockd is alive again
>>>>>>
>>>>>> I look at mbuf, but i seems there is no problem.
>>>>>
>>>>> Well, if you need locks to be visible across multiple
>>>>> clients,
>>>>> then
>>>>> I'm afraid you are stuck with using NFSv4 and the
>>>>> performance
>>>>> you
>>>>> get
>>>>> from it. (There is no way to do file handle affinity for
>>>>> NFSv4
>>>>> because
>>>>> the read and write ops are buried in the compound RPC and
>>>>> not
>>>>> easily
>>>>> recognized.)
>>>>>
>>>>> If the locks don't need to be visible across multiple
>>>>> clients,
>>>>> I'd
>>>>> suggest trying the "nolockd" option with nfsv3.
>>>>>
>>>>>> Here is my rc.conf on server:
>>>>>>
>>>>>> nfs_server_enable="YES"
>>>>>> nfsv4_server_enable="YES"
>>>>>> nfsuserd_enable="YES"
>>>>>> nfsd_server_flags="-u -t -n 256"
>>>>>> mountd_enable="YES"
>>>>>> mountd_flags="-r"
>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>> rpcbind_enable="YES"
>>>>>> rpc_lockd_enable="YES"
>>>>>> rpc_statd_enable="YES"
>>>>>>
>>>>>> Here is the client:
>>>>>>
>>>>>> nfsuserd_enable="YES"
>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>> nfscbd_enable="YES"
>>>>>> rpc_lockd_enable="YES"
>>>>>> rpc_statd_enable="YES"
>>>>>>
>>>>>> Have you got an idea ?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Loïc Blot,
>>>>>> UNIX Systems, Network and Security Engineer
>>>>>> http://www.unix-experience.fr
>>>>>>
>>>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>>> a
>>>>>> écrit:
>>>>>>> Loic Blot wrote:
>>>>>>>
>>>>>>>> Hi rick,
>>>>>>>>
>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
>>>>>>>> sysrc
>>>>>>>> memcached_flags="-v -m 512"
>>>>>>>> Command was very very slow...
>>>>>>>>
>>>>>>>> Here is a dd over NFS:
>>>>>>>>
>>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>>>>>>> bytes/sec)
>>>>>>>
>>>>>>> Can you try the same read using an NFSv3 mount?
>>>>>>> (If it runs much faster, you have probably been bitten by
>>>>>>> the
>>>>>>> ZFS
>>>>>>> "sequential vs random" read heuristic which I've been told
>>>>>>> things
>>>>>>> NFS is doing "random" reads without file handle affinity.
>>>>>>> File
>>>>>>> handle affinity is very hard to do for NFSv4, so it isn't
>>>>>>> done.)
>>>>>
>>>>> I was actually suggesting that you try the "dd" over nfsv3
>>>>> to
>>>>> see
>>>>> how
>>>>> the performance compared with nfsv4. If you do that, please
>>>>> post
>>>>> the
>>>>> comparable results.
>>>>>
>>>>> Someday I would like to try and get ZFS's sequential vs
>>>>> random
>>>>> read
>>>>> heuristic modified and any info on what difference in
>>>>> performance
>>>>> that
>>>>> might make for NFS would be useful.
>>>>>
>>>>> rick
>>>>>
>>>>> rick
>>>>>
>>>>> This is quite slow...
>>>>>
>>>>> You can found some nfsstat below (command isn't finished
>>>>> yet)
>>>>>
>>>>> nfsstat -c -w 1
>>>>>
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 0 0 0 0 0 16 0
>>>>> 2 0 0 0 0 0 17 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 4 0 0 0 0 4 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 0 0 0 0 0 3 0
>>>>> 0 0 0 0 0 0 3 0
>>>>> 37 10 0 8 0 0 14 1
>>>>> 18 16 0 4 1 2 4 0
>>>>> 78 91 0 82 6 12 30 0
>>>>> 19 18 0 2 2 4 2 0
>>>>> 0 0 0 0 2 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 1 0 0 0 0 1 0
>>>>> 4 6 0 0 6 0 3 0
>>>>> 2 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 1 0 0 0 0 0 0 0
>>>>> 0 0 0 0 1 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 6 108 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 98 54 0 86 11 0 25 0
>>>>> 36 24 0 39 25 0 10 1
>>>>> 67 8 0 63 63 0 41 0
>>>>> 34 0 0 35 34 0 0 0
>>>>> 75 0 0 75 77 0 0 0
>>>>> 34 0 0 35 35 0 0 0
>>>>> 75 0 0 74 76 0 0 0
>>>>> 33 0 0 34 33 0 0 0
>>>>> 0 0 0 0 5 0 0 0
>>>>> 0 0 0 0 0 0 6 0
>>>>> 11 0 0 0 0 0 11 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 17 0 0 0 0 1 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 4 5 0 0 0 0 12 0
>>>>> 2 0 0 0 0 0 26 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 4 0 0 0 0 4 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 0 0 0 0 0 2 0
>>>>> 2 0 0 0 0 0 24 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 0 0 0 0 0 7 0
>>>>> 2 1 0 0 0 0 1 0
>>>>> 0 0 0 0 2 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 6 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 6 0 0 0 0 3 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 2 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 71 0 0 0 0 0 0
>>>>> 0 1 0 0 0 0 0 0
>>>>> 2 36 0 0 0 0 1 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 1 0 0 0 0 0 1 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 79 6 0 79 79 0 2 0
>>>>> 25 0 0 25 26 0 6 0
>>>>> 43 18 0 39 46 0 23 0
>>>>> 36 0 0 36 36 0 31 0
>>>>> 68 1 0 66 68 0 0 0
>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>> 36 0 0 36 36 0 0 0
>>>>> 48 0 0 48 49 0 0 0
>>>>> 20 0 0 20 20 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 3 14 0 1 0 0 11 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 0 4 0 0 0 0 4 0
>>>>> 0 0 0 0 0 0 0 0
>>>>> 4 22 0 0 0 0 16 0
>>>>> 2 0 0 0 0 0 23 0
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 8 décembre 2014 09:36 "Loïc Blot"
>>>>> <loic.blot at unix-experience.fr> a
>>>>> écrit:
>>>>>> Hi Rick,
>>>>>> I stopped the jails this week-end and started it this
>>>>>> morning,
>>>>>> i'll
>>>>>> give you some stats this week.
>>>>>>
>>>>>> Here is my nfsstat -m output (with your rsize/wsize
>>>>>> tweaks)
>>>>
>>>>
>>>
>>
> nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
>>>>
>>>>>
>>>>
>>>>
>>>
>>
> etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
>>>>
>>>>> s=2147483647
>>>>>
>>>>> On server side my disks are on a raid controller which show a
>>>>> 512b
>>>>> volume and write performances
>>>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
>>>>> count=100000000 => 450MBps)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>>
>>>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>>> écrit:
>>>>>
>>>>> Loic Blot wrote:
>>>>>
>>>>> Hi,
>>>>> i'm trying to create a virtualisation environment based on
>>>>> jails.
>>>>> Those jails are stored under a big ZFS pool on a FreeBSD
>>>>> 9.3
>>>>> which
>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
>>>>> big
>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
>>>>> only 1
>>>>> was
>>>>> used at this time).
>>>>>
>>>>> The problem is simple, my hypervisors runs 6 jails (used 1%
>>>>> cpu
>>>>> and
>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and
>>>>> works
>>>>> fine at start but the system slows down and after 2-3 days
>>>>> become
>>>>> unusable. When i look at top command i see 80-100% on
>>>>> system
>>>>> and
>>>>> commands are very very slow. Many process are tagged with
>>>>> nfs_cl*.
>>>>>
>>>>> To be honest, I would expect the slowness to be because of
>>>>> slow
>>>>> response
>>>>> from the NFSv4 server, but if you do:
>>>>> # ps axHl
>>>>> on a client when it is slow and post that, it would give us
>>>>> some
>>>>> more
>>>>> information on where the client side processes are sitting.
>>>>> If you also do something like:
>>>>> # nfsstat -c -w 1
>>>>> and let it run for a while, that should show you how many
>>>>> RPCs
>>>>> are
>>>>> being done and which ones.
>>>>>
>>>>> # nfsstat -m
>>>>> will show you what your mount is actually using.
>>>>> The only mount option I can suggest trying is
>>>>> "rsize=32768,wsize=32768",
>>>>> since some network environments have difficulties with 64K.
>>>>>
>>>>> There are a few things you can try on the NFSv4 server side,
>>>>> if
>>>>> it
>>>>> appears
>>>>> that the clients are generating a large RPC load.
>>>>> - disabling the DRC cache for TCP by setting
>>>>> vfs.nfsd.cachetcp=0
>>>>> - If the server is seeing a large write RPC load, then
>>>>> "sync=disabled"
>>>>> might help, although it does run a risk of data loss when
>>>>> the
>>>>> server
>>>>> crashes.
>>>>> Then there are a couple of other ZFS related things (I'm not
>>>>> a
>>>>> ZFS
>>>>> guy,
>>>>> but these have shown up on the mailing lists).
>>>>> - make sure your volumes are 4K aligned and ashift=12 (in
>>>>> case a
>>>>> drive
>>>>> that uses 4K sectors is pretending to be 512byte sectored)
>>>>> - never run over 70-80% full if write performance is an
>>>>> issue
>>>>> - use a zil on an SSD with good write performance
>>>>>
>>>>> The only NFSv4 thing I can tell you is that it is known that
>>>>> ZFS's
>>>>> algorithm for determining sequential vs random I/O fails for
>>>>> NFSv4
>>>>> during writing and this can be a performance hit. The only
>>>>> workaround
>>>>> is to use NFSv3 mounts, since file handle affinity
>>>>> apparently
>>>>> fixes
>>>>> the problem and this is only done for NFSv3.
>>>>>
>>>>> rick
>>>>>
>>>>> I saw that there are TSO issues with igb then i'm trying to
>>>>> disable
>>>>> it with sysctl but the situation wasn't solved.
>>>>>
>>>>> Someone has got ideas ? I can give you more informations if
>>>>> you
>>>>> need.
>>>>>
>>>>> Thanks in advance.
>>>>> Regards,
>>>>>
>>>>> Loïc Blot,
>>>>> UNIX Systems, Network and Security Engineer
>>>>> http://www.unix-experience.fr
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-fs at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list