High Kernel Load with nfsv4
Rick Macklem
rmacklem at uoguelph.ca
Tue Jan 6 03:17:25 UTC 2015
Loic Blot wrote:
> Hi Rick,
> nfsstat -e -s don't show usefull datas on server.
>
Well, as far as I know, it returns valid information.
(See below.)
> Server Info:
> Getattr Setattr Lookup Readlink Read Write Create
> Remove
> 26935254 16911 5755728 302 2334920 3673866 0
> 328332
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> Access
> 77980 28 0 0 3 8900 3
> 1806052
> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId
> SetClIdCf
> 1 1095 0 0 614377 8172 8
> 8
> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH
> Lock
> 1595299 0 44145 1495 0 0 5197490
> 635015
> LockT LockU Close Verify NVerify PutFH PutPubFH
> PutRootFH
> 0 614919 1270938 0 0 22688676 0
> 5
> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
> 42104 197606 275820 0 143 4578
> Server:
> Retfailed Faults Clients
> 0 0 6
> OpenOwner Opens LockOwner Locks Delegs
> 32335 145448 204 181 0
Well, 145448 Opens are a lot of Open files. Each of these uses
a kernel malloc'd data structure that is linked into multiple
linked lists.
The question is..why aren't these Opens being closed?
Since FreeBSD does I/O on an mmap'd file after closing it,
the FreeBSD NFSv4 client is forced to delay doing Close RPCs
until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d. (The
VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't
guaranteed to be called.)
Since there were about 1.5 million Opens and 1.27 million
Closes, it does appear that Opens are being Closed.
Now, I'm not sure I would have imagined 1.5million file Opens
in a few days. My guess is this is the bottleneck.
I'd suggest that you do:
# nfsstat -e -c
on each of the NFSv4 clients and see how many Opens/client
there are. I vaguely remember an upper limit in the client,
but can't remember what it is set to.
--> I suspect the client Open/Lock limit needs to be increased.
(I can't remember if the server also has a limit, but I
think it does.)
Then the size of the hash tables used to search the Opens
may also need to be increased a lot.
Also, I'd suggest you take a look at whatever apps. are
running on the client(s) and try to figure out why they
are Opening so many files?
My guess is that the client(s) are gettig bogged down by all
these Opens.
> Server Cache Stats:
> Inprog Idem Non-idem Misses CacheSize TCPPeak
> 0 0 1 15082947 60 16522
>
> Only GetAttr and Lookup increase and it's only every 4-5 seconds and
> only +2 to +5 into theses values.
>
> Now on client, if i take four processes stack i got
>
> PID TID COMM TDNAME KSTACK
> 63170 102547 mv - mi_switch+0xe1
> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351
> Xfast_syscall+0xfb
>
> Another mv:
> 63140 101738 mv - mi_switch+0xe1
> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
> Xfast_syscall+0xfb
>
> 62070 102170 sendmail - mi_switch+0xe1
> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666
> clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72
> nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1
> lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30
> amd64_syscall+0x351 Xfast_syscall+0xfb
>
> 63200 100930 mv - mi_switch+0xe1
> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
> Xfast_syscall+0xfb
>
The above simply says that thread 102710 is waiting for a Lookup
reply from the server and the other 3 are waiting for the mutex
that protects the state structures in the client. (I suspect
some other thread in the client is wading through the Open list,
if a single client has a lot of these 145K Opens.)
> When client is in this state, server was doing nothing special
> (procstat -kk)
>
> PID TID COMM TDNAME KSTACK
> 895 100538 nfsd nfsd: master mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> amd64_syscall+0x351 Xfast_syscall+0xfb
> 895 100568 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100569 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100570 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100571 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100572 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100573 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100575 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100576 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100577 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100578 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100579 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100580 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100581 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100582 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100583 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100584 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100585 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100586 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100587 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100588 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100589 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100590 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100592 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100593 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100594 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100595 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100596 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100597 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100598 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100599 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100600 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100602 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100603 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100604 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100605 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100606 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100607 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100608 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100609 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100610 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100611 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100612 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100613 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100614 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100615 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100617 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100618 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100619 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100621 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100622 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100623 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100624 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100625 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100626 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100627 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100628 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100629 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100630 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100631 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100632 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100633 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100634 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100635 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100636 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100638 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100639 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100640 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100641 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100642 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100643 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100644 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100645 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100646 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100647 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100648 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100649 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100651 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100652 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100653 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100654 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100655 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100656 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100657 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100658 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100659 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100661 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100662 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100684 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100685 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100686 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100797 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100798 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100799 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100800 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 895 100801 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
>
> I really think it's a client side problem, maybe a lookup problem.
>
> Regards,
>
> Loïc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>
> 5 janvier 2015 14:35 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
> > Loic Blot wrote:
> >
> >> Hi,
> >> happy new year Rick and @freebsd-fs.
> >>
> >> After some days, i looked my NFSv4.1 mount. At server start it was
> >> calm, but after 4 days, here is the top stat...
> >>
> >> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0%
> >> idle
> >>
> >> Definitively i think it's a problem on client side. What can i
> >> look
> >> into running kernel to resolve this issue ?
> >
> > Well, I'd start with:
> > # nfsstat -e -s
> > - run repeatedly on the server (once every N seconds in a loop).
> > Then look at the output, comparing the counts and see which RPCs
> > are being performed by the client(s). You are looking for which
> > RPCs are being done a lot. (If one RPC is almost 100% of the load,
> > then it might be a client/caching issue for whatever that RPC is
> > doing.)
> >
> > Also look at the Open/Lock counts near the end of the output.
> > If the # of Opens/Locks is large, it may be possible to reduce the
> > CPU overheads by using larger hash tables.
> >
> > Then you need to profile the server kernel to see where the CPU
> > is being used.
> > Hopefully someone else can fill you in on how to do that, because
> > I'll admit I don't know how to.
> > Basically you are looking to see if the CPU is being used in
> > the NFS server code or ZFS.
> >
> > Good luck with it, rick
> >
> >> Regards,
> >>
> >> Loïc Blot,
> >> UNIX Systems, Network and Security Engineer
> >> http://www.unix-experience.fr
> >>
> >> 30 décembre 2014 16:16 "Loïc Blot" <loic.blot at unix-experience.fr>
> >> a
> >> écrit:
> >>> Hi Rick,
> >>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1
> >>> (mountoptions:
> >>> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)
> >>>
> >>> Performance is quite stable but it's slow. Not as slow as before
> >>> but slow... services was launched
> >>> but no client are using them and system CPU % was 10-50%.
> >>>
> >>> I don't see anything on NFSv4.1 server, it's perfectly stable and
> >>> functionnal.
> >>>
> >>> Regards,
> >>>
> >>> Loïc Blot,
> >>> UNIX Systems, Network and Security Engineer
> >>> http://www.unix-experience.fr
> >>>
> >>> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>> écrit:
> >>>
> >>>> Loic Blot wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> To clarify because of our exchanges. Here are the current
> >>>>> sysctl
> >>>>> options for server:
> >>>>>
> >>>>> vfs.nfsd.enable_nobodycheck=0
> >>>>> vfs.nfsd.enable_nogroupcheck=0
> >>>>>
> >>>>> vfs.nfsd.maxthreads=200
> >>>>> vfs.nfsd.tcphighwater=10000
> >>>>> vfs.nfsd.tcpcachetimeo=300
> >>>>> vfs.nfsd.server_min_nfsvers=4
> >>>>>
> >>>>> kern.maxvnodes=10000000
> >>>>> kern.ipc.maxsockbuf=4194304
> >>>>> net.inet.tcp.sendbuf_max=4194304
> >>>>> net.inet.tcp.recvbuf_max=4194304
> >>>>>
> >>>>> vfs.lookup_shared=0
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 22 décembre 2014 09:42 "Loïc Blot"
> >>>>> <loic.blot at unix-experience.fr>
> >>>>> a
> >>>>> écrit:
> >>>>>
> >>>>> Hi Rick,
> >>>>> my 5 jails runs this weekend and now i have some stats on this
> >>>>> monday.
> >>>>>
> >>>>> Hopefully deadlock was fixed, yeah, but everything isn't good
> >>>>> :(
> >>>>>
> >>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
> >>>>>
> >>>>> As i can see this is because of nfsd:
> >>>>>
> >>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
> >>>>> 273.68% nfsd: server (nfsd)
> >>>>>
> >>>>> If i look at dmesg i see:
> >>>>> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
> >>>>
> >>>> Well, you have a couple of choices:
> >>>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
> >>>> (NFSv4.1 avoids use of the DRC and instead uses something
> >>>> called sessions. See below.)
> >>>> OR
> >>>>
> >>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
> >>>>
> >>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
> >>>> "nfs server cache flooded" messages. (I think Garrett Wollman
> >>>> uses
> >>>> 100000. (You may still see quite a bit of CPU overheads.)
> >>>>
> >>>> OR
> >>>>
> >>>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid
> >>>> of the CPU overheads). However, there is a risk of data
> >>>> corruption
> >>>> if you have a client->server network partitioning of a moderate
> >>>> duration, because a non-idempotent RPC may get redone, becasue
> >>>> the client times out waiting for a reply. If a non-idempotent
> >>>> RPC gets done twice on the server, data corruption can happen.
> >>>> (The DRC provides improved correctness, but does add overhead.)
> >>>>
> >>>> If #1 works for you, it is the preferred solution, since
> >>>> Sessions
> >>>> in NFSv4.1 solves the correctness problem in a good, space bound
> >>>> way. A session basically has N (usually 32 or 64) slots and only
> >>>> allows one outstanding RPC/slot. As such, it can cache the
> >>>> previous
> >>>> reply for each slot (32 or 64 of them) and guarantee "exactly
> >>>> once"
> >>>> RPC semantics.
> >>>>
> >>>> rick
> >>>>
> >>>>> Here is 'nfsstat -s' output:
> >>>>>
> >>>>> Server Info:
> >>>>> Getattr Setattr Lookup Readlink Read Write Create
> >>>>> Remove
> >>>>> 12600652 1812 2501097 156 1386423 1983729 123
> >>>>> 162067
> >>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> >>>>> Access
> >>>>> 36762 9 0 0 0 3147 0
> >>>>> 623524
> >>>>> Mknod Fsstat Fsinfo PathConf Commit
> >>>>> 0 0 0 0 328117
> >>>>> Server Ret-Failed
> >>>>> 0
> >>>>> Server Faults
> >>>>> 0
> >>>>> Server Cache Stats:
> >>>>> Inprog Idem Non-idem Misses
> >>>>> 0 0 0 12635512
> >>>>> Server Write Gathering:
> >>>>> WriteOps WriteRPC Opsaved
> >>>>> 1983729 1983729 0
> >>>>>
> >>>>> And here is 'procstat -kk' for nfsd (server)
> >>>>>
> >>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
> >>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1
> >>>>> svc_run+0x1de
> >>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
> >>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> ---
> >>>>>
> >>>>> Now if we look at client (FreeBSD 9.3)
> >>>>>
> >>>>> We see system was very busy and do many and many interrupts
> >>>>>
> >>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
> >>>>> idle
> >>>>>
> >>>>> A look at process list shows that there are many sendmail
> >>>>> process
> >>>>> in
> >>>>> state nfstry
> >>>>>
> >>>>> nfstry 18 32:27 0.88% sendmail: Queue runner at 00:30:00 for
> >>>>> /var/spool/clientm
> >>>>>
> >>>>> Here is 'nfsstat -c' output:
> >>>>>
> >>>>> Client Info:
> >>>>> Rpc Counts:
> >>>>> Getattr Setattr Lookup Readlink Read Write Create
> >>>>> Remove
> >>>>> 1051347 1724 2494481 118 903902 1901285 162676
> >>>>> 161899
> >>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
> >>>>> Access
> >>>>> 36744 2 0 114 40 3131 0
> >>>>> 544136
> >>>>> Mknod Fsstat Fsinfo PathConf Commit
> >>>>> 9 0 0 0 245821
> >>>>> Rpc Info:
> >>>>> TimedOut Invalid X Replies Retries Requests
> >>>>> 0 0 0 0 8356557
> >>>>> Cache Info:
> >>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
> >>>>> Misses
> >>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
> >>>>> 1871871
> >>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
> >>>>> Misses
> >>>>> 144035 118 53736 2753 27813 1 57238839
> >>>>> 544205
> >>>>>
> >>>>> If you need more things, tell me, i let the PoC in this state.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>>>> écrit:
> >>>>>
> >>>>> Loic Blot wrote:
> >>>>>
> >>>>>> Hi Rick,
> >>>>>> ok, i don't need locallocks, i haven't understand option was
> >>>>>> for
> >>>>>> that
> >>>>>> usage, i removed it.
> >>>>>> I do more tests on monday.
> >>>>>> Thanks for the deadlock fix, for other people :)
> >>>>>
> >>>>> Good. Please let us know if running with
> >>>>> vfs.nfsd.enable_locallocks=0
> >>>>> gets rid of the deadlocks? (I think it fixes the one you saw.)
> >>>>>
> >>>>> On the performance side, you might also want to try different
> >>>>> values
> >>>>> of
> >>>>> readahead, if the Linux client has such a mount option. (With
> >>>>> the
> >>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea
> >>>>> what
> >>>>> the
> >>>>> optimal readahead value would be.)
> >>>>>
> >>>>> Good luck with it and please let us know how it goes, rick
> >>>>> ps: I now have a patch to fix the deadlock when
> >>>>> vfs.nfsd.enable_locallocks=1
> >>>>> is set. I'll post it for anyone who is interested after I put
> >>>>> it
> >>>>> through some testing.
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>> Loïc BLOT,
> >>>>> UNIX systems, security and network engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
> >>>>>
> >>>>> Loic Blot wrote:
> >>>>>> Hi rick,
> >>>>>> i tried to start a LXC container on Debian Squeeze from my
> >>>>>> freebsd
> >>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
> >>>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
> >>>>>> squeeze
> >>>>>> container, it seems (3 tries, 3 fails).
> >>>>>
> >>>>> Well, I`ll take a look at this `procstat -kk`, but the only
> >>>>> thing
> >>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
> >>>>> nullfs. (I have no idea if you are using any nullfs mounts, but
> >>>>> if so, try getting rid of them.)
> >>>>>
> >>>>> Here`s a high level post about the ZFS and vnode locking
> >>>>> problem,
> >>>>> but there is no patch available, as far as I know.
> >>>>>
> >>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
> >>>>>
> >>>>> rick
> >>>>>
> >>>>> 921 - D 0:00.02 nfsd: server (nfsd)
> >>>>>
> >>>>> Here is the procstat -kk
> >>>>>
> >>>>> PID TID COMM TDNAME KSTACK
> >>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> >>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
> >>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> >>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>> _cv_wait_sig+0x16a
> >>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> >>>>> nfsrvd_dorpc+0xc76
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>>>> écrit:
> >>>>>
> >>>>> Loic Blot wrote:
> >>>>>
> >>>>>> For more informations, here is procstat -kk on nfsd, if you
> >>>>>> need
> >>>>>> more
> >>>>>> hot datas, tell me.
> >>>>>>
> >>>>>> Regards, PID TID COMM TDNAME KSTACK
> >>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
> >>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> >>>>>> amd64_syscall+0x351
> >>>>>
> >>>>> Well, most of the threads are stuck like this one, waiting for
> >>>>> a
> >>>>> vnode
> >>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
> >>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
> >>>>> subject line
> >>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
> >>>>> look.
> >>>>>
> >>>>> The only thing I`ve seen suggested is trying:
> >>>>> sysctl vfs.lookup_shared=0
> >>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
> >>>>> doesn`t
> >>>>> obey the vnode locking rules for lookup and rename, according
> >>>>> to
> >>>>> the posting I saw.
> >>>>>
> >>>>> I`ve added a couple of comments about the other threads below,
> >>>>> but
> >>>>> they are all either waiting for an RPC request or waiting for
> >>>>> the
> >>>>> threads stuck on the ZFS vnode lock to complete.
> >>>>>
> >>>>> rick
> >>>>>
> >>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>
> >>>>> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
> >>>>>
> >>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> >>>>>> _cv_wait_sig+0x16a
> >>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> >>>>>> fork_trampoline+0xe
> >>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> >>>>>> nfsrvd_dorpc+0xc76
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>
> >>>>> This one (and a few others) are waiting for the nfsv4_lock.
> >>>>> This
> >>>>> happens
> >>>>> because other threads are stuck with RPCs in progress. (ie. The
> >>>>> ones
> >>>>> waiting on the vnode lock in zfs_fhtovp().)
> >>>>> For these, the RPC needs to lock out other threads to do the
> >>>>> operation,
> >>>>> so it waits for the nfsv4_lock() which can exclusively lock the
> >>>>> NFSv4
> >>>>> data structures once all other nfsd threads complete their RPCs
> >>>>> in
> >>>>> progress.
> >>>>>
> >>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>>
> >>>>> Same as above.
> >>>>>
> >>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
> >>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>>> zfs_fhtovp+0x38d
> >>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>>> svc_thread_start+0xb
> >>>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>
> >>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
> >>>>>
> >>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
> >>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
> >>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
> >>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
> >>>>> fork_trampoline+0xe
> >>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> >>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
> >>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> >>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> >>>>> zfs_fhtovp+0x38d
> >>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> >>>>> nfssvc_program+0x554 svc_run_internal+0xc77
> >>>>> svc_thread_start+0xb
> >>>>> fork_exit+0x9a fork_trampoline+0xe
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 15 décembre 2014 13:29 "Loïc Blot"
> >>>>> <loic.blot at unix-experience.fr>
> >>>>> a
> >>>>> écrit:
> >>>>>
> >>>>> Hmmm...
> >>>>> now i'm experiencing a deadlock.
> >>>>>
> >>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
> >>>>> (nfsd)
> >>>>>
> >>>>> the only issue was to reboot the server, but after rebooting
> >>>>> deadlock arrives a second time when i
> >>>>> start my jails over NFS.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 15 décembre 2014 10:07 "Loïc Blot"
> >>>>> <loic.blot at unix-experience.fr>
> >>>>> a
> >>>>> écrit:
> >>>>>
> >>>>> Hi Rick,
> >>>>> after talking with my N+1, NFSv4 is required on our
> >>>>> infrastructure.
> >>>>> I tried to upgrade NFSv4+ZFS
> >>>>> server from 9.3 to 10.1, i hope this will resolve some
> >>>>> issues...
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 10 décembre 2014 15:36 "Loïc Blot"
> >>>>> <loic.blot at unix-experience.fr>
> >>>>> a
> >>>>> écrit:
> >>>>>
> >>>>> Hi Rick,
> >>>>> thanks for your suggestion.
> >>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
> >>>>> the
> >>>>> server. kill -9 doesn't affect the
> >>>>> process, it's blocked.... (State: Ds)
> >>>>>
> >>>>> for the performances
> >>>>>
> >>>>> NFSv3: 60Mbps
> >>>>> NFSv4: 45Mbps
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
> >>>>> a
> >>>>> écrit:
> >>>>>
> >>>>> Loic Blot wrote:
> >>>>>
> >>>>>> Hi Rick,
> >>>>>> I'm trying NFSv3.
> >>>>>> Some jails are starting very well but now i have an issue
> >>>>>> with
> >>>>>> lockd
> >>>>>> after some minutes:
> >>>>>>
> >>>>>> nfs server 10.10.X.8:/jails: lockd not responding
> >>>>>> nfs server 10.10.X.8:/jails lockd is alive again
> >>>>>>
> >>>>>> I look at mbuf, but i seems there is no problem.
> >>>>>
> >>>>> Well, if you need locks to be visible across multiple
> >>>>> clients,
> >>>>> then
> >>>>> I'm afraid you are stuck with using NFSv4 and the
> >>>>> performance
> >>>>> you
> >>>>> get
> >>>>> from it. (There is no way to do file handle affinity for
> >>>>> NFSv4
> >>>>> because
> >>>>> the read and write ops are buried in the compound RPC and
> >>>>> not
> >>>>> easily
> >>>>> recognized.)
> >>>>>
> >>>>> If the locks don't need to be visible across multiple
> >>>>> clients,
> >>>>> I'd
> >>>>> suggest trying the "nolockd" option with nfsv3.
> >>>>>
> >>>>>> Here is my rc.conf on server:
> >>>>>>
> >>>>>> nfs_server_enable="YES"
> >>>>>> nfsv4_server_enable="YES"
> >>>>>> nfsuserd_enable="YES"
> >>>>>> nfsd_server_flags="-u -t -n 256"
> >>>>>> mountd_enable="YES"
> >>>>>> mountd_flags="-r"
> >>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
> >>>>>> rpcbind_enable="YES"
> >>>>>> rpc_lockd_enable="YES"
> >>>>>> rpc_statd_enable="YES"
> >>>>>>
> >>>>>> Here is the client:
> >>>>>>
> >>>>>> nfsuserd_enable="YES"
> >>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
> >>>>>> nfscbd_enable="YES"
> >>>>>> rpc_lockd_enable="YES"
> >>>>>> rpc_statd_enable="YES"
> >>>>>>
> >>>>>> Have you got an idea ?
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Loïc Blot,
> >>>>>> UNIX Systems, Network and Security Engineer
> >>>>>> http://www.unix-experience.fr
> >>>>>>
> >>>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
> >>>>>> a
> >>>>>> écrit:
> >>>>>>> Loic Blot wrote:
> >>>>>>>
> >>>>>>>> Hi rick,
> >>>>>>>>
> >>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
> >>>>>>>> sysrc
> >>>>>>>> memcached_flags="-v -m 512"
> >>>>>>>> Command was very very slow...
> >>>>>>>>
> >>>>>>>> Here is a dd over NFS:
> >>>>>>>>
> >>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
> >>>>>>>> bytes/sec)
> >>>>>>>
> >>>>>>> Can you try the same read using an NFSv3 mount?
> >>>>>>> (If it runs much faster, you have probably been bitten by
> >>>>>>> the
> >>>>>>> ZFS
> >>>>>>> "sequential vs random" read heuristic which I've been told
> >>>>>>> things
> >>>>>>> NFS is doing "random" reads without file handle affinity.
> >>>>>>> File
> >>>>>>> handle affinity is very hard to do for NFSv4, so it isn't
> >>>>>>> done.)
> >>>>>
> >>>>> I was actually suggesting that you try the "dd" over nfsv3
> >>>>> to
> >>>>> see
> >>>>> how
> >>>>> the performance compared with nfsv4. If you do that, please
> >>>>> post
> >>>>> the
> >>>>> comparable results.
> >>>>>
> >>>>> Someday I would like to try and get ZFS's sequential vs
> >>>>> random
> >>>>> read
> >>>>> heuristic modified and any info on what difference in
> >>>>> performance
> >>>>> that
> >>>>> might make for NFS would be useful.
> >>>>>
> >>>>> rick
> >>>>>
> >>>>> rick
> >>>>>
> >>>>> This is quite slow...
> >>>>>
> >>>>> You can found some nfsstat below (command isn't finished
> >>>>> yet)
> >>>>>
> >>>>> nfsstat -c -w 1
> >>>>>
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 0 0 0 0 0 16 0
> >>>>> 2 0 0 0 0 0 17 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 4 0 0 0 0 4 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 0 0 0 0 0 3 0
> >>>>> 0 0 0 0 0 0 3 0
> >>>>> 37 10 0 8 0 0 14 1
> >>>>> 18 16 0 4 1 2 4 0
> >>>>> 78 91 0 82 6 12 30 0
> >>>>> 19 18 0 2 2 4 2 0
> >>>>> 0 0 0 0 2 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 1 0 0 0 0 1 0
> >>>>> 4 6 0 0 6 0 3 0
> >>>>> 2 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 1 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 1 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 6 108 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 98 54 0 86 11 0 25 0
> >>>>> 36 24 0 39 25 0 10 1
> >>>>> 67 8 0 63 63 0 41 0
> >>>>> 34 0 0 35 34 0 0 0
> >>>>> 75 0 0 75 77 0 0 0
> >>>>> 34 0 0 35 35 0 0 0
> >>>>> 75 0 0 74 76 0 0 0
> >>>>> 33 0 0 34 33 0 0 0
> >>>>> 0 0 0 0 5 0 0 0
> >>>>> 0 0 0 0 0 0 6 0
> >>>>> 11 0 0 0 0 0 11 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 17 0 0 0 0 1 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 4 5 0 0 0 0 12 0
> >>>>> 2 0 0 0 0 0 26 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 4 0 0 0 0 4 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 0 0 0 0 0 2 0
> >>>>> 2 0 0 0 0 0 24 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 0 0 0 0 0 7 0
> >>>>> 2 1 0 0 0 0 1 0
> >>>>> 0 0 0 0 2 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 6 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 6 0 0 0 0 3 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 2 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 71 0 0 0 0 0 0
> >>>>> 0 1 0 0 0 0 0 0
> >>>>> 2 36 0 0 0 0 1 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 1 0 0 0 0 0 1 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 79 6 0 79 79 0 2 0
> >>>>> 25 0 0 25 26 0 6 0
> >>>>> 43 18 0 39 46 0 23 0
> >>>>> 36 0 0 36 36 0 31 0
> >>>>> 68 1 0 66 68 0 0 0
> >>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> >>>>> 36 0 0 36 36 0 0 0
> >>>>> 48 0 0 48 49 0 0 0
> >>>>> 20 0 0 20 20 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 3 14 0 1 0 0 11 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 0 4 0 0 0 0 4 0
> >>>>> 0 0 0 0 0 0 0 0
> >>>>> 4 22 0 0 0 0 16 0
> >>>>> 2 0 0 0 0 0 23 0
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 8 décembre 2014 09:36 "Loïc Blot"
> >>>>> <loic.blot at unix-experience.fr> a
> >>>>> écrit:
> >>>>>> Hi Rick,
> >>>>>> I stopped the jails this week-end and started it this
> >>>>>> morning,
> >>>>>> i'll
> >>>>>> give you some stats this week.
> >>>>>>
> >>>>>> Here is my nfsstat -m output (with your rsize/wsize
> >>>>>> tweaks)
> >>>>
> >>>>
> >>>
> >>
> > nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
> >>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> > etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
> >>>>
> >>>>> s=2147483647
> >>>>>
> >>>>> On server side my disks are on a raid controller which show a
> >>>>> 512b
> >>>>> volume and write performances
> >>>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
> >>>>> count=100000000 => 450MBps)
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>>
> >>>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
> >>>>> écrit:
> >>>>>
> >>>>> Loic Blot wrote:
> >>>>>
> >>>>> Hi,
> >>>>> i'm trying to create a virtualisation environment based on
> >>>>> jails.
> >>>>> Those jails are stored under a big ZFS pool on a FreeBSD
> >>>>> 9.3
> >>>>> which
> >>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
> >>>>> big
> >>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
> >>>>> only 1
> >>>>> was
> >>>>> used at this time).
> >>>>>
> >>>>> The problem is simple, my hypervisors runs 6 jails (used 1%
> >>>>> cpu
> >>>>> and
> >>>>> 10GB RAM approximatively and less than 1MB bandwidth) and
> >>>>> works
> >>>>> fine at start but the system slows down and after 2-3 days
> >>>>> become
> >>>>> unusable. When i look at top command i see 80-100% on
> >>>>> system
> >>>>> and
> >>>>> commands are very very slow. Many process are tagged with
> >>>>> nfs_cl*.
> >>>>>
> >>>>> To be honest, I would expect the slowness to be because of
> >>>>> slow
> >>>>> response
> >>>>> from the NFSv4 server, but if you do:
> >>>>> # ps axHl
> >>>>> on a client when it is slow and post that, it would give us
> >>>>> some
> >>>>> more
> >>>>> information on where the client side processes are sitting.
> >>>>> If you also do something like:
> >>>>> # nfsstat -c -w 1
> >>>>> and let it run for a while, that should show you how many
> >>>>> RPCs
> >>>>> are
> >>>>> being done and which ones.
> >>>>>
> >>>>> # nfsstat -m
> >>>>> will show you what your mount is actually using.
> >>>>> The only mount option I can suggest trying is
> >>>>> "rsize=32768,wsize=32768",
> >>>>> since some network environments have difficulties with 64K.
> >>>>>
> >>>>> There are a few things you can try on the NFSv4 server side,
> >>>>> if
> >>>>> it
> >>>>> appears
> >>>>> that the clients are generating a large RPC load.
> >>>>> - disabling the DRC cache for TCP by setting
> >>>>> vfs.nfsd.cachetcp=0
> >>>>> - If the server is seeing a large write RPC load, then
> >>>>> "sync=disabled"
> >>>>> might help, although it does run a risk of data loss when
> >>>>> the
> >>>>> server
> >>>>> crashes.
> >>>>> Then there are a couple of other ZFS related things (I'm not
> >>>>> a
> >>>>> ZFS
> >>>>> guy,
> >>>>> but these have shown up on the mailing lists).
> >>>>> - make sure your volumes are 4K aligned and ashift=12 (in
> >>>>> case a
> >>>>> drive
> >>>>> that uses 4K sectors is pretending to be 512byte sectored)
> >>>>> - never run over 70-80% full if write performance is an
> >>>>> issue
> >>>>> - use a zil on an SSD with good write performance
> >>>>>
> >>>>> The only NFSv4 thing I can tell you is that it is known that
> >>>>> ZFS's
> >>>>> algorithm for determining sequential vs random I/O fails for
> >>>>> NFSv4
> >>>>> during writing and this can be a performance hit. The only
> >>>>> workaround
> >>>>> is to use NFSv3 mounts, since file handle affinity
> >>>>> apparently
> >>>>> fixes
> >>>>> the problem and this is only done for NFSv3.
> >>>>>
> >>>>> rick
> >>>>>
> >>>>> I saw that there are TSO issues with igb then i'm trying to
> >>>>> disable
> >>>>> it with sysctl but the situation wasn't solved.
> >>>>>
> >>>>> Someone has got ideas ? I can give you more informations if
> >>>>> you
> >>>>> need.
> >>>>>
> >>>>> Thanks in advance.
> >>>>> Regards,
> >>>>>
> >>>>> Loïc Blot,
> >>>>> UNIX Systems, Network and Security Engineer
> >>>>> http://www.unix-experience.fr
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>>>
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>>>
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>>>
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>>>
> >>>>> _______________________________________________
> >>>>> freebsd-fs at freebsd.org mailing list
> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>> To unsubscribe, send any mail to
> >>>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>>
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
>
More information about the freebsd-fs
mailing list