High Kernel Load with nfsv4

[Loïc Blot]

Hi,
happy new year Rick and @freebsd-fs.

After some days, i looked my NFSv4.1 mount. At server start it was calm, but after 4 days, here is the top stat...

CPU:  0.0% user,  0.0% nice,  100% system,  0.0% interrupt,  0.0% idle

Definitively i think it's a problem on client side. What can i look into running kernel to resolve this issue ?


Regards,

Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr

30 décembre 2014 16:16 "Loïc Blot" <loic.blot at unix-experience.fr> a écrit: 
> Hi Rick,
> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1 (mountoptions:
> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)
> 
> Performance is quite stable but it's slow. Not as slow as before but slow... services was launched
> but no client are using them and system CPU % was 10-50%.
> 
> I don't see anything on NFSv4.1 server, it's perfectly stable and functionnal.
> 
> Regards,
> 
> Loïc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
> 
> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
> 
>> Loic Blot wrote:
>> 
>>> Hi,
>>> 
>>> To clarify because of our exchanges. Here are the current sysctl
>>> options for server:
>>> 
>>> vfs.nfsd.enable_nobodycheck=0
>>> vfs.nfsd.enable_nogroupcheck=0
>>> 
>>> vfs.nfsd.maxthreads=200
>>> vfs.nfsd.tcphighwater=10000
>>> vfs.nfsd.tcpcachetimeo=300
>>> vfs.nfsd.server_min_nfsvers=4
>>> 
>>> kern.maxvnodes=10000000
>>> kern.ipc.maxsockbuf=4194304
>>> net.inet.tcp.sendbuf_max=4194304
>>> net.inet.tcp.recvbuf_max=4194304
>>> 
>>> vfs.lookup_shared=0
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 22 décembre 2014 09:42 "Loïc Blot" <loic.blot at unix-experience.fr> a
>>> écrit:
>>> 
>>> Hi Rick,
>>> my 5 jails runs this weekend and now i have some stats on this
>>> monday.
>>> 
>>> Hopefully deadlock was fixed, yeah, but everything isn't good :(
>>> 
>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>>> 
>>> As i can see this is because of nfsd:
>>> 
>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
>>> 273.68% nfsd: server (nfsd)
>>> 
>>> If i look at dmesg i see:
>>> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
>> 
>> Well, you have a couple of choices:
>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
>> (NFSv4.1 avoids use of the DRC and instead uses something
>> called sessions. See below.)
>> OR
>> 
>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
>> 
>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
>> "nfs server cache flooded" messages. (I think Garrett Wollman uses
>> 100000. (You may still see quite a bit of CPU overheads.)
>> 
>> OR
>> 
>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid
>> of the CPU overheads). However, there is a risk of data corruption
>> if you have a client->server network partitioning of a moderate
>> duration, because a non-idempotent RPC may get redone, becasue
>> the client times out waiting for a reply. If a non-idempotent
>> RPC gets done twice on the server, data corruption can happen.
>> (The DRC provides improved correctness, but does add overhead.)
>> 
>> If #1 works for you, it is the preferred solution, since Sessions
>> in NFSv4.1 solves the correctness problem in a good, space bound
>> way. A session basically has N (usually 32 or 64) slots and only
>> allows one outstanding RPC/slot. As such, it can cache the previous
>> reply for each slot (32 or 64 of them) and guarantee "exactly once"
>> RPC semantics.
>> 
>> rick
>> 
>>> Here is 'nfsstat -s' output:
>>> 
>>> Server Info:
>>> Getattr Setattr Lookup Readlink Read Write Create
>>> Remove
>>> 12600652 1812 2501097 156 1386423 1983729 123
>>> 162067
>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>> Access
>>> 36762 9 0 0 0 3147 0
>>> 623524
>>> Mknod Fsstat Fsinfo PathConf Commit
>>> 0 0 0 0 328117
>>> Server Ret-Failed
>>> 0
>>> Server Faults
>>> 0
>>> Server Cache Stats:
>>> Inprog Idem Non-idem Misses
>>> 0 0 0 12635512
>>> Server Write Gathering:
>>> WriteOps WriteRPC Opsaved
>>> 1983729 1983729 0
>>> 
>>> And here is 'procstat -kk' for nfsd (server)
>>> 
>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> ---
>>> 
>>> Now if we look at client (FreeBSD 9.3)
>>> 
>>> We see system was very busy and do many and many interrupts
>>> 
>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
>>> idle
>>> 
>>> A look at process list shows that there are many sendmail process in
>>> state nfstry
>>> 
>>> nfstry 18 32:27 0.88% sendmail: Queue runner at 00:30:00 for
>>> /var/spool/clientm
>>> 
>>> Here is 'nfsstat -c' output:
>>> 
>>> Client Info:
>>> Rpc Counts:
>>> Getattr Setattr Lookup Readlink Read Write Create
>>> Remove
>>> 1051347 1724 2494481 118 903902 1901285 162676
>>> 161899
>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>> Access
>>> 36744 2 0 114 40 3131 0
>>> 544136
>>> Mknod Fsstat Fsinfo PathConf Commit
>>> 9 0 0 0 245821
>>> Rpc Info:
>>> TimedOut Invalid X Replies Retries Requests
>>> 0 0 0 0 8356557
>>> Cache Info:
>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
>>> Misses
>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
>>> 1871871
>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
>>> Misses
>>> 144035 118 53736 2753 27813 1 57238839
>>> 544205
>>> 
>>> If you need more things, tell me, i let the PoC in this state.
>>> 
>>> Thanks
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
>>> 
>>> Loic Blot wrote:
>>> 
>>>> Hi Rick,
>>>> ok, i don't need locallocks, i haven't understand option was for
>>>> that
>>>> usage, i removed it.
>>>> I do more tests on monday.
>>>> Thanks for the deadlock fix, for other people :)
>>> 
>>> Good. Please let us know if running with vfs.nfsd.enable_locallocks=0
>>> gets rid of the deadlocks? (I think it fixes the one you saw.)
>>> 
>>> On the performance side, you might also want to try different values
>>> of
>>> readahead, if the Linux client has such a mount option. (With the
>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the
>>> optimal readahead value would be.)
>>> 
>>> Good luck with it and please let us know how it goes, rick
>>> ps: I now have a patch to fix the deadlock when
>>> vfs.nfsd.enable_locallocks=1
>>> is set. I'll post it for anyone who is interested after I put it
>>> through some testing.
>>> 
>>> --
>>> Best regards,
>>> Loïc BLOT,
>>> UNIX systems, security and network engineer
>>> http://www.unix-experience.fr
>>> 
>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
>>> 
>>> Loic Blot wrote: 
>>>> Hi rick,
>>>> i tried to start a LXC container on Debian Squeeze from my
>>>> freebsd
>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
>>>> squeeze
>>>> container, it seems (3 tries, 3 fails).
>>> 
>>> Well, I`ll take a look at this `procstat -kk`, but the only thing
>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
>>> nullfs. (I have no idea if you are using any nullfs mounts, but
>>> if so, try getting rid of them.)
>>> 
>>> Here`s a high level post about the ZFS and vnode locking problem,
>>> but there is no patch available, as far as I know.
>>> 
>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>>> 
>>> rick
>>> 
>>> 921 - D 0:00.02 nfsd: server (nfsd)
>>> 
>>> Here is the procstat -kk
>>> 
>>> PID TID COMM TDNAME KSTACK
>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>> nfsrvd_dorpc+0xc76
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>> écrit:
>>> 
>>> Loic Blot wrote:
>>> 
>>>> For more informations, here is procstat -kk on nfsd, if you
>>>> need
>>>> more
>>>> hot datas, tell me.
>>>> 
>>>> Regards, PID TID COMM TDNAME KSTACK
>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>> amd64_syscall+0x351
>>> 
>>> Well, most of the threads are stuck like this one, waiting for
>>> a
>>> vnode
>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
>>> subject line
>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
>>> look.
>>> 
>>> The only thing I`ve seen suggested is trying:
>>> sysctl vfs.lookup_shared=0
>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
>>> doesn`t
>>> obey the vnode locking rules for lookup and rename, according
>>> to
>>> the posting I saw.
>>> 
>>> I`ve added a couple of comments about the other threads below,
>>> but
>>> they are all either waiting for an RPC request or waiting for
>>> the
>>> threads stuck on the ZFS vnode lock to complete.
>>> 
>>> rick
>>> 
>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>> 
>>> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
>>> 
>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>> _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>> nfsrvd_dorpc+0xc76
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>> 
>>> This one (and a few others) are waiting for the nfsv4_lock.
>>> This
>>> happens
>>> because other threads are stuck with RPCs in progress. (ie. The
>>> ones
>>> waiting on the vnode lock in zfs_fhtovp().)
>>> For these, the RPC needs to lock out other threads to do the
>>> operation,
>>> so it waits for the nfsv4_lock() which can exclusively lock the
>>> NFSv4
>>> data structures once all other nfsd threads complete their RPCs
>>> in
>>> progress.
>>> 
>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 
>>> Same as above.
>>> 
>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>> zfs_fhtovp+0x38d
>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>> svc_thread_start+0xb
>>>> fork_exit+0x9a fork_trampoline+0xe
>>> 
>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>>> 
>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 15 décembre 2014 13:29 "Loïc Blot"
>>> <loic.blot at unix-experience.fr>
>>> a
>>> écrit:
>>> 
>>> Hmmm...
>>> now i'm experiencing a deadlock.
>>> 
>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
>>> (nfsd)
>>> 
>>> the only issue was to reboot the server, but after rebooting
>>> deadlock arrives a second time when i
>>> start my jails over NFS.
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 15 décembre 2014 10:07 "Loïc Blot"
>>> <loic.blot at unix-experience.fr>
>>> a
>>> écrit:
>>> 
>>> Hi Rick,
>>> after talking with my N+1, NFSv4 is required on our
>>> infrastructure.
>>> I tried to upgrade NFSv4+ZFS
>>> server from 9.3 to 10.1, i hope this will resolve some
>>> issues...
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 10 décembre 2014 15:36 "Loïc Blot"
>>> <loic.blot at unix-experience.fr>
>>> a
>>> écrit:
>>> 
>>> Hi Rick,
>>> thanks for your suggestion.
>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
>>> the
>>> server. kill -9 doesn't affect the
>>> process, it's blocked.... (State: Ds)
>>> 
>>> for the performances
>>> 
>>> NFSv3: 60Mbps
>>> NFSv4: 45Mbps
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
>>> a
>>> écrit:
>>> 
>>> Loic Blot wrote:
>>> 
>>>> Hi Rick,
>>>> I'm trying NFSv3.
>>>> Some jails are starting very well but now i have an issue
>>>> with
>>>> lockd
>>>> after some minutes:
>>>> 
>>>> nfs server 10.10.X.8:/jails: lockd not responding
>>>> nfs server 10.10.X.8:/jails lockd is alive again
>>>> 
>>>> I look at mbuf, but i seems there is no problem.
>>> 
>>> Well, if you need locks to be visible across multiple
>>> clients,
>>> then
>>> I'm afraid you are stuck with using NFSv4 and the
>>> performance
>>> you
>>> get
>>> from it. (There is no way to do file handle affinity for
>>> NFSv4
>>> because
>>> the read and write ops are buried in the compound RPC and
>>> not
>>> easily
>>> recognized.)
>>> 
>>> If the locks don't need to be visible across multiple
>>> clients,
>>> I'd
>>> suggest trying the "nolockd" option with nfsv3.
>>> 
>>>> Here is my rc.conf on server:
>>>> 
>>>> nfs_server_enable="YES"
>>>> nfsv4_server_enable="YES"
>>>> nfsuserd_enable="YES"
>>>> nfsd_server_flags="-u -t -n 256"
>>>> mountd_enable="YES"
>>>> mountd_flags="-r"
>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>> rpcbind_enable="YES"
>>>> rpc_lockd_enable="YES"
>>>> rpc_statd_enable="YES"
>>>> 
>>>> Here is the client:
>>>> 
>>>> nfsuserd_enable="YES"
>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>> nfscbd_enable="YES"
>>>> rpc_lockd_enable="YES"
>>>> rpc_statd_enable="YES"
>>>> 
>>>> Have you got an idea ?
>>>> 
>>>> Regards,
>>>> 
>>>> Loïc Blot,
>>>> UNIX Systems, Network and Security Engineer
>>>> http://www.unix-experience.fr
>>>> 
>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>> a
>>>> écrit: 
>>>>> Loic Blot wrote:
>>>>> 
>>>>>> Hi rick,
>>>>>> 
>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
>>>>>> sysrc
>>>>>> memcached_flags="-v -m 512"
>>>>>> Command was very very slow...
>>>>>> 
>>>>>> Here is a dd over NFS:
>>>>>> 
>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>>>>> bytes/sec)
>>>>> 
>>>>> Can you try the same read using an NFSv3 mount?
>>>>> (If it runs much faster, you have probably been bitten by
>>>>> the
>>>>> ZFS
>>>>> "sequential vs random" read heuristic which I've been told
>>>>> things
>>>>> NFS is doing "random" reads without file handle affinity.
>>>>> File
>>>>> handle affinity is very hard to do for NFSv4, so it isn't
>>>>> done.)
>>> 
>>> I was actually suggesting that you try the "dd" over nfsv3
>>> to
>>> see
>>> how
>>> the performance compared with nfsv4. If you do that, please
>>> post
>>> the
>>> comparable results.
>>> 
>>> Someday I would like to try and get ZFS's sequential vs
>>> random
>>> read
>>> heuristic modified and any info on what difference in
>>> performance
>>> that
>>> might make for NFS would be useful.
>>> 
>>> rick
>>> 
>>> rick
>>> 
>>> This is quite slow...
>>> 
>>> You can found some nfsstat below (command isn't finished
>>> yet)
>>> 
>>> nfsstat -c -w 1
>>> 
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 0 0 0 0 0 0 0 0
>>> 4 0 0 0 0 0 16 0
>>> 2 0 0 0 0 0 17 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 4 0 0 0 0 4 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 4 0 0 0 0 0 3 0
>>> 0 0 0 0 0 0 3 0
>>> 37 10 0 8 0 0 14 1
>>> 18 16 0 4 1 2 4 0
>>> 78 91 0 82 6 12 30 0
>>> 19 18 0 2 2 4 2 0
>>> 0 0 0 0 2 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 1 0 0 0 0 1 0
>>> 4 6 0 0 6 0 3 0
>>> 2 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 1 0 0 0 0 0 0 0
>>> 0 0 0 0 1 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 6 108 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 98 54 0 86 11 0 25 0
>>> 36 24 0 39 25 0 10 1
>>> 67 8 0 63 63 0 41 0
>>> 34 0 0 35 34 0 0 0
>>> 75 0 0 75 77 0 0 0
>>> 34 0 0 35 35 0 0 0
>>> 75 0 0 74 76 0 0 0
>>> 33 0 0 34 33 0 0 0
>>> 0 0 0 0 5 0 0 0
>>> 0 0 0 0 0 0 6 0
>>> 11 0 0 0 0 0 11 0
>>> 0 0 0 0 0 0 0 0
>>> 0 17 0 0 0 0 1 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 4 5 0 0 0 0 12 0
>>> 2 0 0 0 0 0 26 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 4 0 0 0 0 4 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 4 0 0 0 0 0 2 0
>>> 2 0 0 0 0 0 24 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 4 0 0 0 0 0 7 0
>>> 2 1 0 0 0 0 1 0
>>> 0 0 0 0 2 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 6 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 4 6 0 0 0 0 3 0
>>> 0 0 0 0 0 0 0 0
>>> 2 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 4 71 0 0 0 0 0 0
>>> 0 1 0 0 0 0 0 0
>>> 2 36 0 0 0 0 1 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 1 0 0 0 0 0 1 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 79 6 0 79 79 0 2 0
>>> 25 0 0 25 26 0 6 0
>>> 43 18 0 39 46 0 23 0
>>> 36 0 0 36 36 0 31 0
>>> 68 1 0 66 68 0 0 0
>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>> 36 0 0 36 36 0 0 0
>>> 48 0 0 48 49 0 0 0
>>> 20 0 0 20 20 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 3 14 0 1 0 0 11 0
>>> 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0
>>> 0 4 0 0 0 0 4 0
>>> 0 0 0 0 0 0 0 0
>>> 4 22 0 0 0 0 16 0
>>> 2 0 0 0 0 0 23 0
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 8 décembre 2014 09:36 "Loïc Blot"
>>> <loic.blot at unix-experience.fr> a
>>> écrit: 
>>>> Hi Rick,
>>>> I stopped the jails this week-end and started it this
>>>> morning,
>>>> i'll
>>>> give you some stats this week.
>>>> 
>>>> Here is my nfsstat -m output (with your rsize/wsize
>>>> tweaks)
>> 
>> 
> nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
>> 
>>> 
>> 
>> 
> etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
>> 
>>> s=2147483647
>>> 
>>> On server side my disks are on a raid controller which show a
>>> 512b
>>> volume and write performances
>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
>>> count=100000000 => 450MBps)
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>> écrit:
>>> 
>>> Loic Blot wrote:
>>> 
>>> Hi,
>>> i'm trying to create a virtualisation environment based on
>>> jails.
>>> Those jails are stored under a big ZFS pool on a FreeBSD
>>> 9.3
>>> which
>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
>>> big
>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
>>> only 1
>>> was
>>> used at this time).
>>> 
>>> The problem is simple, my hypervisors runs 6 jails (used 1%
>>> cpu
>>> and
>>> 10GB RAM approximatively and less than 1MB bandwidth) and
>>> works
>>> fine at start but the system slows down and after 2-3 days
>>> become
>>> unusable. When i look at top command i see 80-100% on
>>> system
>>> and
>>> commands are very very slow. Many process are tagged with
>>> nfs_cl*.
>>> 
>>> To be honest, I would expect the slowness to be because of
>>> slow
>>> response
>>> from the NFSv4 server, but if you do:
>>> # ps axHl
>>> on a client when it is slow and post that, it would give us
>>> some
>>> more
>>> information on where the client side processes are sitting.
>>> If you also do something like:
>>> # nfsstat -c -w 1
>>> and let it run for a while, that should show you how many
>>> RPCs
>>> are
>>> being done and which ones.
>>> 
>>> # nfsstat -m
>>> will show you what your mount is actually using.
>>> The only mount option I can suggest trying is
>>> "rsize=32768,wsize=32768",
>>> since some network environments have difficulties with 64K.
>>> 
>>> There are a few things you can try on the NFSv4 server side,
>>> if
>>> it
>>> appears
>>> that the clients are generating a large RPC load.
>>> - disabling the DRC cache for TCP by setting
>>> vfs.nfsd.cachetcp=0
>>> - If the server is seeing a large write RPC load, then
>>> "sync=disabled"
>>> might help, although it does run a risk of data loss when
>>> the
>>> server
>>> crashes.
>>> Then there are a couple of other ZFS related things (I'm not
>>> a
>>> ZFS
>>> guy,
>>> but these have shown up on the mailing lists).
>>> - make sure your volumes are 4K aligned and ashift=12 (in
>>> case a
>>> drive
>>> that uses 4K sectors is pretending to be 512byte sectored)
>>> - never run over 70-80% full if write performance is an
>>> issue
>>> - use a zil on an SSD with good write performance
>>> 
>>> The only NFSv4 thing I can tell you is that it is known that
>>> ZFS's
>>> algorithm for determining sequential vs random I/O fails for
>>> NFSv4
>>> during writing and this can be a performance hit. The only
>>> workaround
>>> is to use NFSv3 mounts, since file handle affinity
>>> apparently
>>> fixes
>>> the problem and this is only done for NFSv3.
>>> 
>>> rick
>>> 
>>> I saw that there are TSO issues with igb then i'm trying to
>>> disable
>>> it with sysctl but the situation wasn't solved.
>>> 
>>> Someone has got ideas ? I can give you more informations if
>>> you
>>> need.
>>> 
>>> Thanks in advance.
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> 
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> 
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> 
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> 
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"