ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4

[Loïc Blot]

Hi Rick,
i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS v4.1 (mountoptions: rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)

Performance is quite stable but it's slow. Not as slow as before but slow... services was launched but no client are using them and system CPU % was 10-50%.

I don't see anything on NFSv4.1 server, it's perfectly stable and functionnal.

Regards,

Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr

23 décembre 2014 00:20 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit: 
> Loic Blot wrote:
> 
>> Hi,
>> 
>> To clarify because of our exchanges. Here are the current sysctl
>> options for server:
>> 
>> vfs.nfsd.enable_nobodycheck=0
>> vfs.nfsd.enable_nogroupcheck=0
>> 
>> vfs.nfsd.maxthreads=200
>> vfs.nfsd.tcphighwater=10000
>> vfs.nfsd.tcpcachetimeo=300
>> vfs.nfsd.server_min_nfsvers=4
>> 
>> kern.maxvnodes=10000000
>> kern.ipc.maxsockbuf=4194304
>> net.inet.tcp.sendbuf_max=4194304
>> net.inet.tcp.recvbuf_max=4194304
>> 
>> vfs.lookup_shared=0
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 22 décembre 2014 09:42 "Loïc Blot" <loic.blot at unix-experience.fr> a
>> écrit:
>> 
>> Hi Rick,
>> my 5 jails runs this weekend and now i have some stats on this
>> monday.
>> 
>> Hopefully deadlock was fixed, yeah, but everything isn't good :(
>> 
>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>> 
>> As i can see this is because of nfsd:
>> 
>> 918 root            96  20    0 12352K  3372K rpcsvc  6  51.4H
>> 273.68% nfsd: server (nfsd)
>> 
>> If i look at dmesg i see:
>> nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
> 
> Well, you have a couple of choices:
> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
> (NFSv4.1 avoids use of the DRC and instead uses something
> called sessions. See below.)
> OR
> 
>> vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
> 
> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
> "nfs server cache flooded" messages. (I think Garrett Wollman uses
> 100000. (You may still see quite a bit of CPU overheads.)
> 
> OR
> 
> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets rid
> of the CPU overheads). However, there is a risk of data corruption
> if you have a client->server network partitioning of a moderate
> duration, because a non-idempotent RPC may get redone, becasue
> the client times out waiting for a reply. If a non-idempotent
> RPC gets done twice on the server, data corruption can happen.
> (The DRC provides improved correctness, but does add overhead.)
> 
> If #1 works for you, it is the preferred solution, since Sessions
> in NFSv4.1 solves the correctness problem in a good, space bound
> way. A session basically has N (usually 32 or 64) slots and only
> allows one outstanding RPC/slot. As such, it can cache the previous
> reply for each slot (32 or 64 of them) and guarantee "exactly once"
> RPC semantics.
> 
> rick
> 
>> Here is 'nfsstat -s' output:
>> 
>> Server Info:
>> Getattr   Setattr    Lookup  Readlink      Read     Write    Create
>> Remove
>> 12600652      1812   2501097       156   1386423   1983729       123
>> 162067
>> Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
>> Access
>> 36762         9         0         0         0      3147         0
>> 623524
>> Mknod    Fsstat    Fsinfo  PathConf    Commit
>> 0         0         0         0    328117
>> Server Ret-Failed
>> 0
>> Server Faults
>> 0
>> Server Cache Stats:
>> Inprog      Idem  Non-idem    Misses
>> 0         0         0  12635512
>> Server Write Gathering:
>> WriteOps  WriteRPC   Opsaved
>> 1983729   1983729         0
>> 
>> And here is 'procstat -kk' for nfsd (server)
>> 
>> 918 100528 nfsd             nfsd: master     mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>> amd64_syscall+0x351 Xfast_syscall+0xfb
>> 918 100568 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100569 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100570 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100571 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100572 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100573 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100574 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100575 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100576 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100577 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100578 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100579 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100580 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100581 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100582 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100583 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100584 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100585 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100586 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100587 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100588 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100589 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100590 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100591 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100592 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100593 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100594 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100595 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100596 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100597 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100598 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100599 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100600 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100601 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100602 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100603 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100604 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100605 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100606 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100607 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100608 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100609 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100610 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100611 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100612 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100613 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100614 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100615 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100616 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100617 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100618 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100619 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100620 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100621 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100622 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100623 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100624 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100625 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100626 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100627 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100628 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100629 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100630 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100631 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100632 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100633 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100634 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100635 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100636 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100637 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100638 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100639 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100640 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100641 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100642 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100643 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100644 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100645 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100646 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100647 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100648 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100649 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100650 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100651 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100652 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100653 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100654 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100655 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100656 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100657 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100658 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100659 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100660 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100661 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100662 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> ---
>> 
>> Now if we look at client (FreeBSD 9.3)
>> 
>> We see system was very busy and do many and many interrupts
>> 
>> CPU:  0.0% user,  0.0% nice, 37.8% system, 51.2% interrupt, 11.0%
>> idle
>> 
>> A look at process list shows that there are many sendmail process in
>> state nfstry
>> 
>> nfstry 18  32:27   0.88% sendmail: Queue runner at 00:30:00 for
>> /var/spool/clientm
>> 
>> Here is 'nfsstat -c' output:
>> 
>> Client Info:
>> Rpc Counts:
>> Getattr   Setattr    Lookup  Readlink      Read     Write    Create
>> Remove
>> 1051347      1724   2494481       118    903902   1901285    162676
>> 161899
>> Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
>> Access
>> 36744         2         0       114        40      3131         0
>> 544136
>> Mknod    Fsstat    Fsinfo  PathConf    Commit
>> 9         0         0         0    245821
>> Rpc Info:
>> TimedOut   Invalid X Replies   Retries  Requests
>> 0         0         0         0   8356557
>> Cache Info:
>> Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits
>> Misses
>> 108754455    491475  54229224   2437229  46814561    821723   5132123
>> 1871871
>> BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits
>> Misses
>> 144035       118     53736      2753     27813         1  57238839
>> 544205
>> 
>> If you need more things, tell me, i let the PoC in this state.
>> 
>> Thanks
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
>> 
>> Loic Blot wrote:
>> 
>>> Hi Rick,
>>> ok, i don't need locallocks, i haven't understand option was for
>>> that
>>> usage, i removed it.
>>> I do more tests on monday.
>>> Thanks for the deadlock fix, for other people :)
>> 
>> Good. Please let us know if running with vfs.nfsd.enable_locallocks=0
>> gets rid of the deadlocks? (I think it fixes the one you saw.)
>> 
>> On the performance side, you might also want to try different values
>> of
>> readahead, if the Linux client has such a mount option. (With the
>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the
>> optimal readahead value would be.)
>> 
>> Good luck with it and please let us know how it goes, rick
>> ps: I now have a patch to fix the deadlock when
>> vfs.nfsd.enable_locallocks=1
>> is set. I'll post it for anyone who is interested after I put it
>> through some testing.
>> 
>> --
>> Best regards,
>> Loïc BLOT,
>> UNIX systems, security and network engineer
>> http://www.unix-experience.fr
>> 
>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
>> 
>> Loic Blot wrote:
>>> Hi rick,
>>> i tried to start a LXC container on Debian Squeeze from my
>>> freebsd
>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
>>> squeeze
>>> container, it seems (3 tries, 3 fails).
>> 
>> Well, I`ll take a look at this `procstat -kk`, but the only thing
>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
>> nullfs. (I have no idea if you are using any nullfs mounts, but
>> if so, try getting rid of them.)
>> 
>> Here`s a high level post about the ZFS and vnode locking problem,
>> but there is no patch available, as far as I know.
>> 
>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>> 
>> rick
>> 
>> 921  -  D      0:00.02 nfsd: server (nfsd)
>> 
>> Here is the procstat -kk
>> 
>> PID    TID COMM             TDNAME           KSTACK
>> 921 100538 nfsd             nfsd: master     mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>> 921 100572 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100573 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100574 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100575 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100576 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100577 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100578 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100579 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100580 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100581 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100582 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100583 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100584 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100585 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100586 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100587 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100588 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100589 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100590 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100591 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100592 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100593 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100594 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100595 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100596 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100597 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100598 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100599 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100600 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100601 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100602 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100603 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100604 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100605 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100606 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100607 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100608 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100609 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100610 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100611 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100612 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100613 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100614 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100615 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100616 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 921 100617 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100618 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 921 100619 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100620 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100621 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100622 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100623 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100624 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100625 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100626 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100627 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100628 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100629 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100630 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100631 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100632 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100633 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100634 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100635 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100636 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100637 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100638 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100639 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100640 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100641 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100642 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100643 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100644 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100645 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100646 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100647 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100648 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100649 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100650 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100651 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100652 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100653 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100654 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100655 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100656 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100657 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100658 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100659 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100660 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100661 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100662 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100663 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100664 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100665 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>> _cv_wait_sig+0x16a
>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 921 100666 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>> nfsrvd_dorpc+0xc76
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a
>> écrit:
>> 
>> Loic Blot wrote:
>> 
>>> For more informations, here is procstat -kk on nfsd, if you
>>> need
>>> more
>>> hot datas, tell me.
>>> 
>>> Regards,  PID    TID COMM             TDNAME           KSTACK
>>> 918 100529 nfsd             nfsd: master     mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>> amd64_syscall+0x351
>> 
>> Well, most of the threads are stuck like this one, waiting for
>> a
>> vnode
>> lock in ZFS. All of them appear to be in zfs_fhtovp().
>> I`m not a ZFS guy, so I can`t help much. I`ll try changing the
>> subject line
>> to include ZFS vnode lock, so maybe the ZFS guys will take a
>> look.
>> 
>> The only thing I`ve seen suggested is trying:
>> sysctl vfs.lookup_shared=0
>> to disable shared vop_lookup()s. Apparently zfs_lookup()
>> doesn`t
>> obey the vnode locking rules for lookup and rename, according
>> to
>> the posting I saw.
>> 
>> I`ve added a couple of comments about the other threads below,
>> but
>> they are all either waiting for an RPC request or waiting for
>> the
>> threads stuck on the ZFS vnode lock to complete.
>> 
>> rick
>> 
>>> 918 100564 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>> 
>> Fyi, this thread is just waiting for an RPC to arrive. (Normal)
>> 
>>> 918 100565 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100566 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100567 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100568 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100569 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100570 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>> _cv_wait_sig+0x16a
>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>> fork_trampoline+0xe
>>> 918 100571 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>> 918 100572 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>> nfsrvd_dorpc+0xc76
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>> 
>> This one (and a few others) are waiting for the nfsv4_lock.
>> This
>> happens
>> because other threads are stuck with RPCs in progress. (ie. The
>> ones
>> waiting on the vnode lock in zfs_fhtovp().)
>> For these, the RPC needs to lock out other threads to do the
>> operation,
>> so it waits for the nfsv4_lock() which can exclusively lock the
>> NFSv4
>> data structures once all other nfsd threads complete their RPCs
>> in
>> progress.
>> 
>>> 918 100573 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 
>> Same as above.
>> 
>>> 918 100574 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100575 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100576 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100577 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100578 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100579 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100580 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100581 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100582 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100583 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100584 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100585 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100586 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100587 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100588 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100589 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100590 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100591 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100592 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100593 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100594 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100595 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100596 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100597 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100598 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100599 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100600 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100601 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100602 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100603 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100604 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100605 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100606 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>>> 918 100607 nfsd             nfsd: service    mi_switch+0xe1
>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>> zfs_fhtovp+0x38d
>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>> svc_thread_start+0xb
>>> fork_exit+0x9a fork_trampoline+0xe
>> 
>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>> 
>> 918 100608 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100609 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100610 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
>> fork_trampoline+0xe
>> 918 100611 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100612 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100613 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100614 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100615 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100616 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100617 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100618 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100619 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100620 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100621 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100622 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100623 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>> 918 100624 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100625 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100626 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100627 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100628 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100629 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100630 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100631 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100632 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100633 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100634 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100635 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100636 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100637 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100638 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100639 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100640 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100641 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100642 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100643 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100644 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100645 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100646 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100647 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100648 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100649 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100650 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100651 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100652 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100653 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100654 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100655 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100656 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100657 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 918 100658 nfsd             nfsd: service    mi_switch+0xe1
>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>> zfs_fhtovp+0x38d
>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>> nfssvc_program+0x554 svc_run_internal+0xc77
>> svc_thread_start+0xb
>> fork_exit+0x9a fork_trampoline+0xe
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 15 décembre 2014 13:29 "Loïc Blot"
>> <loic.blot at unix-experience.fr>
>> a
>> écrit:
>> 
>> Hmmm...
>> now i'm experiencing a deadlock.
>> 
>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
>> (nfsd)
>> 
>> the only issue was to reboot the server, but after rebooting
>> deadlock arrives a second time when i
>> start my jails over NFS.
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 15 décembre 2014 10:07 "Loïc Blot"
>> <loic.blot at unix-experience.fr>
>> a
>> écrit:
>> 
>> Hi Rick,
>> after talking with my N+1, NFSv4 is required on our
>> infrastructure.
>> I tried to upgrade NFSv4+ZFS
>> server from 9.3 to 10.1, i hope this will resolve some
>> issues...
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 10 décembre 2014 15:36 "Loïc Blot"
>> <loic.blot at unix-experience.fr>
>> a
>> écrit:
>> 
>> Hi Rick,
>> thanks for your suggestion.
>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
>> the
>> server. kill -9 doesn't affect the
>> process, it's blocked.... (State: Ds)
>> 
>> for the performances
>> 
>> NFSv3: 60Mbps
>> NFSv4: 45Mbps
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
>> a
>> écrit:
>> 
>> Loic Blot wrote:
>> 
>>> Hi Rick,
>>> I'm trying NFSv3.
>>> Some jails are starting very well but now i have an issue
>>> with
>>> lockd
>>> after some minutes:
>>> 
>>> nfs server 10.10.X.8:/jails: lockd not responding
>>> nfs server 10.10.X.8:/jails lockd is alive again
>>> 
>>> I look at mbuf, but i seems there is no problem.
>> 
>> Well, if you need locks to be visible across multiple
>> clients,
>> then
>> I'm afraid you are stuck with using NFSv4 and the
>> performance
>> you
>> get
>> from it. (There is no way to do file handle affinity for
>> NFSv4
>> because
>> the read and write ops are buried in the compound RPC and
>> not
>> easily
>> recognized.)
>> 
>> If the locks don't need to be visible across multiple
>> clients,
>> I'd
>> suggest trying the "nolockd" option with nfsv3.
>> 
>>> Here is my rc.conf on server:
>>> 
>>> nfs_server_enable="YES"
>>> nfsv4_server_enable="YES"
>>> nfsuserd_enable="YES"
>>> nfsd_server_flags="-u -t -n 256"
>>> mountd_enable="YES"
>>> mountd_flags="-r"
>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>> rpcbind_enable="YES"
>>> rpc_lockd_enable="YES"
>>> rpc_statd_enable="YES"
>>> 
>>> Here is the client:
>>> 
>>> nfsuserd_enable="YES"
>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>> nfscbd_enable="YES"
>>> rpc_lockd_enable="YES"
>>> rpc_statd_enable="YES"
>>> 
>>> Have you got an idea ?
>>> 
>>> Regards,
>>> 
>>> Loïc Blot,
>>> UNIX Systems, Network and Security Engineer
>>> http://www.unix-experience.fr
>>> 
>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
>>> a
>>> écrit:
>>>> Loic Blot wrote:
>>>> 
>>>>> Hi rick,
>>>>> 
>>>>> I waited 3 hours (no lag at jail launch) and now I do:
>>>>> sysrc
>>>>> memcached_flags="-v -m 512"
>>>>> Command was very very slow...
>>>>> 
>>>>> Here is a dd over NFS:
>>>>> 
>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>>>> bytes/sec)
>>>> 
>>>> Can you try the same read using an NFSv3 mount?
>>>> (If it runs much faster, you have probably been bitten by
>>>> the
>>>> ZFS
>>>> "sequential vs random" read heuristic which I've been told
>>>> things
>>>> NFS is doing "random" reads without file handle affinity.
>>>> File
>>>> handle affinity is very hard to do for NFSv4, so it isn't
>>>> done.)
>> 
>> I was actually suggesting that you try the "dd" over nfsv3
>> to
>> see
>> how
>> the performance compared with nfsv4. If you do that, please
>> post
>> the
>> comparable results.
>> 
>> Someday I would like to try and get ZFS's sequential vs
>> random
>> read
>> heuristic modified and any info on what difference in
>> performance
>> that
>> might make for NFS would be useful.
>> 
>> rick
>> 
>> rick
>> 
>> This is quite slow...
>> 
>> You can found some nfsstat below (command isn't finished
>> yet)
>> 
>> nfsstat -c -w 1
>> 
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 0 0 0 0 0 0 0 0
>> 4 0 0 0 0 0 16 0
>> 2 0 0 0 0 0 17 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 4 0 0 0 0 4 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 4 0 0 0 0 0 3 0
>> 0 0 0 0 0 0 3 0
>> 37 10 0 8 0 0 14 1
>> 18 16 0 4 1 2 4 0
>> 78 91 0 82 6 12 30 0
>> 19 18 0 2 2 4 2 0
>> 0 0 0 0 2 0 0 0
>> 0 0 0 0 0 0 0 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 1 0 0 0 0 1 0
>> 4 6 0 0 6 0 3 0
>> 2 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 1 0 0 0 0 0 0 0
>> 0 0 0 0 1 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 6 108 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 98 54 0 86 11 0 25 0
>> 36 24 0 39 25 0 10 1
>> 67 8 0 63 63 0 41 0
>> 34 0 0 35 34 0 0 0
>> 75 0 0 75 77 0 0 0
>> 34 0 0 35 35 0 0 0
>> 75 0 0 74 76 0 0 0
>> 33 0 0 34 33 0 0 0
>> 0 0 0 0 5 0 0 0
>> 0 0 0 0 0 0 6 0
>> 11 0 0 0 0 0 11 0
>> 0 0 0 0 0 0 0 0
>> 0 17 0 0 0 0 1 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 4 5 0 0 0 0 12 0
>> 2 0 0 0 0 0 26 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 4 0 0 0 0 4 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 4 0 0 0 0 0 2 0
>> 2 0 0 0 0 0 24 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 4 0 0 0 0 0 7 0
>> 2 1 0 0 0 0 1 0
>> 0 0 0 0 2 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 6 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 4 6 0 0 0 0 3 0
>> 0 0 0 0 0 0 0 0
>> 2 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 4 71 0 0 0 0 0 0
>> 0 1 0 0 0 0 0 0
>> 2 36 0 0 0 0 1 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 1 0 0 0 0 0 1 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 79 6 0 79 79 0 2 0
>> 25 0 0 25 26 0 6 0
>> 43 18 0 39 46 0 23 0
>> 36 0 0 36 36 0 31 0
>> 68 1 0 66 68 0 0 0
>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>> 36 0 0 36 36 0 0 0
>> 48 0 0 48 49 0 0 0
>> 20 0 0 20 20 0 0 0
>> 0 0 0 0 0 0 0 0
>> 3 14 0 1 0 0 11 0
>> 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0
>> 0 4 0 0 0 0 4 0
>> 0 0 0 0 0 0 0 0
>> 4 22 0 0 0 0 16 0
>> 2 0 0 0 0 0 23 0
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 8 décembre 2014 09:36 "Loïc Blot"
>> <loic.blot at unix-experience.fr> a
>> écrit:
>>> Hi Rick,
>>> I stopped the jails this week-end and started it this
>>> morning,
>>> i'll
>>> give you some stats this week.
>>> 
>>> Here is my nfsstat -m output (with your rsize/wsize
>>> tweaks)
>> 
>> 
> nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
>> 
>> 
> etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
>> 
>> s=2147483647
>> 
>> On server side my disks are on a raid controller which show a
>> 512b
>> volume and write performances
>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
>> count=100000000 => 450MBps)
>> 
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> 
>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
>> écrit:
>> 
>> Loic Blot wrote:
>> 
>> Hi,
>> i'm trying to create a virtualisation environment based on
>> jails.
>> Those jails are stored under a big ZFS pool on a FreeBSD
>> 9.3
>> which
>> export a NFSv4 volume. This NFSv4 volume was mounted on a
>> big
>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
>> only 1
>> was
>> used at this time).
>> 
>> The problem is simple, my hypervisors runs 6 jails (used 1%
>> cpu
>> and
>> 10GB RAM approximatively and less than 1MB bandwidth) and
>> works
>> fine at start but the system slows down and after 2-3 days
>> become
>> unusable. When i look at top command i see 80-100% on
>> system
>> and
>> commands are very very slow. Many process are tagged with
>> nfs_cl*.
>> 
>> To be honest, I would expect the slowness to be because of
>> slow
>> response
>> from the NFSv4 server, but if you do:
>> # ps axHl
>> on a client when it is slow and post that, it would give us
>> some
>> more
>> information on where the client side processes are sitting.
>> If you also do something like:
>> # nfsstat -c -w 1
>> and let it run for a while, that should show you how many
>> RPCs
>> are
>> being done and which ones.
>> 
>> # nfsstat -m
>> will show you what your mount is actually using.
>> The only mount option I can suggest trying is
>> "rsize=32768,wsize=32768",
>> since some network environments have difficulties with 64K.
>> 
>> There are a few things you can try on the NFSv4 server side,
>> if
>> it
>> appears
>> that the clients are generating a large RPC load.
>> - disabling the DRC cache for TCP by setting
>> vfs.nfsd.cachetcp=0
>> - If the server is seeing a large write RPC load, then
>> "sync=disabled"
>> might help, although it does run a risk of data loss when
>> the
>> server
>> crashes.
>> Then there are a couple of other ZFS related things (I'm not
>> a
>> ZFS
>> guy,
>> but these have shown up on the mailing lists).
>> - make sure your volumes are 4K aligned and ashift=12 (in
>> case a
>> drive
>> that uses 4K sectors is pretending to be 512byte sectored)
>> - never run over 70-80% full if write performance is an
>> issue
>> - use a zil on an SSD with good write performance
>> 
>> The only NFSv4 thing I can tell you is that it is known that
>> ZFS's
>> algorithm for determining sequential vs random I/O fails for
>> NFSv4
>> during writing and this can be a performance hit. The only
>> workaround
>> is to use NFSv3 mounts, since file handle affinity
>> apparently
>> fixes
>> the problem and this is only done for NFSv3.
>> 
>> rick
>> 
>> I saw that there are TSO issues with igb then i'm trying to
>> disable
>> it with sysctl but the situation wasn't solved.
>> 
>> Someone has got ideas ? I can give you more informations if
>> you
>> need.
>> 
>> Thanks in advance.
>> Regards,
>> 
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to
>> "freebsd-fs-unsubscribe at freebsd.org"
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to
>> "freebsd-fs-unsubscribe at freebsd.org"
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to
>> "freebsd-fs-unsubscribe at freebsd.org"
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to
>> "freebsd-fs-unsubscribe at freebsd.org"
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to
>> "freebsd-fs-unsubscribe at freebsd.org"
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"