High Kernel Load with nfsv4
Loïc Blot
loic.blot at unix-experience.fr
Wed Jan 14 09:16:37 UTC 2015
Hi Rick,
you fix doesn't help, i'm sorry. It improves the NFS server speed, but don't remove the problem.
I think have loader.conf variables for there options could be great.
I have wrote a cpp bench for testing my NFS server configuration but i can't reproduce the problem whereas the bench let many file descriptors opened (randomly).
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
7 janvier 2015 00:29 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
> Loic Blot wrote:
>
>> Hi Rick,
>>
>> i saw that some people has issues with igb cards with NFS
>> For example:
>> http://freebsd.1045724.n5.nabble.com/NFS-over-LAGG-lacp-poor-performance-td5906349.html
>>
>> can my problem be related ? I use igb with default queue number. Here
>> are my vmstat -i outputs
>
> I have no idea. Maybe someone familiar with this will respond?
>
> I do think that the large # of NFSv4 Opens (which are actually a form of lock)
> could be a factor. The client and server has to search those lists for a match
> for many NFSv4 operations, including all reads/writes.
>
> On the server side, the default hash table sizes are very small. This is in
> part that I did testing on 256Mbyte i386 systems, so that the values were
> safe for such a machine.
> I'd suggest you increase the following on the server's kernel.
> In sys/fs/nfs/nfs.h:
> NFSSTATEHASHSIZE - This one is in every client header, so if you have a large#
> of clients, you don't want to increase it too much. However
> for a fairly large server handling not too many clients, I'd
> try something like 1000 instead of 10.
> (I just tried 100 on the small i386 laptop I have handy and it
> seemed ok for a small test.)
> NFSLOCKHASHSIZE - This one is a single global table, so I'd bump it way up,
> 20000 maybe?
> In sys/fs/nfsport.h:
> NFSRV_V4STATELIMIT - The comment notes that the default of 500000 seems safe for
> a 256Mbyte i386, so I'd bump it to something like 2000000
> for your case.
> You will have to rebuild a kernel from sources after editing these values and
> boot it on the server. Maybe these should become tunables so building a kernel
> isn't necessary?
>
> I looked and there isn't much that can be done in the client. At this point,
> the open_owners and opens are single lists for a client (a mount point on a
> client machine for FreeBSD). If you post what you get for "nfsstat -e -c"
> on a typical client in your setup, that would tell me if it is the open_owners
> (which I suspect) or opens that will be a long list. (I would have to code
> a patch to make either of these a hash table instead of a single linked
> list. I should do this. It was on my to-do list, but got forgotten.;-)
>
> rick
>
>> Server side:
>>
>> interrupt total rate
>> irq1: atkbd0 18 0
>> irq20: ehci1 2790134 2
>> irq21: ehci0 2547642 2
>> cpu0:timer 36299188 35
>> irq264: ciss0 6352476 6
>> irq265: igb0:que 0 2716692 2
>> irq266: igb0:que 1 32205278 31
>> irq267: igb0:que 2 38395109 37
>> irq268: igb0:que 3 1413468 1
>> irq269: igb0:que 4 39207930 38
>> irq270: igb0:que 5 1622715 1
>> irq271: igb0:que 6 1634676 1
>> irq272: igb0:que 7 1190123 1
>> irq273: igb0:link 2 0
>> cpu1:timer 14074423 13
>> cpu8:timer 12204739 11
>> cpu9:timer 11384192 11
>> cpu3:timer 10461566 10
>> cpu4:timer 12785103 12
>> cpu6:timer 10739344 10
>> cpu5:timer 10978294 10
>> cpu7:timer 10599705 10
>> cpu2:timer 13998891 13
>> cpu10:timer 11602361 11
>> cpu11:timer 11568523 11
>> Total 296772592 290
>>
>> And client side:
>> interrupt total rate
>> irq9: acpi0 4 0
>> irq22: ehci1 950519 2
>> irq23: ehci0 1865060 4
>> cpu0:timer 248128035 546
>> irq268: mfi0 406896 0
>> irq269: igb0:que 0 2510556 5
>> irq270: igb0:que 1 2825336 6
>> irq271: igb0:que 2 2092958 4
>> irq272: igb0:que 3 1960849 4
>> irq273: igb0:que 4 2645369 5
>> irq274: igb0:que 5 2735187 6
>> irq275: igb0:que 6 2290531 5
>> irq276: igb0:que 7 2384370 5
>> irq277: igb0:link 2 0
>> irq287: igb2:que 0 1465051 3
>> irq288: igb2:que 1 856381 1
>> irq289: igb2:que 2 809318 1
>> irq290: igb2:que 3 897154 1
>> irq291: igb2:que 4 875755 1
>> irq292: igb2:que 5 35866117 78
>> irq293: igb2:que 6 846517 1
>> irq294: igb2:que 7 857979 1
>> irq295: igb2:link 2 0
>> irq296: igb3:que 0 535212 1
>> irq297: igb3:que 1 454359 1
>> irq298: igb3:que 2 454142 1
>> irq299: igb3:que 3 454623 1
>> irq300: igb3:que 4 456297 1
>> irq301: igb3:que 5 455482 1
>> irq302: igb3:que 6 456128 1
>> irq303: igb3:que 7 454680 1
>> irq304: igb3:link 3 0
>> irq305: ahci0 75 0
>> cpu1:timer 257233702 566
>> cpu13:timer 255603184 562
>> cpu7:timer 258492826 569
>> cpu12:timer 255819351 563
>> cpu6:timer 258493465 569
>> cpu15:timer 254694003 560
>> cpu3:timer 258171320 568
>> cpu22:timer 256506877 564
>> cpu5:timer 253401435 558
>> cpu16:timer 255412360 562
>> cpu11:timer 257318013 566
>> cpu20:timer 253648060 558
>> cpu2:timer 257864543 567
>> cpu17:timer 261828899 576
>> cpu9:timer 257497326 567
>> cpu18:timer 258451190 569
>> cpu8:timer 257784504 567
>> cpu14:timer 254923723 561
>> cpu10:timer 257265498 566
>> cpu19:timer 258775946 569
>> cpu4:timer 256368658 564
>> cpu23:timer 255050534 561
>> cpu21:timer 257663842 567
>> Total 6225260206 13710
>>
>> Please note igb2 on client side is the dedicated link for NFSv4
>>
>> Regards,
>>
>> Loïc Blot,
>> UNIX Systems, Network and Security Engineer
>> http://www.unix-experience.fr
>>
>> 6 janvier 2015 04:17 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
>>> Loic Blot wrote:
>>>
>>>> Hi Rick,
>>>> nfsstat -e -s don't show usefull datas on server.
>>>
>>> Well, as far as I know, it returns valid information.
>>> (See below.)
>>>
>>>> Server Info:
>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>> Remove
>>>> 26935254 16911 5755728 302 2334920 3673866 0
>>>> 328332
>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>> Access
>>>> 77980 28 0 0 3 8900 3
>>>> 1806052
>>>> Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId
>>>> SetClIdCf
>>>> 1 1095 0 0 614377 8172 8
>>>> 8
>>>> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH
>>>> Lock
>>>> 1595299 0 44145 1495 0 0 5197490
>>>> 635015
>>>> LockT LockU Close Verify NVerify PutFH PutPubFH
>>>> PutRootFH
>>>> 0 614919 1270938 0 0 22688676 0
>>>> 5
>>>> Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create
>>>> 42104 197606 275820 0 143 4578
>>>> Server:
>>>> Retfailed Faults Clients
>>>> 0 0 6
>>>> OpenOwner Opens LockOwner Locks Delegs
>>>> 32335 145448 204 181 0
>>>
>>> Well, 145448 Opens are a lot of Open files. Each of these uses
>>> a kernel malloc'd data structure that is linked into multiple
>>> linked lists.
>>>
>>> The question is..why aren't these Opens being closed?
>>> Since FreeBSD does I/O on an mmap'd file after closing it,
>>> the FreeBSD NFSv4 client is forced to delay doing Close RPCs
>>> until the vnode is VOP_INACTIVE()/VOP_RECLAIM()'d. (The
>>> VOP_RECLAIM() case is needed, since VOP_INACTIVE() isn't
>>> guaranteed to be called.)
>>>
>>> Since there were about 1.5 million Opens and 1.27 million
>>> Closes, it does appear that Opens are being Closed.
>>> Now, I'm not sure I would have imagined 1.5million file Opens
>>> in a few days. My guess is this is the bottleneck.
>>>
>>> I'd suggest that you do:
>>> # nfsstat -e -c
>>> on each of the NFSv4 clients and see how many Opens/client
>>> there are. I vaguely remember an upper limit in the client,
>>> but can't remember what it is set to.
>>> --> I suspect the client Open/Lock limit needs to be increased.
>>> (I can't remember if the server also has a limit, but I
>>> think it does.)
>>> Then the size of the hash tables used to search the Opens
>>> may also need to be increased a lot.
>>>
>>> Also, I'd suggest you take a look at whatever apps. are
>>> running on the client(s) and try to figure out why they
>>> are Opening so many files?
>>>
>>> My guess is that the client(s) are gettig bogged down by all
>>> these Opens.
>>>
>>>> Server Cache Stats:
>>>> Inprog Idem Non-idem Misses CacheSize TCPPeak
>>>> 0 0 1 15082947 60 16522
>>>>
>>>> Only GetAttr and Lookup increase and it's only every 4-5 seconds
>>>> and
>>>> only +2 to +5 into theses values.
>>>>
>>>> Now on client, if i take four processes stack i got
>>>>
>>>> PID TID COMM TDNAME KSTACK
>>>> 63170 102547 mv - mi_switch+0xe1
>>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>>>> vn_open_cred+0x21d kern_openat+0x26f amd64_syscall+0x351
>>>> Xfast_syscall+0xfb
>>>>
>>>> Another mv:
>>>> 63140 101738 mv - mi_switch+0xe1
>>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>>>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
>>>> Xfast_syscall+0xfb
>>>>
>>>> 62070 102170 sendmail - mi_switch+0xe1
>>>> sleepq_timedwait+0x3a _sleep+0x26e clnt_vc_call+0x666
>>>> clnt_reconnect_call+0x4fa newnfs_request+0xa8c nfscl_request+0x72
>>>> nfsrpc_lookup+0x1fb nfs_lookup+0x508 VOP_LOOKUP_APV+0xa1
>>>> lookup+0x59c namei+0x4d4 kern_statat_vnhook+0xae sys_lstat+0x30
>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>>>
>>>> 63200 100930 mv - mi_switch+0xe1
>>>> turnstile_wait+0x42a __mtx_lock_sleep+0x253 nfscl_nodeleg+0x65
>>>> nfs_lookup+0x3d0 VOP_LOOKUP_APV+0xa1 lookup+0x59c namei+0x4d4
>>>> kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x351
>>>> Xfast_syscall+0xfb
>>>
>>> The above simply says that thread 102710 is waiting for a Lookup
>>> reply from the server and the other 3 are waiting for the mutex
>>> that protects the state structures in the client. (I suspect
>>> some other thread in the client is wading through the Open list,
>>> if a single client has a lot of these 145K Opens.)
>>>
>>>> When client is in this state, server was doing nothing special
>>>> (procstat -kk)
>>>>
>>>> PID TID COMM TDNAME KSTACK
>>>> 895 100538 nfsd nfsd: master mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de
>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>>> 895 100568 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100569 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100570 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100571 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100572 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100573 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100575 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100576 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100577 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100578 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100579 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100580 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100581 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100582 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100583 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100584 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100585 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100586 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100587 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100588 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100589 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100590 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100592 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100593 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100594 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100595 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100596 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100597 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100598 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100599 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100600 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100602 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100603 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100604 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100605 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100606 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100607 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100608 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100609 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100610 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100611 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100612 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100613 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100614 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100615 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100617 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100618 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100619 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100621 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100622 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100623 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100624 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100625 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100626 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100627 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100628 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100629 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100630 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100631 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100632 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100633 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100634 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100635 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100636 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100638 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100639 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100640 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100641 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100642 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100643 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100644 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100645 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100646 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100647 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100648 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100649 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100651 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100652 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100653 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100654 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100655 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100656 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100657 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100658 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100659 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100661 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100662 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100684 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100685 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100686 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100797 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100798 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100799 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100800 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>> 895 100801 nfsd nfsd: service mi_switch+0xe1
>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a
>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>> fork_trampoline+0xe
>>>>
>>>> I really think it's a client side problem, maybe a lookup problem.
>>>>
>>>> Regards,
>>>>
>>>> Loïc Blot,
>>>> UNIX Systems, Network and Security Engineer
>>>> http://www.unix-experience.fr
>>>>
>>>> 5 janvier 2015 14:35 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>> écrit:
>>>>> Loic Blot wrote:
>>>>>
>>>>>> Hi,
>>>>>> happy new year Rick and @freebsd-fs.
>>>>>>
>>>>>> After some days, i looked my NFSv4.1 mount. At server start it
>>>>>> was
>>>>>> calm, but after 4 days, here is the top stat...
>>>>>>
>>>>>> CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0%
>>>>>> idle
>>>>>>
>>>>>> Definitively i think it's a problem on client side. What can i
>>>>>> look
>>>>>> into running kernel to resolve this issue ?
>>>>>
>>>>> Well, I'd start with:
>>>>> # nfsstat -e -s
>>>>> - run repeatedly on the server (once every N seconds in a loop).
>>>>> Then look at the output, comparing the counts and see which RPCs
>>>>> are being performed by the client(s). You are looking for which
>>>>> RPCs are being done a lot. (If one RPC is almost 100% of the
>>>>> load,
>>>>> then it might be a client/caching issue for whatever that RPC is
>>>>> doing.)
>>>>>
>>>>> Also look at the Open/Lock counts near the end of the output.
>>>>> If the # of Opens/Locks is large, it may be possible to reduce
>>>>> the
>>>>> CPU overheads by using larger hash tables.
>>>>>
>>>>> Then you need to profile the server kernel to see where the CPU
>>>>> is being used.
>>>>> Hopefully someone else can fill you in on how to do that, because
>>>>> I'll admit I don't know how to.
>>>>> Basically you are looking to see if the CPU is being used in
>>>>> the NFS server code or ZFS.
>>>>>
>>>>> Good luck with it, rick
>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Loïc Blot,
>>>>>> UNIX Systems, Network and Security Engineer
>>>>>> http://www.unix-experience.fr
>>>>>>
>>>>>> 30 décembre 2014 16:16 "Loïc Blot"
>>>>>> <loic.blot at unix-experience.fr>
>>>>>> a
>>>>>> écrit:
>>>>>>> Hi Rick,
>>>>>>> i upgraded my jail host from FreeBSD 9.3 to 10.1 to use NFS
>>>>>>> v4.1
>>>>>>> (mountoptions:
>>>>>>> rw,rsize=32768,wsize=32768,tcp,nfsv4,minorversion=1)
>>>>>>>
>>>>>>> Performance is quite stable but it's slow. Not as slow as
>>>>>>> before
>>>>>>> but slow... services was launched
>>>>>>> but no client are using them and system CPU % was 10-50%.
>>>>>>>
>>>>>>> I don't see anything on NFSv4.1 server, it's perfectly stable
>>>>>>> and
>>>>>>> functionnal.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Loïc Blot,
>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>> http://www.unix-experience.fr
>>>>>>>
>>>>>>> 23 décembre 2014 00:20 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>>>>> écrit:
>>>>>>>
>>>>>>>> Loic Blot wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> To clarify because of our exchanges. Here are the current
>>>>>>>>> sysctl
>>>>>>>>> options for server:
>>>>>>>>>
>>>>>>>>> vfs.nfsd.enable_nobodycheck=0
>>>>>>>>> vfs.nfsd.enable_nogroupcheck=0
>>>>>>>>>
>>>>>>>>> vfs.nfsd.maxthreads=200
>>>>>>>>> vfs.nfsd.tcphighwater=10000
>>>>>>>>> vfs.nfsd.tcpcachetimeo=300
>>>>>>>>> vfs.nfsd.server_min_nfsvers=4
>>>>>>>>>
>>>>>>>>> kern.maxvnodes=10000000
>>>>>>>>> kern.ipc.maxsockbuf=4194304
>>>>>>>>> net.inet.tcp.sendbuf_max=4194304
>>>>>>>>> net.inet.tcp.recvbuf_max=4194304
>>>>>>>>>
>>>>>>>>> vfs.lookup_shared=0
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 22 décembre 2014 09:42 "Loïc Blot"
>>>>>>>>> <loic.blot at unix-experience.fr>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Hi Rick,
>>>>>>>>> my 5 jails runs this weekend and now i have some stats on
>>>>>>>>> this
>>>>>>>>> monday.
>>>>>>>>>
>>>>>>>>> Hopefully deadlock was fixed, yeah, but everything isn't good
>>>>>>>>> :(
>>>>>>>>>
>>>>>>>>> On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
>>>>>>>>>
>>>>>>>>> As i can see this is because of nfsd:
>>>>>>>>>
>>>>>>>>> 918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H
>>>>>>>>> 273.68% nfsd: server (nfsd)
>>>>>>>>>
>>>>>>>>> If i look at dmesg i see:
>>>>>>>>> nfsd server cache flooded, try increasing
>>>>>>>>> vfs.nfsd.tcphighwater
>>>>>>>>
>>>>>>>> Well, you have a couple of choices:
>>>>>>>> 1 - Use NFSv4.1 (add "minorversion=1" to your mount options).
>>>>>>>> (NFSv4.1 avoids use of the DRC and instead uses something
>>>>>>>> called sessions. See below.)
>>>>>>>> OR
>>>>>>>>
>>>>>>>>> vfs.nfsd.tcphighwater was set to 10000, i increase it to
>>>>>>>>> 15000
>>>>>>>>
>>>>>>>> 2 - Bump vfs.nfsd.tcphighwater way up, until you no longer see
>>>>>>>> "nfs server cache flooded" messages. (I think Garrett Wollman
>>>>>>>> uses
>>>>>>>> 100000. (You may still see quite a bit of CPU overheads.)
>>>>>>>>
>>>>>>>> OR
>>>>>>>>
>>>>>>>> 3 - Set vfs.nfsd.cachetcp=0 (which disables the DRC and gets
>>>>>>>> rid
>>>>>>>> of the CPU overheads). However, there is a risk of data
>>>>>>>> corruption
>>>>>>>> if you have a client->server network partitioning of a
>>>>>>>> moderate
>>>>>>>> duration, because a non-idempotent RPC may get redone, becasue
>>>>>>>> the client times out waiting for a reply. If a non-idempotent
>>>>>>>> RPC gets done twice on the server, data corruption can happen.
>>>>>>>> (The DRC provides improved correctness, but does add
>>>>>>>> overhead.)
>>>>>>>>
>>>>>>>> If #1 works for you, it is the preferred solution, since
>>>>>>>> Sessions
>>>>>>>> in NFSv4.1 solves the correctness problem in a good, space
>>>>>>>> bound
>>>>>>>> way. A session basically has N (usually 32 or 64) slots and
>>>>>>>> only
>>>>>>>> allows one outstanding RPC/slot. As such, it can cache the
>>>>>>>> previous
>>>>>>>> reply for each slot (32 or 64 of them) and guarantee "exactly
>>>>>>>> once"
>>>>>>>> RPC semantics.
>>>>>>>>
>>>>>>>> rick
>>>>>>>>
>>>>>>>>> Here is 'nfsstat -s' output:
>>>>>>>>>
>>>>>>>>> Server Info:
>>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>>>>>> Remove
>>>>>>>>> 12600652 1812 2501097 156 1386423 1983729 123
>>>>>>>>> 162067
>>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>>>>>> Access
>>>>>>>>> 36762 9 0 0 0 3147 0
>>>>>>>>> 623524
>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>>>>>> 0 0 0 0 328117
>>>>>>>>> Server Ret-Failed
>>>>>>>>> 0
>>>>>>>>> Server Faults
>>>>>>>>> 0
>>>>>>>>> Server Cache Stats:
>>>>>>>>> Inprog Idem Non-idem Misses
>>>>>>>>> 0 0 0 12635512
>>>>>>>>> Server Write Gathering:
>>>>>>>>> WriteOps WriteRPC Opsaved
>>>>>>>>> 1983729 1983729 0
>>>>>>>>>
>>>>>>>>> And here is 'procstat -kk' for nfsd (server)
>>>>>>>>>
>>>>>>>>> 918 100528 nfsd nfsd: master mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10
>>>>>>>>> _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1
>>>>>>>>> svc_run+0x1de
>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>>>> amd64_syscall+0x351 Xfast_syscall+0xfb
>>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100659 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100660 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100661 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100662 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> Now if we look at client (FreeBSD 9.3)
>>>>>>>>>
>>>>>>>>> We see system was very busy and do many and many interrupts
>>>>>>>>>
>>>>>>>>> CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt,
>>>>>>>>> 11.0%
>>>>>>>>> idle
>>>>>>>>>
>>>>>>>>> A look at process list shows that there are many sendmail
>>>>>>>>> process
>>>>>>>>> in
>>>>>>>>> state nfstry
>>>>>>>>>
>>>>>>>>> nfstry 18 32:27 0.88% sendmail: Queue runner at 00:30:00 for
>>>>>>>>> /var/spool/clientm
>>>>>>>>>
>>>>>>>>> Here is 'nfsstat -c' output:
>>>>>>>>>
>>>>>>>>> Client Info:
>>>>>>>>> Rpc Counts:
>>>>>>>>> Getattr Setattr Lookup Readlink Read Write Create
>>>>>>>>> Remove
>>>>>>>>> 1051347 1724 2494481 118 903902 1901285 162676
>>>>>>>>> 161899
>>>>>>>>> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
>>>>>>>>> Access
>>>>>>>>> 36744 2 0 114 40 3131 0
>>>>>>>>> 544136
>>>>>>>>> Mknod Fsstat Fsinfo PathConf Commit
>>>>>>>>> 9 0 0 0 245821
>>>>>>>>> Rpc Info:
>>>>>>>>> TimedOut Invalid X Replies Retries Requests
>>>>>>>>> 0 0 0 0 8356557
>>>>>>>>> Cache Info:
>>>>>>>>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits
>>>>>>>>> Misses
>>>>>>>>> 108754455 491475 54229224 2437229 46814561 821723 5132123
>>>>>>>>> 1871871
>>>>>>>>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits
>>>>>>>>> Misses
>>>>>>>>> 144035 118 53736 2753 27813 1 57238839
>>>>>>>>> 544205
>>>>>>>>>
>>>>>>>>> If you need more things, tell me, i let the PoC in this
>>>>>>>>> state.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Loic Blot wrote:
>>>>>>>>>
>>>>>>>>>> Hi Rick,
>>>>>>>>>> ok, i don't need locallocks, i haven't understand option was
>>>>>>>>>> for
>>>>>>>>>> that
>>>>>>>>>> usage, i removed it.
>>>>>>>>>> I do more tests on monday.
>>>>>>>>>> Thanks for the deadlock fix, for other people :)
>>>>>>>>>
>>>>>>>>> Good. Please let us know if running with
>>>>>>>>> vfs.nfsd.enable_locallocks=0
>>>>>>>>> gets rid of the deadlocks? (I think it fixes the one you
>>>>>>>>> saw.)
>>>>>>>>>
>>>>>>>>> On the performance side, you might also want to try different
>>>>>>>>> values
>>>>>>>>> of
>>>>>>>>> readahead, if the Linux client has such a mount option. (With
>>>>>>>>> the
>>>>>>>>> NFSv4-ZFS sequential vs random I/O heuristic, I have no idea
>>>>>>>>> what
>>>>>>>>> the
>>>>>>>>> optimal readahead value would be.)
>>>>>>>>>
>>>>>>>>> Good luck with it and please let us know how it goes, rick
>>>>>>>>> ps: I now have a patch to fix the deadlock when
>>>>>>>>> vfs.nfsd.enable_locallocks=1
>>>>>>>>> is set. I'll post it for anyone who is interested after I put
>>>>>>>>> it
>>>>>>>>> through some testing.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best regards,
>>>>>>>>> Loïc BLOT,
>>>>>>>>> UNIX systems, security and network engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>> Loic Blot wrote:
>>>>>>>>>> Hi rick,
>>>>>>>>>> i tried to start a LXC container on Debian Squeeze from my
>>>>>>>>>> freebsd
>>>>>>>>>> ZFS+NFSv4 server and i also have a deadlock on nfsd
>>>>>>>>>> (vfs.lookup_shared=0). Deadlock procs each time i launch a
>>>>>>>>>> squeeze
>>>>>>>>>> container, it seems (3 tries, 3 fails).
>>>>>>>>>
>>>>>>>>> Well, I`ll take a look at this `procstat -kk`, but the only
>>>>>>>>> thing
>>>>>>>>> I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not
>>>>>>>>> use
>>>>>>>>> nullfs. (I have no idea if you are using any nullfs mounts,
>>>>>>>>> but
>>>>>>>>> if so, try getting rid of them.)
>>>>>>>>>
>>>>>>>>> Here`s a high level post about the ZFS and vnode locking
>>>>>>>>> problem,
>>>>>>>>> but there is no patch available, as far as I know.
>>>>>>>>>
>>>>>>>>> http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
>>>>>>>>>
>>>>>>>>> rick
>>>>>>>>>
>>>>>>>>> 921 - D 0:00.02 nfsd: server (nfsd)
>>>>>>>>>
>>>>>>>>> Here is the procstat -kk
>>>>>>>>>
>>>>>>>>> PID TID COMM TDNAME KSTACK
>>>>>>>>> 921 100538 nfsd nfsd: master mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
>>>>>>>>> nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>>>> 921 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f
>>>>>>>>> nfsrvd_lock+0x5b1
>>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 921 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 921 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100659 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100660 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100661 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100662 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100663 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100664 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100665 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 921 100666 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>>>>>> nfsrvd_dorpc+0xc76
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Loic Blot wrote:
>>>>>>>>>
>>>>>>>>>> For more informations, here is procstat -kk on nfsd, if you
>>>>>>>>>> need
>>>>>>>>>> more
>>>>>>>>>> hot datas, tell me.
>>>>>>>>>>
>>>>>>>>>> Regards, PID TID COMM TDNAME KSTACK
>>>>>>>>>> 918 100529 nfsd nfsd: master mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
>>>>>>>>>> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
>>>>>>>>>> amd64_syscall+0x351
>>>>>>>>>
>>>>>>>>> Well, most of the threads are stuck like this one, waiting
>>>>>>>>> for
>>>>>>>>> a
>>>>>>>>> vnode
>>>>>>>>> lock in ZFS. All of them appear to be in zfs_fhtovp().
>>>>>>>>> I`m not a ZFS guy, so I can`t help much. I`ll try changing
>>>>>>>>> the
>>>>>>>>> subject line
>>>>>>>>> to include ZFS vnode lock, so maybe the ZFS guys will take a
>>>>>>>>> look.
>>>>>>>>>
>>>>>>>>> The only thing I`ve seen suggested is trying:
>>>>>>>>> sysctl vfs.lookup_shared=0
>>>>>>>>> to disable shared vop_lookup()s. Apparently zfs_lookup()
>>>>>>>>> doesn`t
>>>>>>>>> obey the vnode locking rules for lookup and rename, according
>>>>>>>>> to
>>>>>>>>> the posting I saw.
>>>>>>>>>
>>>>>>>>> I`ve added a couple of comments about the other threads
>>>>>>>>> below,
>>>>>>>>> but
>>>>>>>>> they are all either waiting for an RPC request or waiting for
>>>>>>>>> the
>>>>>>>>> threads stuck on the ZFS vnode lock to complete.
>>>>>>>>>
>>>>>>>>> rick
>>>>>>>>>
>>>>>>>>>> 918 100564 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> Fyi, this thread is just waiting for an RPC to arrive.
>>>>>>>>> (Normal)
>>>>>>>>>
>>>>>>>>>> 918 100565 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100566 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100567 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100568 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100569 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100570 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
>>>>>>>>>> _cv_wait_sig+0x16a
>>>>>>>>>> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>>> fork_trampoline+0xe
>>>>>>>>>> 918 100571 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100572 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>>> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
>>>>>>>>>> nfsrvd_dorpc+0xc76
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> This one (and a few others) are waiting for the nfsv4_lock.
>>>>>>>>> This
>>>>>>>>> happens
>>>>>>>>> because other threads are stuck with RPCs in progress. (ie.
>>>>>>>>> The
>>>>>>>>> ones
>>>>>>>>> waiting on the vnode lock in zfs_fhtovp().)
>>>>>>>>> For these, the RPC needs to lock out other threads to do the
>>>>>>>>> operation,
>>>>>>>>> so it waits for the nfsv4_lock() which can exclusively lock
>>>>>>>>> the
>>>>>>>>> NFSv4
>>>>>>>>> data structures once all other nfsd threads complete their
>>>>>>>>> RPCs
>>>>>>>>> in
>>>>>>>>> progress.
>>>>>>>>>
>>>>>>>>>> 918 100573 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> Same as above.
>>>>>>>>>
>>>>>>>>>> 918 100574 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100575 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100576 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100577 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100578 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100579 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100580 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100581 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100582 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100583 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100584 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100585 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100586 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100587 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100588 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100589 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100590 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100591 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100592 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100593 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100594 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100595 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100596 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100597 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100598 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100599 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100600 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100601 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100602 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100603 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100604 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100605 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100606 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>> 918 100607 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>>> svc_thread_start+0xb
>>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
>>>>>>>>>
>>>>>>>>> 918 100608 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f
>>>>>>>>> nfsrvd_lock+0x5b1
>>>>>>>>> nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100609 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100610 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
>>>>>>>>> nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
>>>>>>>>> fork_trampoline+0xe
>>>>>>>>> 918 100611 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100612 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100613 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100614 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100615 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100616 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100617 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100618 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100619 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100620 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100621 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100622 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100623 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
>>>>>>>>> nfsrvd_dorpc+0x316 nfssvc_program+0x554
>>>>>>>>> svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100624 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100625 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100626 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100627 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100628 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100629 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100630 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100631 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100632 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100633 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100634 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100635 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100636 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100637 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100638 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100639 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100640 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100641 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100642 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100643 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100644 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100645 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100646 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100647 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100648 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100649 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100650 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100651 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100652 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100653 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100654 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100655 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100656 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100657 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>> 918 100658 nfsd nfsd: service mi_switch+0xe1
>>>>>>>>> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
>>>>>>>>> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
>>>>>>>>> zfs_fhtovp+0x38d
>>>>>>>>> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
>>>>>>>>> nfssvc_program+0x554 svc_run_internal+0xc77
>>>>>>>>> svc_thread_start+0xb
>>>>>>>>> fork_exit+0x9a fork_trampoline+0xe
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 15 décembre 2014 13:29 "Loïc Blot"
>>>>>>>>> <loic.blot at unix-experience.fr>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Hmmm...
>>>>>>>>> now i'm experiencing a deadlock.
>>>>>>>>>
>>>>>>>>> 0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
>>>>>>>>> (nfsd)
>>>>>>>>>
>>>>>>>>> the only issue was to reboot the server, but after rebooting
>>>>>>>>> deadlock arrives a second time when i
>>>>>>>>> start my jails over NFS.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 15 décembre 2014 10:07 "Loïc Blot"
>>>>>>>>> <loic.blot at unix-experience.fr>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Hi Rick,
>>>>>>>>> after talking with my N+1, NFSv4 is required on our
>>>>>>>>> infrastructure.
>>>>>>>>> I tried to upgrade NFSv4+ZFS
>>>>>>>>> server from 9.3 to 10.1, i hope this will resolve some
>>>>>>>>> issues...
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 10 décembre 2014 15:36 "Loïc Blot"
>>>>>>>>> <loic.blot at unix-experience.fr>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Hi Rick,
>>>>>>>>> thanks for your suggestion.
>>>>>>>>> For my locking bug, rpc.lockd is stucked in rpcrecv state on
>>>>>>>>> the
>>>>>>>>> server. kill -9 doesn't affect the
>>>>>>>>> process, it's blocked.... (State: Ds)
>>>>>>>>>
>>>>>>>>> for the performances
>>>>>>>>>
>>>>>>>>> NFSv3: 60Mbps
>>>>>>>>> NFSv4: 45Mbps
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>>>>>> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Loic Blot wrote:
>>>>>>>>>
>>>>>>>>>> Hi Rick,
>>>>>>>>>> I'm trying NFSv3.
>>>>>>>>>> Some jails are starting very well but now i have an issue
>>>>>>>>>> with
>>>>>>>>>> lockd
>>>>>>>>>> after some minutes:
>>>>>>>>>>
>>>>>>>>>> nfs server 10.10.X.8:/jails: lockd not responding
>>>>>>>>>> nfs server 10.10.X.8:/jails lockd is alive again
>>>>>>>>>>
>>>>>>>>>> I look at mbuf, but i seems there is no problem.
>>>>>>>>>
>>>>>>>>> Well, if you need locks to be visible across multiple
>>>>>>>>> clients,
>>>>>>>>> then
>>>>>>>>> I'm afraid you are stuck with using NFSv4 and the
>>>>>>>>> performance
>>>>>>>>> you
>>>>>>>>> get
>>>>>>>>> from it. (There is no way to do file handle affinity for
>>>>>>>>> NFSv4
>>>>>>>>> because
>>>>>>>>> the read and write ops are buried in the compound RPC and
>>>>>>>>> not
>>>>>>>>> easily
>>>>>>>>> recognized.)
>>>>>>>>>
>>>>>>>>> If the locks don't need to be visible across multiple
>>>>>>>>> clients,
>>>>>>>>> I'd
>>>>>>>>> suggest trying the "nolockd" option with nfsv3.
>>>>>>>>>
>>>>>>>>>> Here is my rc.conf on server:
>>>>>>>>>>
>>>>>>>>>> nfs_server_enable="YES"
>>>>>>>>>> nfsv4_server_enable="YES"
>>>>>>>>>> nfsuserd_enable="YES"
>>>>>>>>>> nfsd_server_flags="-u -t -n 256"
>>>>>>>>>> mountd_enable="YES"
>>>>>>>>>> mountd_flags="-r"
>>>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>>>>>> rpcbind_enable="YES"
>>>>>>>>>> rpc_lockd_enable="YES"
>>>>>>>>>> rpc_statd_enable="YES"
>>>>>>>>>>
>>>>>>>>>> Here is the client:
>>>>>>>>>>
>>>>>>>>>> nfsuserd_enable="YES"
>>>>>>>>>> nfsuserd_flags="-usertimeout 0 -force 20"
>>>>>>>>>> nfscbd_enable="YES"
>>>>>>>>>> rpc_lockd_enable="YES"
>>>>>>>>>> rpc_statd_enable="YES"
>>>>>>>>>>
>>>>>>>>>> Have you got an idea ?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Loïc Blot,
>>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>>
>>>>>>>>>> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
>>>>>>>>>> a
>>>>>>>>>> écrit:
>>>>>>>>>>> Loic Blot wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi rick,
>>>>>>>>>>>>
>>>>>>>>>>>> I waited 3 hours (no lag at jail launch) and now I do:
>>>>>>>>>>>> sysrc
>>>>>>>>>>>> memcached_flags="-v -m 512"
>>>>>>>>>>>> Command was very very slow...
>>>>>>>>>>>>
>>>>>>>>>>>> Here is a dd over NFS:
>>>>>>>>>>>>
>>>>>>>>>>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>>>>>>>>>>> bytes/sec)
>>>>>>>>>>>
>>>>>>>>>>> Can you try the same read using an NFSv3 mount?
>>>>>>>>>>> (If it runs much faster, you have probably been bitten by
>>>>>>>>>>> the
>>>>>>>>>>> ZFS
>>>>>>>>>>> "sequential vs random" read heuristic which I've been told
>>>>>>>>>>> things
>>>>>>>>>>> NFS is doing "random" reads without file handle affinity.
>>>>>>>>>>> File
>>>>>>>>>>> handle affinity is very hard to do for NFSv4, so it isn't
>>>>>>>>>>> done.)
>>>>>>>>>
>>>>>>>>> I was actually suggesting that you try the "dd" over nfsv3
>>>>>>>>> to
>>>>>>>>> see
>>>>>>>>> how
>>>>>>>>> the performance compared with nfsv4. If you do that, please
>>>>>>>>> post
>>>>>>>>> the
>>>>>>>>> comparable results.
>>>>>>>>>
>>>>>>>>> Someday I would like to try and get ZFS's sequential vs
>>>>>>>>> random
>>>>>>>>> read
>>>>>>>>> heuristic modified and any info on what difference in
>>>>>>>>> performance
>>>>>>>>> that
>>>>>>>>> might make for NFS would be useful.
>>>>>>>>>
>>>>>>>>> rick
>>>>>>>>>
>>>>>>>>> rick
>>>>>>>>>
>>>>>>>>> This is quite slow...
>>>>>>>>>
>>>>>>>>> You can found some nfsstat below (command isn't finished
>>>>>>>>> yet)
>>>>>>>>>
>>>>>>>>> nfsstat -c -w 1
>>>>>>>>>
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 0 0 0 0 0 16 0
>>>>>>>>> 2 0 0 0 0 0 17 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 0 0 0 0 0 3 0
>>>>>>>>> 0 0 0 0 0 0 3 0
>>>>>>>>> 37 10 0 8 0 0 14 1
>>>>>>>>> 18 16 0 4 1 2 4 0
>>>>>>>>> 78 91 0 82 6 12 30 0
>>>>>>>>> 19 18 0 2 2 4 2 0
>>>>>>>>> 0 0 0 0 2 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 1 0 0 0 0 1 0
>>>>>>>>> 4 6 0 0 6 0 3 0
>>>>>>>>> 2 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 1 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 1 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 6 108 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 98 54 0 86 11 0 25 0
>>>>>>>>> 36 24 0 39 25 0 10 1
>>>>>>>>> 67 8 0 63 63 0 41 0
>>>>>>>>> 34 0 0 35 34 0 0 0
>>>>>>>>> 75 0 0 75 77 0 0 0
>>>>>>>>> 34 0 0 35 35 0 0 0
>>>>>>>>> 75 0 0 74 76 0 0 0
>>>>>>>>> 33 0 0 34 33 0 0 0
>>>>>>>>> 0 0 0 0 5 0 0 0
>>>>>>>>> 0 0 0 0 0 0 6 0
>>>>>>>>> 11 0 0 0 0 0 11 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 17 0 0 0 0 1 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 4 5 0 0 0 0 12 0
>>>>>>>>> 2 0 0 0 0 0 26 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 0 0 0 0 0 2 0
>>>>>>>>> 2 0 0 0 0 0 24 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 0 0 0 0 0 7 0
>>>>>>>>> 2 1 0 0 0 0 1 0
>>>>>>>>> 0 0 0 0 2 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 6 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 6 0 0 0 0 3 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 2 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 71 0 0 0 0 0 0
>>>>>>>>> 0 1 0 0 0 0 0 0
>>>>>>>>> 2 36 0 0 0 0 1 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 1 0 0 0 0 0 1 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 79 6 0 79 79 0 2 0
>>>>>>>>> 25 0 0 25 26 0 6 0
>>>>>>>>> 43 18 0 39 46 0 23 0
>>>>>>>>> 36 0 0 36 36 0 31 0
>>>>>>>>> 68 1 0 66 68 0 0 0
>>>>>>>>> GtAttr Lookup Rdlink Read Write Rename Access Rddir
>>>>>>>>> 36 0 0 36 36 0 0 0
>>>>>>>>> 48 0 0 48 49 0 0 0
>>>>>>>>> 20 0 0 20 20 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 3 14 0 1 0 0 11 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 0 4 0 0 0 0 4 0
>>>>>>>>> 0 0 0 0 0 0 0 0
>>>>>>>>> 4 22 0 0 0 0 16 0
>>>>>>>>> 2 0 0 0 0 0 23 0
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 8 décembre 2014 09:36 "Loïc Blot"
>>>>>>>>> <loic.blot at unix-experience.fr> a
>>>>>>>>> écrit:
>>>>>>>>>> Hi Rick,
>>>>>>>>>> I stopped the jails this week-end and started it this
>>>>>>>>>> morning,
>>>>>>>>>> i'll
>>>>>>>>>> give you some stats this week.
>>>>>>>>>>
>>>>>>>>>> Here is my nfsstat -m output (with your rsize/wsize
>>>>>>>>>> tweaks)
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
>>>>>>>>
>>>>>>>>> s=2147483647
>>>>>>>>>
>>>>>>>>> On server side my disks are on a raid controller which show a
>>>>>>>>> 512b
>>>>>>>>> volume and write performances
>>>>>>>>> are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
>>>>>>>>> count=100000000 => 450MBps)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>>
>>>>>>>>> 5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
>>>>>>>>> écrit:
>>>>>>>>>
>>>>>>>>> Loic Blot wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> i'm trying to create a virtualisation environment based on
>>>>>>>>> jails.
>>>>>>>>> Those jails are stored under a big ZFS pool on a FreeBSD
>>>>>>>>> 9.3
>>>>>>>>> which
>>>>>>>>> export a NFSv4 volume. This NFSv4 volume was mounted on a
>>>>>>>>> big
>>>>>>>>> hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
>>>>>>>>> only 1
>>>>>>>>> was
>>>>>>>>> used at this time).
>>>>>>>>>
>>>>>>>>> The problem is simple, my hypervisors runs 6 jails (used 1%
>>>>>>>>> cpu
>>>>>>>>> and
>>>>>>>>> 10GB RAM approximatively and less than 1MB bandwidth) and
>>>>>>>>> works
>>>>>>>>> fine at start but the system slows down and after 2-3 days
>>>>>>>>> become
>>>>>>>>> unusable. When i look at top command i see 80-100% on
>>>>>>>>> system
>>>>>>>>> and
>>>>>>>>> commands are very very slow. Many process are tagged with
>>>>>>>>> nfs_cl*.
>>>>>>>>>
>>>>>>>>> To be honest, I would expect the slowness to be because of
>>>>>>>>> slow
>>>>>>>>> response
>>>>>>>>> from the NFSv4 server, but if you do:
>>>>>>>>> # ps axHl
>>>>>>>>> on a client when it is slow and post that, it would give us
>>>>>>>>> some
>>>>>>>>> more
>>>>>>>>> information on where the client side processes are sitting.
>>>>>>>>> If you also do something like:
>>>>>>>>> # nfsstat -c -w 1
>>>>>>>>> and let it run for a while, that should show you how many
>>>>>>>>> RPCs
>>>>>>>>> are
>>>>>>>>> being done and which ones.
>>>>>>>>>
>>>>>>>>> # nfsstat -m
>>>>>>>>> will show you what your mount is actually using.
>>>>>>>>> The only mount option I can suggest trying is
>>>>>>>>> "rsize=32768,wsize=32768",
>>>>>>>>> since some network environments have difficulties with 64K.
>>>>>>>>>
>>>>>>>>> There are a few things you can try on the NFSv4 server side,
>>>>>>>>> if
>>>>>>>>> it
>>>>>>>>> appears
>>>>>>>>> that the clients are generating a large RPC load.
>>>>>>>>> - disabling the DRC cache for TCP by setting
>>>>>>>>> vfs.nfsd.cachetcp=0
>>>>>>>>> - If the server is seeing a large write RPC load, then
>>>>>>>>> "sync=disabled"
>>>>>>>>> might help, although it does run a risk of data loss when
>>>>>>>>> the
>>>>>>>>> server
>>>>>>>>> crashes.
>>>>>>>>> Then there are a couple of other ZFS related things (I'm not
>>>>>>>>> a
>>>>>>>>> ZFS
>>>>>>>>> guy,
>>>>>>>>> but these have shown up on the mailing lists).
>>>>>>>>> - make sure your volumes are 4K aligned and ashift=12 (in
>>>>>>>>> case a
>>>>>>>>> drive
>>>>>>>>> that uses 4K sectors is pretending to be 512byte sectored)
>>>>>>>>> - never run over 70-80% full if write performance is an
>>>>>>>>> issue
>>>>>>>>> - use a zil on an SSD with good write performance
>>>>>>>>>
>>>>>>>>> The only NFSv4 thing I can tell you is that it is known that
>>>>>>>>> ZFS's
>>>>>>>>> algorithm for determining sequential vs random I/O fails for
>>>>>>>>> NFSv4
>>>>>>>>> during writing and this can be a performance hit. The only
>>>>>>>>> workaround
>>>>>>>>> is to use NFSv3 mounts, since file handle affinity
>>>>>>>>> apparently
>>>>>>>>> fixes
>>>>>>>>> the problem and this is only done for NFSv3.
>>>>>>>>>
>>>>>>>>> rick
>>>>>>>>>
>>>>>>>>> I saw that there are TSO issues with igb then i'm trying to
>>>>>>>>> disable
>>>>>>>>> it with sysctl but the situation wasn't solved.
>>>>>>>>>
>>>>>>>>> Someone has got ideas ? I can give you more informations if
>>>>>>>>> you
>>>>>>>>> need.
>>>>>>>>>
>>>>>>>>> Thanks in advance.
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Loïc Blot,
>>>>>>>>> UNIX Systems, Network and Security Engineer
>>>>>>>>> http://www.unix-experience.fr
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list