nfsd hang in sosend_generic
Nikolay Denev
ndenev at gmail.com
Wed Nov 21 09:02:12 UTC 2012
Hello,
First of all, I'm not sure if this is actually nfsd issue and not network stack issue.
I've just had nfsd hang in unkillable state while doing some IO from Linux host running Oracle DB using Oracle's Direct NFS.
I was watching from some time how the Direct NFS client loads the NFS server differently, i.e.:
with the linux kernel NFS client I see single TCP session to port 2049 and all traffic goes there, while the Direct NFS client
is much more aggressive and creates multiple TCP sessions, and often was able to generate pretty big Send/Recv-Q's on FreeBSD's side.
I'm mentioning this as probably is related.
Here's the procstat -kk of the hanged nfsd process :
PID TID COMM TDNAME KSTACK
1221 100550 nfsd nfsd: master mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_run+0x8f nfsrvd_nfsd+0x193 nfssvc_nfsd+0x9b sys_nfssvc+0x90 amd64_syscall+0x5ea Xfast_syscall+0xf7
1221 101286 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sleep+0x2ad sosend_generic+0x25f svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101287 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101288 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101317 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101318 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101319 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101320 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101321 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101322 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101323 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101324 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101325 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101326 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101327 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101328 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101329 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101330 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101331 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101332 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101333 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101334 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101335 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101336 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101337 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101338 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101339 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101340 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101341 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101342 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101343 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101344 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101345 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101346 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101347 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101348 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101349 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101350 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101351 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101352 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101353 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101354 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101355 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101356 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101357 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101358 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101359 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101360 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101361 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101362 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101363 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101364 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101365 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101366 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101367 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101368 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101369 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101370 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101371 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101372 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101373 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101374 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101375 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101376 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101377 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101378 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101379 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101380 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101381 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101382 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101383 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101384 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101385 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101386 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101387 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101388 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101389 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101390 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101391 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101392 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101393 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101394 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101395 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101396 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101397 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101398 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101399 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101400 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101401 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101402 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101403 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101404 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101405 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101406 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101407 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101408 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101409 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101410 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101411 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101412 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101413 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101414 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101415 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101416 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101417 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101418 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101419 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101420 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101421 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101422 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101423 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101424 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101425 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101426 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101427 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101428 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101429 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101430 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101431 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101432 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101433 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101434 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101435 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101436 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101437 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101438 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1221 101439 nfsd nfsd: service mi_switch+0x194 sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
Here is a netstat output for the nfs sessions from FreeBSD server side:
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 37215456 10.101.0.1.2049 10.101.0.2.42856 ESTABLISHED
tcp4 0 14561020 10.101.0.1.2049 10.101.0.2.62854 FIN_WAIT_1
tcp4 0 3068132 10.100.0.1.2049 10.100.0.2.9712 FIN_WAIT_1
Linux host sees this :
tcp 1 0 10.101.0.2:9270 10.101.0.1:2049 CLOSE_WAIT
tcp 477940 0 10.100.0.2:9712 10.100.0.1:2049 ESTABLISHED
tcp 1 0 10.101.0.2:10588 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:12254 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:12438 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:17583 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:20285 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:20678 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:22892 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:28850 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:33851 10.100.0.1:2049 CLOSE_WAIT
tcp 165 0 10.100.0.2:34190 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:35643 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:39498 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:39724 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:40742 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:41674 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:42942 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:42956 10.100.0.1:2049 CLOSE_WAIT
tcp 477976 0 10.101.0.2:42856 10.101.0.1:2049 ESTABLISHED
tcp 1 0 10.100.0.2:42045 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:42048 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:43063 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:44771 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:49568 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:50813 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:51418 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:54507 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:57201 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:58553 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:59638 10.101.0.1:2049 CLOSE_WAIT
tcp 1 0 10.100.0.2:62289 10.100.0.1:2049 CLOSE_WAIT
tcp 1 0 10.101.0.2:61848 10.101.0.1:2049 CLOSE_WAIT
tcp 476952 0 10.101.0.2:62854 10.101.0.1:2049 ESTABLISHED
Then I used "tcpdrop" on FreeBSD's side to drop the sessions, the nfsd was able to die and be restarted.
During the "hanged" period, all NFS mounts from the Linux host were inaccessible, and IO hanged.
The nfsd is running with drc2/drc3 and lkshared patches from Rick Macklem.
More information about the freebsd-fs
mailing list