Processes wedging on ZFS accesses

[David Adam]

Hi all,

We're still having trouble with our 10.3-RELEASE-p3 fileserver using a ZFS 
pool.

After a certain amount of uptime (usually a week or so), a Samba process 
will get stuck in D-state:

max  2075  0.0  0.2 339928 26616  -  D    26May16      0:19.59 
/usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf

Running find(1) over the hierarchy that the smbd process has open will 
also wedge in a D-state.

Our backups also seem to get stuck, presumably in the same spot.

`procstat -k` on the stuck processes (smbd and our stuck python-based 
backup program) shows:
  PID    TID COMM             TDNAME           KSTACK
 2075 100587 smbd             -                mi_switch+0xe1 
sleepq_wait+0x3a _sx_slock_hard+0x31b namei+0x1c5 vn_open_cred+0x24d 
zfs_getextattr+0x1f2 VOP_GETEXTATTR_APV+0xa7 extattr_get_vp+0x15d 
sys_extattr_get_file+0xf4 amd64_syscall+0x40f Xfast_syscall+0xfb
 2075 100623 smbd             -                mi_switch+0xe1 
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xca0 vop_stdlock+0x3c 
VOP_LOCK1_APV+0xab _vn_lock+0x43 knlist_remove_kq+0x24 filt_vfsdetach+0x22 
knote_fdclose+0xef closefp+0x42 amd64_syscall+0x40f Xfast_syscall+0xfb
21676 101572 python2.7        -                mi_switch+0xe1 
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x91a vop_stdlock+0x3c 
VOP_LOCK1_APV+0xab _vn_lock+0x43 vget+0x73 cache_lookup+0x5d5 
vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x4d4 
kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x40f 
Xfast_syscall+0xfb
36144 101585 python2.7        -                mi_switch+0xe1 
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x91a vop_stdlock+0x3c 
VOP_LOCK1_APV+0xab _vn_lock+0x43 vget+0x73 cache_lookup+0x5d5 
vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x4d4 
kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x40f 
Xfast_syscall+0xfb

Memory doesn't appear to be a problem, and we have the ARC wired to 10 GB 
maximum:
Mem: 80M Active, 2127M Inact, 13G Wired, 36K Cache, 1643M Buf, 308M Free
ARC: 9921M Total, 7056M MFU, 2434M MRU, 977K Anon, 162M Header, 267M Other
Swap: 20G Total, 20G Free

I'm getting a DDB kernel built, but is there any other information that is 
useful? One of the problems is that Samba won't start a new process for 
the user whose process is already wedged, so eventually user logins stop 
working.

Thanks

David Adam
zanchey at ucc.gu.uwa.edu.au