Zfs locking up process

[Rajil Saraswat]

Hello

I have server running Freenas 9.3 with a few jails. The machine has two new
disks setup in mirror. I have a dataset (/mnt/tank/media) which is shared
in two jails.

Unfortunately, sometimes when I do a ls in a jail in the shared directory I
see that the process just hangs.

Today in the jail I did an 'su'  and process just hung. On the host if i do
ls /mnt/tank/media it also hangs.

The su process (pid 77477) is taking up 100% cpu in the jail. It seems that
zfs is holding up the process. Any idea what could be wrong?

Thanks
Rajil

On the host
[root at freenas] ~# ps O nwchan
  PID NWCHAN           TT  STAT     TIME COMMAND
14248 fffffe000962c8a8 v0  Is+   0:00.73 python /etc/netcli (python2.7)
14249 fffffe000962c4a8 v1  Is+   0:00.00 /usr/libexec/getty Pc ttyv1
14250 fffffe000962c0a8 v2  Is+   0:00.00 /usr/libexec/getty Pc ttyv2
14251 fffffe000962cca8 v3  Is+   0:00.00 /usr/libexec/getty Pc ttyv3
14252 fffffe00096284a8 v4  Is+   0:00.00 /usr/libexec/getty Pc ttyv4
14253 fffffe000962eca8 v5  Is+   0:00.00 /usr/libexec/getty Pc ttyv5
14254 fffffe000962e8a8 v6  Is+   0:00.00 /usr/libexec/getty Pc ttyv6
14255 fffffe000962e4a8 v7  Is+   0:00.00 /usr/libexec/getty Pc ttyv7
77477 -                 2- R+J  36:49.59 _su (csh)
77541 fffffe0043d31488  3- D+J   0:00.01 _su (csh)
45586 fffffe00a44050a0  0  Is    0:00.02 -csh (csh)
74258 fffffe00076824a8  0  I+J   0:00.01 csh
77586 fffffe0043d31488  4  Ds+   0:00.01 -csh (csh)
77621 fffffe0043d31488  5- D+    0:00.00 ls -GFa
77626 fffffe01e7a214a8  6  Is+   0:00.02 -csh (csh)
79795 fffffe0055e540a0  8  Ss    0:00.01 -csh (csh)
79798 -                 8  R+    0:00.00 ps O nwchan

[root at freenas] ~# ps O wchan
  PID WCHAN TT  STAT     TIME COMMAND
14248 ttyin v0  Is+   0:00.73 python /etc/netcli (python2.7)
14249 ttyin v1  Is+   0:00.00 /usr/libexec/getty Pc ttyv1
14250 ttyin v2  Is+   0:00.00 /usr/libexec/getty Pc ttyv2
14251 ttyin v3  Is+   0:00.00 /usr/libexec/getty Pc ttyv3
14252 ttyin v4  Is+   0:00.00 /usr/libexec/getty Pc ttyv4
14253 ttyin v5  Is+   0:00.00 /usr/libexec/getty Pc ttyv5
14254 ttyin v6  Is+   0:00.00 /usr/libexec/getty Pc ttyv6
14255 ttyin v7  Is+   0:00.00 /usr/libexec/getty Pc ttyv7
77477 -      2- R+J  36:49.59 _su (csh)
77541 zfs    3- D+J   0:00.01 _su (csh)
45586 pause  0  Is    0:00.02 -csh (csh)
74258 ttyin  0  I+J   0:00.01 csh
77586 zfs    4  Ds+   0:00.01 -csh (csh)
77621 zfs    5- D+    0:00.00 ls -GFa
77626 ttyin  6  Is+   0:00.02 -csh (csh)
79795 pause  8  Ss    0:00.01 -csh (csh)
79800 -      8  R+    0:00.00 ps O wchan

Inside the jail

[r at sshserver ~]$ ps -axl
UID   PID  PPID CPU PRI NI   VSZ  RSS MWCHAN STAT TT      TIME COMMAND
   0  9438     1   0  31  0 12084 1916 select IsJ  ??   0:00.00 dhclient:
epair3b [priv] (dhclient)
  65  9490     1   0  20  0 12084 1976 select IsJ  ??   0:00.00 dhclient:
epair3b (dhclient)
   0  9878     1   0  20  0 12080 1868 select SsJ  ??   0:00.03
/usr/sbin/syslogd -s
   0  9948     1   0  20  0 14184 1860 nanslp IsJ  ??   0:00.03
/usr/sbin/cron -s
   0 45452     1   0  22  0 52036 4916 select IsJ  ??   0:00.01 sshd: r
[priv] (sshd)
1001 45454 45452   0  20  0 52036 5000 select IJ   ??   0:00.01 sshd: r at pts/1
(sshd)
   0 71094     1   0  20  0 26728 4456 select SsJ  ??   0:00.00
/usr/local/sbin/sshd
   0 79100 71094   0  22  0 52036 4912 select IsJ  ??   0:00.01 sshd: r
[priv] (sshd)
1001 79102 79100   0  20  0 52036 4948 select IJ   ??   0:00.00 sshd: r at pts/7
(sshd)
   0 80400 71094   0  23  0 52036 4912 select IsJ  ??   0:00.01 sshd: r
[priv] (sshd)
1001 80402 80400   0  20  0 52036 4920 select IJ   ??   0:00.00 sshd: r at pts/8
(sshd)
   0 80634 71094   0  22  0 52036 4912 select SsJ  ??   0:00.01 sshd: r
[priv] (sshd)
1001 80636 80634   0  20  0 52036 4920 select SJ   ??   0:00.00 sshd: r at pts/9
(sshd)
   0 77477     1   0  20  0 17568 2556 -      R+J   2- 76:29.08 _su (csh)
   0 77541     1   0  20  0 17568 3156 zfs    D+J   3-  0:00.01 _su (csh)
1001 45455 45454   0  20  0 17512 3252 ttyin  Is+J  1   0:00.00 -bash (bash)
1001 79103 79102   0  20  0 17512 3332 ttyin  Is+J  7   0:00.00 -bash (bash)
1001 80403 80402   0  20  0 17512 3332 ttyin  Is+J  8   0:00.00 -bash (bash)
1001 80637 80636   0  20  0 17512 3332 wait   SsJ   9   0:00.00 -bash (bash)
1001 80641 80637   0  20  0 16296 1852 -      R+J   9   0:00.00 ps -axl