Next Steps to Debug ZFS Hang?

[Nick Sivo]

Hello,


I've been having trouble with ZFS on my server. For the most part it works splendidly, but occasionally I'll experience permanent hangs.


For example, right now on one of my ZFS filesystems (the others are fine), I can read, write, and stat files, but if I run ls in any directory, ls and the terminal will hang. CTRL-C, and kill -9 can't kill it:


In top:
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND

 5868 nsivo         1  20    0 14456K  1016K zfs     0   0:00  0.00% ls


In ps:
USER      PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED        TIME COMMAND

nsivo    5868   0.0  0.0   14456    1016  2- D+    2:35PM     0:00.00 ls


Eventually the entire system hangs, and can't be shutdown cleanly.


What are the next steps to debug this? I'm a software developer, but am not familiar with kernel debugging. Is there a way to discover in which syscall ls is stuck? Ideally without requiring a crash dump?


Thanks for reading,
Nick



-Nick