kern/151648: [zfs] disk wait bug

[Bartosz Nowicki]

>Number:         151648
>Category:       kern
>Synopsis:       [zfs] disk wait bug
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Oct 22 23:30:11 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Bartosz Nowicki
>Release:        8.1-STABLE
>Organization:
admin.net.pl
>Environment:
FreeBSD mydevil.mydevil.net 8.1-STABLE FreeBSD 8.1-STABLE #4: Fri Oct  1 00:04:06 UTC 2010     root@:/usr/obj/usr/src/sys/mydevil  amd64
>Description:
During normal usage, nginx www server stopped responding. Problem looks like ZFS issue:

# ps lU www
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
   80  3645 67746   0  54 10 46012 33764 kqread SN    ??    0:00,06 nginx: worker process (nginx)
   80  8410     1   0  54 10 46012 34120 zfs    DN    ??    0:01,14 nginx: worker process (nginx)
   80 15225 67746   0  54 10 46012 33388 zfs    DN    ??    0:02,50 nginx: worker process (nginx)
   80 15226 67746   0  54 10 46012 33372 zfs    DN    ??    0:00,47 nginx: worker process (nginx)
   80 18427     1   0  54 10 46012 34160 zfs    DN    ??    6:51,11 nginx: worker process (nginx)
   80 22955 67746   0  54 10 46012 33416 zfs    DN    ??    0:01,07 nginx: worker process (nginx)
   80 22956 67746   0  54 10 46012 33400 zfs    DN    ??    0:00,68 nginx: worker process (nginx)
   80 26904 67746   0  54 10 46012 33776 zfs    DN    ??    0:00,53 nginx: worker process (nginx)
   80 26905 67746   0  54 10 46012 33424 zfs    DN    ??    0:00,96 nginx: worker process (nginx)
   80 29355     1   0  54 10 46012 34136 zfs    DN    ??    0:38,33 nginx: worker process (nginx)
   80 29356     1   0  54 10 46012 34144 zfs    DN    ??    0:40,52 nginx: worker process (nginx)
   80 43125 67746   0  54 10 46012 33700 zfs    DN    ??    0:00,15 nginx: worker process (nginx)
   80 94412 67746   0  54 10 46012 33724 zfs    DN    ??    0:00,01 nginx: worker process (nginx)

# procstat -k 26904
  PID    TID COMM             TDNAME           KSTACK                       
26904 100390 nginx            -                mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vget cache_lookup vfs_cache_lookup VOP_LOOKUP_APV lookup namei vn_open_cred kern_openat syscall Xfast_syscall 

# procstat -k 29355
  PID    TID COMM             TDNAME           KSTACK                       
29355 102233 nginx            -                mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vget cache_lookup vfs_cache_lookup VOP_LOOKUP_APV lookup namei vn_open_cred kern_openat syscall Xfast_syscall

Processes are also immportal. Kill gives no result.

# kill -9 29355
# ps lp 29355
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
   80 29355     1   0  54 10 46012 34136 zfs    DN    ??    0:38,33 nginx: worker process (nginx)
>How-To-Repeat:
Start nginx server, wait several hours.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: