kern/115856: ZFS thought it was degraded when it should have been faulted

[Troy Ablan]

>Number:         115856
>Category:       kern
>Synopsis:       ZFS thought it was degraded when it should have been faulted
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 27 09:50:01 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Troy Ablan
>Release:        7.0-CURRENT-200708
>Organization:
>Environment:
>Description:
For some reason, zfs thought a zpool was degraded when it should have been faulted.  The kernel panicked at this point.  The devs are geli-encrypted.  I was bringing them online one-by-one.  There was a double failure during a replace of one of the devices on the previous boot.

[root at fbsd ~]# zpool status
  pool: whomp
 state: FAULTED
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: none requested
config:

        NAME                  STATE     READ WRITE CKSUM
        whomp                 FAULTED      6     0     0  corrupted data
          raidz2              DEGRADED     6     0     0
            ad0s2.eli         ONLINE       0     0     0
            ad1s2             UNAVAIL      0     0     0  corrupted data
            ad2s2             UNAVAIL      0     0     0  corrupted data
            ad3s2             UNAVAIL      0     0     0  corrupted data
            ad12s2.eli        UNAVAIL      0     0     0  cannot open
            replacing         UNAVAIL      0     0     0  insufficient replicas
              ad13s2.eli/old  UNAVAIL      0     0     0  cannot open
              ad13s2.eli      UNAVAIL      0     0     0  cannot open
            ad14s2            UNAVAIL      0     0     0  corrupted data
            ad15s2            UNAVAIL      0     0     0  corrupted data
[root at fbsd ~]# zpool status
  pool: whomp
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: none requested
config:

        NAME                  STATE     READ WRITE CKSUM
        whomp                 DEGRADED     0     0     0
          raidz2              DEGRADED     0     0     0
            ad0s2.eli         ONLINE       0     0     0
            ad1s2.eli         ONLINE       0     0     0
            ad2s2.eli         ONLINE       0     0     0
            ad3s2.eli         ONLINE       0     0     0
            ad12s2.eli        UNAVAIL      0     0     0  cannot open
            replacing         UNAVAIL      0     0     0  insufficient replicas
              ad13s2.eli/old  UNAVAIL      0     0     0  cannot open
              ad13s2.eli      UNAVAIL      0     0     0  cannot open
            ad14s2.eli        ONLINE       0     0     0
            ad15s2            UNAVAIL      0     0     0  corrupted data

errors: No known data errors

>How-To-Repeat:
create a raidz2 of 8 geli devices (or devices that wouldn't be available at boot)
geli attach them one by one
offline one of the devices, and zpool replace it (in place)
hard-reboot before the replace finishes

geli attach the devices one by one, doing a zpool status in between each one.  zfs will think the pool is degraded while it's still missing three.

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: