kern/140888: boot fail from zfs root while the pool resilvering

[Alexei Volkov]

>Number:         140888
>Category:       kern
>Synopsis:       boot fail from zfs root while the pool resilvering
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 26 07:20:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Alexei Volkov
>Release:        8.0-RELEASE
>Organization:
SoftLynx
>Environment:
FreeBSD livecd8 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Tue Nov 24 08:29:59 UTC 2009 root at 80AMD64:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
On the system boots directly from ZFS mirror or raidz pool (http://wiki.freebsd.org/RootOnZFS) replacing one of the components with subsequent accident power failure or reboot while the resilvering is running, stops boot process with the message:

ZFS: can only boot from disk, mirror or raidz vdevs
ZFS: inconsistent nvlist contents
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: tank0:/boot/kernel/kernel
boot:
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: tank0:/boot/kernel/kernel
boot:


>How-To-Repeat:
Install the system as described on http://wiki.freebsd.org/RootOnZFS  for non single device installation i.e. mirror, raidz.

Eventually has something like 

[root at fresh-inst:~]# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME              STATE     READ WRITE CKSUM
        tank0             ONLINE       0     0     0
          raidz1          ONLINE       0     0     0
            gpt/QM00002   ONLINE       0     0     0
            gpt/SN091234  ONLINE       0     0     0

errors: No known data errors

Lets assume that one of the component has a prefail condition and have to be replaced with new one. Power off the system and replace one of the HDD with onother one. Boot back to OS. Booting just fine for now.

Get zpool status to see missing component.

[root at fresh-inst:~]# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

        NAME              STATE     READ WRITE CKSUM
        tank0             DEGRADED     0     0     0
          raidz1          DEGRADED     0     0     0
            gpt/QM00002   UNAVAIL      0   327     0  cannot open
            gpt/SN091234  ONLINE       0     0     0

errors: No known data errors

Partition the new disk as required and get new gpt component ready for zpool replacement.

[root at fresh-inst:~]# gpart show -l
=>     34  8388541  ad0  GPT  (4.0G)
       34      128    1  (null)  (64K)
      162  8388413    2  SN091234  (4.0G)

=>     34  8388541  ad1  GPT  (4.0G)
       34      128    1  (null)  (64K)
      162  8388413    2  SN023432  (4.0G)

Run replacement command.

[root at fresh-inst:~]# zpool replace tank0 gpt/QM00002 gpt/SN023432

[root at fresh-inst:~]# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 7.88% done, 0h4m to go
config:

        NAME                STATE     READ WRITE CKSUM
        tank0               DEGRADED     0     0     0
          raidz1            DEGRADED     0     0     0
            replacing       DEGRADED     0     0     0
              gpt/QM00002   UNAVAIL      0 2.17K     0  cannot open
              gpt/SN023432  ONLINE       0     0     0  39.5M resilvered
            gpt/SN091234    ONLINE       0     0     0  372K resilvered

errors: No known data errors

Initiate regular reboot (but could simulate be an instant power failure).

[root at fresh-inst:~]# reboot

The systems fails to boot with the following message:

Booting from Hard Disk...
ZFS: can only boot from disk, mirror or raidz vdevs
ZFS: inconsistent nvlist contents
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: tank0:/boot/kernel/kernel
boot:
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: tank0:/boot/kernel/kernel
boot:





>Fix:
As a workaround reboot from CD/DVD in fixit mode, then import the pool and wait resilvering complete.

>Release-Note:
>Audit-Trail:
>Unformatted: