misc/167065: [zfs] boot fails when a spare is the boot disk

Peter Maloney peter.maloney at brockmann-consult.de
Wed Apr 18 16:10:06 UTC 2012

>Number:         167065
>Category:       misc
>Synopsis:       [zfs] boot fails when a spare is the boot disk
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Apr 18 16:10:05 UTC 2012
>Originator:     Peter Maloney
>Release:        8-STABLE Feb 4 2012
Brockmann Consult
FreeBSD bcnas1bak.bc.local 8.2-STABLE-20120204 FreeBSD 8.2-STABLE-20120204 #1: Fri Feb 10 15:35:01 CET 2012     root at bcnas1bak.bc.local:/usr/obj/usr/src/sys/GENERIC  amd64

The story:

A disk faulted. It was the first disk in the chassis (so it is the one the controller selects to boot). 

I added a spare (as the 2nd disk in the chassis), and replaced the faulted disk with the spare. 

Then I got the system to hang for unrelated reasons, removed the bad disk, and rebooted.

(unrelated reasons: I added some zpool v16 zfs v4 disks I created on FreeBSD, tried mouting in Linux with zfs-fuse, which told me a device was missing, and then put them back in the FreeBSD and imported, which also said a device was missing, and then "zpool import -m -F -N" or something similar made it hang. parted on Linux said the disks were ext3, which may be related to the problem.)

And on boot, to my surprise, it complained saying I can only boot from mirrors, raidz, etc. (a list which did not include spares). The second disk was defective in a different way, so I didn't try putting it first in the chassis, but that is a workaround, not a bug fix.
Create a virtual machine with disks in this order:

boot mirror disk 1
spare disk
boot mirror disk 2

Boot it up. (using gpt slices, matching what I set up before). Replace the boot mirror disk 1 with the spare with:
# gpart create -s gpt da1
# gpart add -b 34 -s 128 -t freebsd-boot da1
# gpart add -b 129024 -s 32768 -t freebsd-swap da1
# gpart add -b 258048 -s 8386560 -t freebsd-zfs -l sp_root0 da1
# zpool add zroot spare gpt/sp_root0
# zpool replace zroot gpt/root0 gpt/sp_root0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

Wait for resilver.

$ zpool status zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 1.33G in 0h5m with 0 errors on Wed Apr 18 19:42:19 2012

        NAME                  STATE     READ WRITE CKSUM
        zroot                 ONLINE       0     0     0
          mirror-0            ONLINE       0     0     0
            spare-0           ONLINE       0     0     0
              gpt/root0       ONLINE       0     0     0
              gpt/sp_root0    ONLINE       0     0     0
            gpt/root1         ONLINE       0     0     0
          665168513187118073  INUSE     was /dev/gpt/sp_root0

errors: No known data errors

# shutdown -p now

Remove disk 1 and 3 (the old boot disks).

Start up again.

During boot, you get this output:
ZFS: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0

FreeBSD/x86 boot
Default: zroot:/boot/kernel/kernel
ZFS: unexpected object set type 0

FreeBSD/x86 boot
Default: zroot:/boot/kernel/kernel
Workaround was to boot the DVD, load the zfs modules, and then "zpool detach" the bad disk (converting the spare to a normal mirror), and detach the other root disk, which I wanted to remove anyway (to upgrade firmware).


More information about the freebsd-bugs mailing list