kern/164861: Cannot boot from zfs built in degraded mode

Chris Jones chris at
Tue Feb 7 17:10:15 UTC 2012

>Number:         164861
>Category:       kern
>Synopsis:       Cannot boot from zfs built in degraded mode
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 07 17:10:14 UTC 2012
>Originator:     Chris Jones
>Release:        FreeBSD 9.0-RELEASE amd64
System: FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root at amd64

If a raidz1 or mirror is built in degraded mode, it will be unbootable
(using GPT). Adding the missing disk and resilvering will render it
bootable, even if the replaced disk is removed again prior to boot.

I've reproduced this in various combinations of raidz1 and mirror; USB
flash disks and SATA drives; i386 and amd64.

Upon booting, these messages are printed by the primary/secondary
bootstrap loader:
ZFS: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
...then it tries to directly load the kernel a couple of times and
gives up.
Using a 4GB USB flash drive da0:

gpart create -s gpt da0
gpart add -s 64k -t freebsd-boot -l boot0 da0
gpart add -t freebsd-zfs -l disk0 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

dd if=/dev/zero of=/usr/bigfile oseek=4G count=1

zpool create -f -o altroot=/mnt zroot raidz1 /dev/gpt/disk0 /usr/bigfile

dump 0afL - / | ( cd /mnt && restore rf - )

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot
mv /tmp/zpool.cache /mnt/boot/zfs/zpool.cache
zfs unmount -a
zpool set bootfs=zroot zroot
zpool set cachefile='' zroot
zfs set mountpoint=legacy zroot
Going back to a running system, I inserted the 4GB flash disk da0 plus
another 4GB flash disk da1. Then:

gpart create -s gpt da1
gpart add -s 64k -t freebsd-boot -l boot1 da1
gpart add -t freebsd-zfs -l disk1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
zpool replace zroot <old-device-number> da1p2
<wait for resilver>

It booted just fine.

Then I powered off the machine, removed da1, and powered on again. It
booted just fine.

More information about the freebsd-bugs mailing list