misc/160706: zfs bootloader fails when a non-root vdev exists on a slice before the root slice

[Peter Maloney]

>Number:         160706
>Category:       misc
>Synopsis:       zfs bootloader fails when a non-root vdev exists on a slice before the root slice
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Sep 13 14:00:20 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Peter Maloney
>Release:        
>Organization:
Brockmann Consult
>Environment:
FreeBSD  8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
zfs bootloader fails when a non-root vdev exists on a slice before the root slice

>How-To-Repeat:
gpart create -s gpt da0

gpart add -b 34 -s 64k -t freebsd-boot da0
gpart add -s 512M -t freebsd-swap -l swap0 da0
gpart add -s 4G -t freebsd-zfs -l log0 da0
gpart add -s 170G -t freebsd-zfs -l cache0 da0
gpart add -t freebsd-zfs -l root0 da0

gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 da0

zpool create zroot mirror gpt/root0 gpt/root1

glabel label da1 tank1d1
..
glabel label da<x> tank<y>d<z>

zpool create tank \
    raidz2 label/tank1d1 label/tank1d2 ... \ 
    raidz2 ...

At this point the system works fine. And survives rebooting.

zpool add tank cache gpt/cache0 gpt/cache1
zpool add tank log mirror gpt/log0 gpt/log1

Now if you boot the system, you get an screen looking like:
=========================
-
FreeBSD/x86 boot
Default: tank:/boot/kernel/kernel
boot:
|
FreeBSD/x86 boot
Default: tank:/boot/kernel/kernel
boot: _








=========================

I have screenshots (which I plan to attach to this report if such a button appears after submitting it). And a note about the screenshots, the shell looks broken as a side effect of editing the installer iso file. I only added scripts and the mps driver to it. And the same problem happens without my changes in FreeBSD-8.2-RELEASE and FreeBSD-8.2-STABLE-201105.


FYI: 
I used mirrors, and installed FreeBSD using this guide, and added things such as the cache and log slices.

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror

>Fix:
Fix unknown... 

Probably a source code change is needed somewhere in sys/boot/zfs.
Possibilities could be:
- retry the same loop/scan again skipping failed zfs pools
- skip zfs vdevs/pools that have no /boot directory
- skip log and cache vdevs (which I guess would be only a workaround and specific to my case)

Workaround: Make sure your bootable system is the first zfs slice on your boot disk. 

eg.

    gpart add -b 34 -s 64k -t freebsd-boot da0
    gpart add -s 512M -t freebsd-swap -l swap0 da0
    gpart add -s 80G -t freebsd-zfs -l root0 da0
    gpart add -s 4G -t freebsd-zfs -l log0 da0
    gpart add -t freebsd-zfs -l cache0 da0


(where root0 is the zfs root slice)

>Release-Note:
>Audit-Trail:
>Unformatted: