Problem with ZFS userboot changes r262331 and VM with mixed partitions

[Kurt Lidl]

> Hi,
>
> I encountered problems with the new changes to userboot
> to boot ZFS.
>
> About a month ago, I created a FreeBSD 10 VM with the following
> layout:
>
> =>       34  419430333  md0  GPT  (200G)
>          34        128    1  freebsd-boot  (64K)
>         162    2097152    2  freebsd-ufs  (1.0G)
>     2097314    8388608    3  freebsd-swap  (4.0G)
>    10485922  408944444    4  freebsd-zfs  (195G)
>   419430366          1       - free -  (512B)
>
>
> The FreeBSD root file system is on p4, which is ZFS.
> On the root file system, I have a symlink:
>
> /boot -> /bootdir
>
> /bootdir mounts p2 which is UFS.
>
> So in this scenario, the kernel lives on UFS.
> When I did this about a month ago, during the BHyve boot process,
>
> (1)  userboot loaded the kernel from p2 (UFS)
> (2)  The kernel booted, loaded zfs.ko, and then proceeded to mount p4 where
>       the root file system lived (ZFS)
>
> After r262331, when I try to boot the BHyve VM
>
> (1)  I can see p4 from the loader prompt
> (2)  The kernel doesn't load
> (3)  If I try to load /boot/kernel/kernel from the loader prompt, I get:
>
> ZFS: i/o error - all block copies unavailable
>
> I don't fully understand all this logic.  Is there a bug that can be fixed here?

I ran into this same problem a couple of weeks ago when I first played
with the ZFS boot support in userboot.so.

The following set of operations on the hypervisor can be used to add
bootable support to your zfs installation.

(I don't know what the name of your zpool is on the emulated host.
If the name of that zpool is the same as the zpool on your hypervisor
host, you might have to go at this a different way.)  I have taken
to naming the zpool for the any clients to be the same as the name
of the virtual machine, so I can easily mount it on the host running
the hypervisor and fiddle with it.

For the purposes of this example, my virtual machine is called "vm0".
The zpool for that host lives in a zpool called "zdata" on my
hypervisor machine.  In my case, my /boot was a symlink that
pointed to /bootdir/boot, so you might need to adjust the
following slightly.

zfs import -f -R /mnt vm0

fsck_ufs /dev/zvol/zdata/vm0p2
mount /dev/zvol/zdata/vm0p2 /mnt/bootdir
cd /mnt
chflags -h nosunlink boot
rm boot
cp -R bootdir/* .
umount /mnt/bootdir
cd /
zpool export vm0

Good luck.

-Kurt