Issues with multiple-vdev ZFS root

[Dmitry Marakasov]

Hi!

I've recently moved to ZFS root installation with 2-way mirror with
an intent to add another pair of disks later, however when I've
actually tried to add second vdev, I've got "root pool can not have
multiple vdevs or separate logs" error. It's possible to fool this
error by removing bootfs property from the pool, adding second vdev and
the re-adding the property. I've decided to first experiment in
virtualbox.

It turned out that after adding second mirror this way, system was
still bootable. However, after copying /boot over (mv /boot /boot.old &&
cp -RPp /boot.old /boot) the system won't boot with

ZFS: i/o error - all block copies unavailable
Invalid format

error (see [1]).
It seems like the files have landed onto second vdev and gptzfsboot is
no longer able to get them.

Still, I've managed to solve that by increasing number of copies for the
pool to 3 and copying /boot over again, which likely made the files
available from first vdev again.

Later I was hinted that the cause of this is that virtualbox BIOS
only reports first disk, so the whole second vdev is unavailable for
gptzfsboot.

My question is whether these conclusions are correct:

- FreeBSD supports any configuration for root zfs pool, including
  one with multiple vdevs.
- However, one should ensure that BIOS reports all disks or increase
  number of copies for bootable dataset (and this should probably be
  documented somewhere)
- The 'root pool can not have multiple vdevs' check should be removed

[1] http://amdmi3.ru/files/zfs-boot-fail.png

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3 at amdmi3.ru  ..:  jabber: amdmi3 at jabber.ru    http://www.amdmi3.ru