Broken ZFS boot on upgrade

[Jonathan Anderson]

Hello friendly FreeBSD people,

I’ve gone and done it: I upgraded a key ZFS-on-root machine from 11.2 to
12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on
this list might be able to help me figure out what's wrong and what I can
do about it?

It looks like the ZFS code in the bootloader can't find anything in my root
directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If
I boot a rescue image from USB I can mount everything in the pool (`zpool
import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs
zroot/ROOT/default` from the loader prompt it gives me an empty result (so,
e.g., no /boot). Booting fails with messages such as, "i/o error - all block
copies unavailable".

My pool consists of three mirrored vdevs, in which the first mirror uses GPT
 partitioning (for the boot partitions) and the other two mirrors use
whole disks.
I recall reading somewhere that the bootloader ZFS code doesn't like
non-partition-based
vdevs... is that true? If so, perhaps the issue is that my upgrade caused
/boot to live on one of the newer whole-disk-based mirrors, hiding it from
the bootloader's view?

I've backed up all of my data (a 28h process) and am resigned to
re-installing if need be, but I would like to understand where I went wrong
to avoid making the same mistakes again. What would you recommend for
someone building a new array-of-mirrors pool? I see that the Handbook says
that certain things *can* be done, but it’s not clear to me whether they
should or even must be done, e.g.:

> partitions with bootcode and file systems needed for booting can be
added. This allows booting from disks that are also members of a pool.
There is no performance penalty on FreeBSD when using a partition rather
than a whole disk

The Handbook suggests that it's possible to break disks into multiple
partitions that are added separately to a vdev, but is this a sensible
thing to do? Is there any other ZFS lore that hasn't made it to the
Handbook but that ought to be kept in mind from the outset?

Thanks,


Jon--
Assistant Professor
Electrical and Computer Engineering
Memorial University of Newfoundland

https://www.engr.mun.ca/~anderson