Broken ZFS boot on upgrade

[Peter Jeremy]

On 2019-Nov-10 22:16:26 -0330, Jonathan Anderson <jonathan.anderson at mun.ca> wrote:
>I’ve gone and done it: I upgraded a key ZFS-on-root machine from 11.2 to
>12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on
>this list might be able to help me figure out what's wrong and what I can
>do about it?

Based on your symptoms, it sounds like you might have a corrupt zpool.cache.
/boot/zfs/zpool.cache should be rewritten on every boot but I had one system
where that wasn't occurring and a FreeBSD upgrade (I don't currently recall
the actual versions) resulted in more thorough validation checks, which
failed.

Can you share your actual layout ("gpart show", "zpool status", details of
the non-partitioned disks, etc) - that might help us identify a problem.

>It looks like the ZFS code in the bootloader can't find anything in my root
>directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If
>I boot a rescue image from USB I can mount everything in the pool (`zpool
>import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs
>zroot/ROOT/default` from the loader prompt it gives me an empty result (so,
>e.g., no /boot). Booting fails with messages such as, "i/o error - all block
>copies unavailable".

If you boot from a rescue image and either delete (or rename) your
existing zpool.cache or run
# zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot
(where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does
that help?

>My pool consists of three mirrored vdevs, in which the first mirror uses GPT
> partitioning (for the boot partitions) and the other two mirrors use
>whole disks.

Whole disks are not recommended for anything other than building partitions
in.  Ideally, you want all the disks to have GPT (or similar) partitions.
If you don't need anything else on the disk, just create a single partition
occupying the entire disk[1].  (I'd also recommend having a small root zpool
that is a single, preferably mirrored, vdev, rather than a large root spread
over multiple vdevs).

>I recall reading somewhere that the bootloader ZFS code doesn't like
>non-partition-based
>vdevs... is that true? If so, perhaps the issue is that my upgrade caused
>/boot to live on one of the newer whole-disk-based mirrors, hiding it from
>the bootloader's view?

That might be possible though it's not clear why it wouldn't have caused a
problem in the past.  Note that the bootloader is performing I/O via the
BIOS so if the BIOS can't see the disks, the bootloader won't be able to
read them.

>> partitions with bootcode and file systems needed for booting can be
>added. This allows booting from disks that are also members of a pool.
>There is no performance penalty on FreeBSD when using a partition rather
>than a whole disk

Actually, Solaris might insist that you add a whole disk but, under
the hood, ZFS actually partitions the disk with a single partition
occupying the entire disk.

>The Handbook suggests that it's possible to break disks into multiple
>partitions that are added separately to a vdev, but is this a sensible
>thing to do? Is there any other ZFS lore that hasn't made it to the
>Handbook but that ought to be kept in mind from the outset?

You can have multiple partitions on a disk and put different partitions
into different vdevs but there's no point in having different partitions
on the same disk in the same vdev - that will reduce both performance
and resilience.

[1] Though leaving a free GB or so can help if you need to replace a disk and
the replacement is slightly smaller.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20191111/655ec965/attachment.sig>