"can't load 'kernel'" on ZFS root

[Peter Jeremy]

I have a several systems with ZFS on mirrored root and have run into
problems during recent 8-stable upgrades with loader reporting it
was unable to load the kernel.

On the first system, I did an installkernel, rebooted and all came
up OK.  I then did an installworld, rebooted and got (re-typed):
 Loading /boot/defaults/loader.conf
 Unable to load a kernel!
 |
 can't load 'kernel'
and the loader prompt.

After some research, I found bin/144214 which suggested my bootblocks
were out of date.  I booted from another system and installed new
bootblocks on both disks, rebooted and got the same result.

I then tried backing up and restoring all the files in /boot that were
touched by installworld.  This time when I tried to boot, I was
greeted with "ZFS i/o error - all block copies unavailable" followed
by the bootblock prompt.  I tried loading zroot:/boot/zfsloader.old
without success.

At this point, I gave up and rolled back to a snapshot prior to the
installworld (but after the installkernel) and everything worked.

I have since verified that the bootblocks match the newly built
pmbr and gptzfsboot and that the changes in r208892 are present
in the sourcetree.

On the second system, I was unable to boot after the installkernel
and rolled back - I haven't checked how old the bootblocks are there.

Firstly, does anyone have any ideas on how to move forward?  I will
rebuild both zroots at some point but that's not a fix.  (I am
reasonably confident I can reproduce the problem if someone has
a patch to test).

Secondly, I found that ZFS booting is extremely fragile - if your
default kernel & modules won't load automatically, there's no way to
boot the backup kernel because (as far as I can find) there's no way
to manually load /boot/zfs/zpool.cache - without which the kernel
can't mount the root FS.  In both above cases, recovering the system
required booting from recovery media.  There needs to be a documented
method for booting from a snapshot or clone.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20110907/6450f938/attachment.pgp