Upgrade of RELENG_8 ZFS boot pool leads to unbootable system

[Paul Mather]

On Jan 2, 2013, at 2:10 PM, Matthew Seaman <m.seaman at infracaninophile.co.uk> wrote:

> On 02/01/2013 17:49, Paul Mather wrote:
>> Yesterday, I updated my RELENG_8 ZFS-only system that has worked like a champ for ages.  After a successful install{kernel,world} and reboot, I noticed the 20121130 entry in /usr/src/UPDATING and upgraded my ZFS pool via "zfs upgrade -a".  I also upgraded my boot blocks as requested, and as per the "ZFS notes" section of /usr/src/UPDATING.
>> 
>> Unfortunately rebooting with the upgraded pool failed.  The "windmill" boot spinner spins for a tiny amount of time and then stops dead. :-(  I don't get to the boot loader menu at all.
>> 
>> I downloaded a very recent RELENG_8 snapshot (FreeBSD-8.3-RELENG_8-r244923-JPSNAP-amd64-amd64-memstick.img) from pub.allbsd.org and was able to boot successfully from USB using that.  I entered Fixit Mode and tried to write the boot blocks on the memstick image onto my hard drives but the resultant system still wouldn't boot.  The commands I used (from Fixit Mode) are these:
>> 
>> 	gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad4
>> 	gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad6
>> 
>> (ad4 and ad6 are my two hard drives.)
>> 
>> If I "load zfs" before booting the USB memstick then I can see my old pool listed when I do "zfs import".  I haven't tried importing the pool because I'm not sure if that would make the problem worse.
>> 
>> Does anyone have any advice in restoring this system to bootability?  I followed the standard "root on ZFS" recipe using a two drive mirror when installing the system initially.  Each drive uses GPT with three partitions: freebsd-boot, freebsd-swap, and freebsd-zfs in that order.  Like I said at the start, all this worked for a long time until just now when I upgraded the pool to enable "feature flags" support. :-(
>> 
>> Any help is appreciated.
> 
> I think you may be running into problems with zpool.cache.  This has
> been fixed in current, which now has the ability to find the root zpool
> without a valid zpool.cache, but that I suspect is faint comfort for you.
> 
> To recover from a toasted zpool.cache, you need to boot from alternate
> media and then import your root zpool.  It's easiest to do that to a
> temporary directory.  The important bit is to copy the zpool.cache onto
> your actual zroot device:
> 
> -- Boot from install media to 'Live CD' and log in as root (no password)

Given the above, does this need to be a -CURRENT Live CD?  I've been using the RELENG_8 snapshot memstick.img mentioned in my original message above.


> 
> # kldload zfs               -- should load opensolaris.ko automatically
> # cd /tmp                   -- this should be a writable MFS; you'll
>                               need to arrange something similar if
>                               not.
> # zpool import -o cachefile=/tmp/zpool.cache -R /tmp/zroot zroot
>                            -- this should create a zpool.cache file


I tried this and it complained about the pool being in use by another system---the original system that won't boot any more.  I expected this, and added "-f" to force an import.

> # cp zpool.cache /tmp/zroot/boot/zfs/


This part also failed for me.  My "zroot" fileset has a mountpoint property set to "legacy".  I had to mount this manually, via "mount -t zfs zroot /tmp/zroot" to make the /tmp/zroot/boot/zfs directory accessible.


> # zfs umount -a
> # shutdown -r
> 
> Eject the install media, and the system should boot up from your root spool.


Unfortunately, it didn't boot from the root pool.  I get the same thing happening: the "windmill" spins for a very short time and then stops dead.  I don't even make it to the "BTX Loader" output, let alone the boot loader menu options. :-(

Thank you for the suggestions.

Cheers,

Paul.