Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system after upgrade to 10.1-RELEASE

[Anton Eliasson]

Den 2014-11-19 08:22, Trond Endrestøl skrev:
> On Tue, 18 Nov 2014 23:05+0100, Anton Eliasson wrote:
>
>> Hi!
>> After upgrading a server from FreeBSD 10.0-RELEASE to 10.1-RELEASE it no
>> longer starts up. The bootloader shows its usual menu and then proceeds to
>> load zfs.ko, opensolaris.ko and some third module. Then kernel boots and
>> continues all the way to mountroot where it fails with the message "Mounting
>> from zfs:zroot/ROOT/default failed with error 2: unknown file system." The
>> entire message:
>> https://cloud.solid.lth.se/public.php?service=files&t=27c15a131ca63b19f619a7ee558fe8d1
>>
>> Entering zfs:zroot, zfs:zroot/ROOT, zfs:zroot/ROOT/default or zfs:anythingelse
>> results in the same error.
>>
>> FreeBSD 10.0 was installed this summer. I believe I used the installer's
>> guided procedure for installing on a mirrored ZFS root. Each drive has a GPT
>> table with a 512k freebsd-boot, a freebsd-swap and a freebsd-zfs partition. No
>> separate /boot.
>>
>> There is no vfs.root.mountfrom directive in /boot/loader.conf and no / in
>> /etc/fstab. If I set vfs.root.mountfrom="zfs:zroot" it fails with the same
>> message but a slightly different path (zfs:zroot).
>>
>> The motherboard is a Supermicro X9SRH-7TF. I can import and mount the root
>> filesystem with a FreeBSD live CD without problems.
>>
>> Some config files pulled from a very similar installation (same server model,
>> installed at the same time). They should be more or less identical on the
>> server that won't boot:
>>
>> /etc/rc.conf:
>>
>>      hostname="..."
>>      keymap="swedish.iso.kbd"
>>      # Use synchronous DHCP; pause boot until DHCP is completed.
>>      ifconfig_ix0="SYNCDHCP"
>>      sshd_enable="YES"
>>      ntpd_enable="YES"
>>      # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
>>      dumpdev="AUTO"
>>      zfs_enable="YES"
>>
>>      # IPFW
>>      [...]
>>
>> /boot/loader.conf:
>>
>>      zfs_load="YES"
>>      geom_mirror_load="YES"
>>
>> /etc/fstab:
>>
>>      fdesc    /dev/fd        fdescfs        rw    0    0
>>      /dev/mirror/swap none swap sw 0 0
>>
>> Thanks!
> According to the screenshot, the bootfs property on zroot is set to
> zroot/ROOT/default. Does that dataset exist?
>
> While at the loader prompt, what does these commands tell you?
>
>    lszfs zroot
$MOS
$FREE
$ORIGIN
ROOT
tmp
usr
var
>    lszfs zroot/ROOT
default

And `lszfs zroot/ROOT/default` shows nothing.
> If you see some other potential dataset, try setting currdev using
> this as a template before booting:
>
> set currdev="zfs:zroot/ROOT/some-other-dataset:"
>                  ^                             ^
>                      These :'s are important.
`show currdev` shows that it is already set to zfs:zroot/ROOT/default: 
(with a colon at the end)
>
> Next, unload the old stuff, and try to load the new stuff:
>
> unload
> load /boot/kernel/kernel
> load /boot/kernel/opensolaris.ko
> load /boot/kernel/zfs.ko
It looks like it loads the kernel, but then it fails on the following 
two commands because the kernel is not loaded:
https://cloud.solid.lth.se/public.php?service=files&t=5028527feac7982c76206b691212d15b
https://cloud.solid.lth.se/public.php?service=files&t=0c0d50aac583167a11a8eaa4de5f08ce

Doing boot -s after that gives the same error as before.

However, loading the old kernel with the following commands works better:

unload
load /boot/kernel.old/kernel
load /boot/kernel.old/opensolaris.ko
load /boot/kernel/zfs.ko
lsmod

Screenshots:
https://cloud.solid.lth.se/public.php?service=files&t=9c6c1bbc96b2c816aae05d2718215e2a
https://cloud.solid.lth.se/public.php?service=files&t=8aee8df4bacf65b9ceae02fa65b53711

Issuing boot -s after that shows some text and then suddenly reboots the 
computer. I can see "ZFS filesystem version 5" in the output so it looks 
like it is successfully loaded.

Selecting the old kernel in the boot menu and trying to boot normally 
fails because init is not found: 
https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18

Could this be a mismatch in kernel and kernel module versions? How would 
I fix that?

-- 
Best Regards,
Anton Eliasson