Advice needed with ZFS on root filesystem (installing remotely via mfsBSD)

[Torbjorn Kristoffersen]

I am trying to install FreeBSD 8.0-RC1 on amd64, and I plan to use ZFS
(mirrored) as the root filesystem. I have been struggling
with this for many hours -- it just won't boot up and I can't
understand what I'm doing wrong.

Before I describe my steps, keep in mind that I am doing this using
the Remote Install method
of mfsBSD.  My mfsBSD system boots up just fine and I log in via SSH.
 However, seeing that it's a remote
installation I'm unable to see any error message if the system does
not boot up correctly. A sad situation, indeed.

What is wrong with my approach ? Please see below.

--

First I create a GPL label on my two disks
    % gpart create -s gpt ad4
    % gpart create -s gpt ad6

Then I add a boot-partition that will hold the ZFS bootcode
    % gpart add -b 34 -s 128 -t freebsd-boot ad4
    % gpart add -b 34 -s 128 -t freebsd-boot ad6

I then add the freebsd-zfs partition on each disk
    % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad4
    % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad6


The results:
    % gpart show ad4
    =>      34  1465149101  ad4  GPT  (699G)
              34         128    1  freebsd-boot  (64K)
             162  1465148973    2  freebsd-zfs  (699G)

    % gpart show ad6
    =>        34  1465149101  ad6  GPT  (699G)
              34         128    1  freebsd-boot  (64K)
             162  1465148973    2  freebsd-zfs  (699G)

I add GPT bootcode to the MBR and add the ZFS bootcode:
    % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad4
    % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad6

I make sure that the first partition is set to active
    % fdisk -a /dev/ad4
    ...
    % fdisk -a /dev/ad6

I create zpool on ad4p2 and ad6p2 (mirrored)
    % zpool create zroot mirror ad4p2 ad6p2

Create filesystem
    % zfs create -p zroot

Make sure that I enable the bootfs
    % zpool set bootfs=zroot zroot

At this point, /zroot is mounted on my filesystem, so I go ahead and
do a sysinstall
and install FreeBSD.  I make sure that /zroot is set as "root folder"
in sysinstall's Options menu.

I add zfs_load="YES" to /zroot/boot/loader.conf
and   vfs.root.mountfrom="zfs:zroot" to /zroot/boot/loader.conf

And I also copy /boot/zfs/zpool.cache to /zroot/boot/zfs/zpool.cache

Then I do
    % chroot /zroot
    % sysinstall
...  I go into configuration, set the proper network settings, make
sure that sshd will get started. I change
root's password and create a user.  My /etc/rc.conf (on zroot) looks like this:

    ifconfig_re0="inet xxx.xxx.121.11  netmask 255.255.255.192"
    defaultrouter="xxx.xxx.121.1"
    sshd_enable="YES"
    hostname="grim"

At this point I have also tried compiling a kernel, making sure that
GEOM_PART_BSD, GEOM_PART_MBR and other
required options are included.  I also installed a ZFS aware /boot/loader (**)

(Note: I will create a swap partition later once I get the installation booted.)

At this point
    % exit     (from the chroot)
    % zfs unmount -a
    % reboot

At this point the computer never comes back and I have to use my
colo's rescue system to put mfsBSD back on.

One thing I noticed at that point, is that the freebsd-boot on ad4 is
gone and the freebsd-zfs partition has been replaced with a smaller
freebsd-ufs. I did not touch "Partitions" or "Labels" in the
sysinstall installation. What could possibly cause this?    (In case
it matters; ad6 is unchanged.)


Kind Regards,
Torbjorn Kristoffersen

(**) http://wiki.freebsd.org/ZFSOnRootWithZFSboot#installLoader