ZFS snapshot restore not quite working; missing steps?

[Jeff Chan]

We're trying to do a ZFS snapshot restore of a ZFS non-RAIDZ* FreeBSD
9.2-RELEASE to a new bare system configured with ZFS RAIDZ, and it's not
quite working right.  The restore itself seems to complete, but we're
not able to successfully boot the resulting system.  We can't recall
the exact error we got, but think the ZFS cache file was not found.

Presumably we're missing some steps, don't have all magic for ZFS
booting from a FreeBSD root partition installed/configured correctly,
etc.  What did we miss?

(We have a private network to serve the snapshot over NFS from a third
server on 192.168.0.2.)


(There may be some line wrap below.)

(Comments in (parens))

(Some outputs in [brackets])




On the old system, call it foo:


foo: [103]% zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                169G  3.40T   545M  legacy
zroot/home          15.1G  3.40T  10.3G  /home
zroot/home/username 4.04G  3.40T  2.74G  /home/username
zroot/bar            128G  3.40T    97K  /bar
zroot/bar/prod       126G  3.40T  73.8G  /bar/prod
zroot/bar/test      2.18G  3.40T  2.18G  /bar/test
zroot/tmp           1.35G  3.40T  1.35G  /tmp
zroot/usr           9.60G  3.40T  9.52G  /usr
zroot/var           14.0G  3.40T  14.0G  /var
foo: [104]% zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  3.62T   170G  3.46T     4%  1.00x  ONLINE  -



On the new system:


Boot from a FreeBSD 9.2-RELEASE installer USB flash drive
into LiveCD, then:

(bring up NFS private LAN)
ifconfig ix3 192.168.0.3 netmask 255.255.255.0 up
[ix2: Could not setup receive structures
[ix2: Could not setup receive structures
->
sysctl kern.ipc.nmbclusters=131072
sysctl kern.ipc.nmbjumbo9=38400

(mount remote NFS backup directory)
mount -t nfs 192.168.0.2:/home/backup /mnt
mkdir /var/mnt

sysctl kern.disks
[kern.disks: da0 mfid2 mfid1 mfid0]

gpart create -s gpt mfid0
gpart create -s gpt mfid1
gpart create -s gpt mfid2
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 mfid0
gpart add -s 222 -a 4k -t freebsd-boot -l boot1 mfid1
gpart add -s 222 -a 4k -t freebsd-boot -l boot2 mfid2
gpart add -s 8g -a 4k -t freebsd-swap -l swap0 mfid0
gpart add -s 8g -a 4k -t freebsd-swap -l swap1 mfid1
gpart add -s 8g -a 4k -t freebsd-swap -l swap2 mfid2
gpart add -a 4k -t freebsd-zfs -l disk0 mfid0
gpart add -a 4k -t freebsd-zfs -l disk1 mfid1
gpart add -a 4k -t freebsd-zfs -l disk2 mfid2
(Clear any old zfs data)
dd if=/dev/zero of=/dev/mfid0p3 count=560 bs=512
dd if=/dev/zero of=/dev/mfid1p3 count=560 bs=512
dd if=/dev/zero of=/dev/mfid2p3 count=560 bs=512
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid2
zpool create -f -m none -o altroot=/var/mnt -o cachefile=/tmp/zpool.cache zroot raidz mfid0 mfid1 mfid2

gunzip -c /mnt/foo-backup-full/zroot at 20140516183425.gz | zfs receive -vdFu zroot
zpool set bootfs=zroot zroot
zpool get all zroot
zfs get all zroot

zfs set mountpoint=/var/mnt zroot
(add swap to fstab)
vi /var/mnt/var/mnt/etc/fstab
[/dev/mfid0p2    none    swap    sw      0       0
[/dev/mfid1p2    none    swap    sw      0       0
[/dev/mfid2p2    none    swap    sw      0       0

vi /var/mnt/var/mnt/etc/rc.conf    [ change adapter name & IP ]
vi /var/mnt/var/mnt/etc/pf.conf    [ change adapter name ]
[vi /var/mnt/var/mnt/boot/loader.conf]   (not performed)
cd /
zpool export zroot
zpool import -o altroot=/var/mnt -o cachefile=/tmp/zpool.cache zroot
[zfs set mountpoint=/ zroot]    (not performed)
cp /tmp/zpool.cache /var/mnt/boot/zfs
zpool get all zroot
zpool set cachefile=/boot/zfs/zpool.cache   zroot
zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/home zroot/home
zfs set mountpoint=/bar zroot/bar
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

[remove USB]
reboot



Any hints at what we missed/goofed?

Cheers,

Jeff C.
--
Jeff Chan
mailto:jeffc at supranet.net