ZFS + GPT with root on memory stick and mirrored SATA drives

Carl Chave carl at chave.us
Mon Jan 10 05:22:14 UTC 2011

Posting the below for input.  The bulk of this is from a guide that
Morgan Wesström posted to this list.  Some of it is taken from the
root on ZFS wiki entries on freebsd.org.  Some from a pjd post here:

And then there's this that Svein Skogen posted to the list:

I usually (today) set up something similar. I sysinstall FreeBSD onto a
CF card with the "one-big-root" method, then create a zpool (on
spinning-metal-storage) where I create the usr, tmp, var fs'es, tar|tar
the originals over and fix the mountpoint info on the zfs'es. Then I add
swap on a zvol (since I don't know how to properly use a kernel dump, I
don't need swap to store it).

I'm setting up a new home server and I always agonize over
partitioning.  So the steps below install the base system with zfs
root on a usb stick and /tmp /usr /var and swap on mirrored sata
I've tested these steps and everything works but before I press on
with actually configuring and using the server, does anybody have any
input on whether I should or shouldn't do it this way?  ZFS best
practices suggests that having elements of the root filesystem on
different pools is a bad idea.  So that might be strike 1.

Memory Stick
/sys --> /usr/src/sys

Hard disk zpool
swap on zvol

Separate zfs datasets

Install Procedure (Mostly by Morgan Wesström)
Select your country and keyboard layout.

Enter the Fixit environment and use the live filesystem on your DVD.

Your usb memory stick will most likely be da0 but you can (and should)
check it with "camcontrol devlist" before you continue.

Create a new GPT partitioning scheme:
 # gpart create -s gpt da0

Create a 64KiB partition for the zfs bootcode starting at LBA 1920:
 # gpart add -b 1920 -s 128 -t freebsd-boot da0

Create a zfs partition spanning the remainder of the usb memory stick
and give it a label we can refer to:
 # gpart add -t freebsd-zfs -l FreeBSDonUSB da0

(The starting LBA for the first partition is there to align the
partitions to the flash memory's erase block size. This is
particularly important for the main zfs partition. The main partition
above will start at exactly 1MiB (LBA 2048) which will align it to any
erase block size used today. This alignment is also of great
importance if you use this guide to install FreeBSD to one of the
newer harddrives using 4096 byte sectors.)

Install the protective MBR to LBA 0 and the zfs bootcode to the first partition:
 # gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 da0

Create /boot/zfs (for zpool.cache) and load the zfs kernel modules:
 # mkdir /boot/zfs
 # kldload /dist/boot/kernel/opensolaris.ko
 # kldload /dist/boot/kernel/zfs.ko

Create a zfs pool and set its bootfs property:
 # zpool create zrootusb /dev/gpt/FreeBSDonUSB
 # zpool set bootfs=zrootusb zrootusb

Switch to fletcher4 checksums and turn off access time modifications:
 # zfs set checksum=fletcher4 zrootusb
 # zfs set atime=off zrootusb

Create zfs mirrored data pool on SATA disks
 # zpool create zdata mirror /dev/ad4 /dev/ad6
 # zfs set canmount=off zdata
 # zfs set mountpoint=/zrootusb zdata
 # zfs set checksum=fletcher4 zdata
 # zfs create zdata/tmp
 # zfs create zdata/usr
 # zfs create zdata/usr/home
 # zfs create zdata/usr/local
 # zfs create zdata/usr/obj
 # zfs create zdata/usr/ports
 # zfs create zdata/usr/ports/distfiles
 # zfs create zdata/usr/ports/packages
 # zfs create zdata/usr/src
 # zfs create zdata/var
 # zfs create zdata/var/log
 # zfs create zdata/var/audit
 # zfs create zdata/var/tmp

Create swap zvol on zdata pool
 # zfs create -V 5G zdata/swap
 # zfs set org.freebsd:swap=on zdata/swap
 # zfs set checksum=off zdata/swap

Extract at a minimum, base and the generic kernel:
 # cd /dist/8.1-RELEASE/base
 # DESTDIR=/zrootusb ./install.sh
 # cd ../kernels
 # DESTDIR=/zrootusb ./install.sh generic

Delete the empty, default kernel directory and move the generic kernel
into its place:
 # rmdir /zrootusb/boot/kernel
 # mv /zrootusb/boot/GENERIC /zrootusb/boot/kernel

Make sure the zfs modules are loaded at boot:
 # cat > /zrootusb/boot/loader.conf

Create /etc/rc.conf. Adjust and add to your own needs:
 # cat > /zrootusb/etc/rc.conf

Setup your time zone:
 # cp /zrootusb/usr/share/zoneinfo/EST5EDT /zrootusb/etc/localtime

Create an empty fstab to avoid startup warnings:
 # touch /zrootusb/etc/fstab

Set the root password in the new environment:
 # cd /
 # chroot /zrootusb /bin/sh
 # passwd root
 # exit

Copy zpool.cache:
 # cp /boot/zfs/zpool.cache /zrootusb/boot/zfs

Unmount filesystems and set mountpoints:
 # zfs unmount -a
 # zfs set mountpoint=legacy zrootusb
 # zfs set mountpoint=/ zdata

Exit SYSINSTALL and reboot. You now have a fully functional and
bootable FreeBSD installation on your usb memory stick.

More information about the freebsd-questions mailing list