ZFS + GPT with root on memory stick and mirrored SATA drives

[krad]

On 10 January 2011 04:58, Carl Chave <carl at chave.us> wrote:

> Posting the below for input.  The bulk of this is from a guide that
> Morgan Wesström posted to this list.  Some of it is taken from the
> root on ZFS wiki entries on freebsd.org.  Some from a pjd post here:
>
> http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/
>
> And then there's this that Svein Skogen posted to the list:
>
> I usually (today) set up something similar. I sysinstall FreeBSD onto a
> CF card with the "one-big-root" method, then create a zpool (on
> spinning-metal-storage) where I create the usr, tmp, var fs'es, tar|tar
> the originals over and fix the mountpoint info on the zfs'es. Then I add
> swap on a zvol (since I don't know how to properly use a kernel dump, I
> don't need swap to store it).
>
> I'm setting up a new home server and I always agonize over
> partitioning.  So the steps below install the base system with zfs
> root on a usb stick and /tmp /usr /var and swap on mirrored sata
> drives.
> I've tested these steps and everything works but before I press on
> with actually configuring and using the server, does anybody have any
> input on whether I should or shouldn't do it this way?  ZFS best
> practices suggests that having elements of the root filesystem on
> different pools is a bad idea.  So that might be strike 1.
>
> Memory Stick
> ------------
> /
> /bin
> /boot
> /dev
> /etc
> /lib
> /libexec
> /media
> /mnt
> /proc
> /rescue
> /root
> /sbin
> /sys --> /usr/src/sys
>
> Hard disk zpool
> ---------------
> /tmp
> /usr
> /var
> swap on zvol
>
> Separate zfs datasets
> ---------------------
> /tmp
> /usr
> /usr/home
> /usr/local
> /usr/obj
> /usr/ports
> /usr/ports/distfiles
> /usr/ports/packages
> /usr/src
> /var
> /var/log
> /var/audit
> /var/tmp
>
> Install Procedure (Mostly by Morgan Wesström)
> ---------------------------------------------------------------------
> Select your country and keyboard layout.
>
> Enter the Fixit environment and use the live filesystem on your DVD.
>
> Your usb memory stick will most likely be da0 but you can (and should)
> check it with "camcontrol devlist" before you continue.
>
> Create a new GPT partitioning scheme:
>  # gpart create -s gpt da0
>
> Create a 64KiB partition for the zfs bootcode starting at LBA 1920:
>  # gpart add -b 1920 -s 128 -t freebsd-boot da0
>
> Create a zfs partition spanning the remainder of the usb memory stick
> and give it a label we can refer to:
>  # gpart add -t freebsd-zfs -l FreeBSDonUSB da0
>
> (The starting LBA for the first partition is there to align the
> partitions to the flash memory's erase block size. This is
> particularly important for the main zfs partition. The main partition
> above will start at exactly 1MiB (LBA 2048) which will align it to any
> erase block size used today. This alignment is also of great
> importance if you use this guide to install FreeBSD to one of the
> newer harddrives using 4096 byte sectors.)
>
> Install the protective MBR to LBA 0 and the zfs bootcode to the first
> partition:
>  # gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 da0
>
> Create /boot/zfs (for zpool.cache) and load the zfs kernel modules:
>  # mkdir /boot/zfs
>  # kldload /dist/boot/kernel/opensolaris.ko
>  # kldload /dist/boot/kernel/zfs.ko
>
> Create a zfs pool and set its bootfs property:
>  # zpool create zrootusb /dev/gpt/FreeBSDonUSB
>  # zpool set bootfs=zrootusb zrootusb
>
> Switch to fletcher4 checksums and turn off access time modifications:
>  # zfs set checksum=fletcher4 zrootusb
>  # zfs set atime=off zrootusb
>
> Create zfs mirrored data pool on SATA disks
>  # zpool create zdata mirror /dev/ad4 /dev/ad6
>  # zfs set canmount=off zdata
>  # zfs set mountpoint=/zrootusb zdata
>  # zfs set checksum=fletcher4 zdata
>  # zfs create zdata/tmp
>  # zfs create zdata/usr
>  # zfs create zdata/usr/home
>  # zfs create zdata/usr/local
>  # zfs create zdata/usr/obj
>  # zfs create zdata/usr/ports
>  # zfs create zdata/usr/ports/distfiles
>  # zfs create zdata/usr/ports/packages
>  # zfs create zdata/usr/src
>  # zfs create zdata/var
>  # zfs create zdata/var/log
>  # zfs create zdata/var/audit
>  # zfs create zdata/var/tmp
>
> Create swap zvol on zdata pool
>  # zfs create -V 5G zdata/swap
>  # zfs set org.freebsd:swap=on zdata/swap
>  # zfs set checksum=off zdata/swap
>
> Extract at a minimum, base and the generic kernel:
>  # cd /dist/8.1-RELEASE/base
>  # DESTDIR=/zrootusb ./install.sh
>  # cd ../kernels
>  # DESTDIR=/zrootusb ./install.sh generic
>
> Delete the empty, default kernel directory and move the generic kernel
> into its place:
>  # rmdir /zrootusb/boot/kernel
>  # mv /zrootusb/boot/GENERIC /zrootusb/boot/kernel
>
> Make sure the zfs modules are loaded at boot:
>  # cat > /zrootusb/boot/loader.conf
>   zfs_load="YES"
>   vfs.root.mountfrom="zfs:zrootusb"
>   kern.cam.boot_delay=10000
>  ^d
>
> Create /etc/rc.conf. Adjust and add to your own needs:
>  # cat > /zrootusb/etc/rc.conf
>   hostname="sodserve"
>   sshd_enable="YES"
>   zfs_enable="YES"
>   ^d
>
> Setup your time zone:
>  # cp /zrootusb/usr/share/zoneinfo/EST5EDT /zrootusb/etc/localtime
>
> Create an empty fstab to avoid startup warnings:
>  # touch /zrootusb/etc/fstab
>
> Set the root password in the new environment:
>  # cd /
>  # chroot /zrootusb /bin/sh
>  # passwd root
>  # exit
>
> Copy zpool.cache:
>  # cp /boot/zfs/zpool.cache /zrootusb/boot/zfs
>
> Unmount filesystems and set mountpoints:
>  # zfs unmount -a
>  # zfs set mountpoint=legacy zrootusb
>  # zfs set mountpoint=/ zdata
>
>
> Exit SYSINSTALL and reboot. You now have a fully functional and
> bootable FreeBSD installation on your usb memory stick.
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe at freebsd.org"
>


on thing you may consider doing is mirroring the way opensolaris does its
boot environments. This makes it easy to flip flop back between os versions
for testing or if things go wrong.

eg from my home system

# zfs list | grep be
system/be                             4.09G   123G  29.5K  /system/be
system/be/root20101123                2.63G   123G  1.76G  legacy
system/be/root20101218                1.47G   123G   992M  legacy

# grep vfs /boot/loader.conf; zpool get bootfs system
vfs.root.mountfrom="zfs:system/be/root20101218"
vfs.root.mountfrom.options=rw
NAME    PROPERTY  VALUE                   SOURCE
system  bootfs    system/be/root20101218  local


After I have done a build world and  kernel I run this script. Sending and
receiving isnt 100% necessary, but I find it cleaner than snapping and
cloning as it doesnt end up in dependencies. Although it does eat up more
disk space (assuming no dedup) but then we aren't talking huge amounts of
disk space

# cat /usr/local/scripts/install_world
#!/usr/local/bin/bash

if [ $UID != 0 ] ; then
  echo your not root !! ; exit 1
fi

date=`date '+%Y%m%d'`
oroot=`grep "vfs.root.mountfrom=\"zfs:system/" /boot/loader.conf | sed -e
"s#^.*\"zfs:system/be/##" -e "s#\"##"`
nroot="root$date"
snap="autoup-$RANDOM"
zpool=system

export DESTDIR=/$zpool/be/$nroot


if [ "$oroot" =  "$nroot" ] ; then
 echo "i cant update twice in one day"; exit 1
fi

echo building in $zpool/be/$nroot

zfs snapshot $zpool/be/$oroot@$snap && \
zfs send $zpool/be/$oroot@$snap | zfs receive -vv $zpool/be/$nroot    && \
cd /usr/src && \
make installkernel && \
make installworld && \
sed -i -e "s#$zpool/be/$oroot#$zpool/be/$nroot#"
/$zpool/be/$nroot/boot/loader.conf && \
echo -en "\n\nNow run these two commands to make the changes live, and
reboot
 zfs set mountpoint=legacy $zpool/be/$nroot
 zpool set bootfs=$zpool/be/$nroot $zpool\n\n"