zfs_enable vs zfs_load in loader.conf (but neither works)

[Andriy Gapon]

Now some high level information on how ZFS boot works and a little bit more
detailed information on how a root filesystem is chosen in ZFS case.  The
information is applicable to recent versions of FreeBSD in head, stable/9
(including upcoming 9.2) and stable/8 (including 8.4).

- boot0-like stage always takes boot2-like stage from the same disk using simple
rules
- boot2-like stage probes all disks and partitions it can understand for ZFS pools
- default pool is the first pool detected by probing which starts at boot disk
- default filesystem is determined by bootfs property
- boot2-like stage allows to select a different pool, a specific filesystem in
the pool and a specific loader

boot0-like stage is pmbr in the case of GPT partitioning.
boot0-like stage is the first block of zfsboot in the case of whole-disk ZFS.
boot2-like stage is either gptzfsboot or zfsboot correspondingly.

- loader uses boot pool and filesystem information passed by boot2-like stage
- loader exposes loaddev and currdev variables, initially they point to the pool
and filesystem obtained from boot2-like stage
- currdev can be changed (e.g. at the prompt) while loaddev is read only
- kernel and modules are loaded from currdev by default

- kernel mounts root from a filesystem specified by vfs.root.mountfrom variable
that is passed by loader to kenv
- value of the variable is determined as follows:
- loader tries to set this variable based on "/" entry, if any, in /etc/fstab,
if any, in the filesystem specified by currdev
- the variable can be explicitly set in loader.conf or at the prompt; the
explicit assignment overrides the fstab-based auto-detected value
- for ZFS, if the above methods do not produce any value, vfs.root.mountfrom is
set based on currdev

So, you can see that all three methods mentioned in this thread can work equally
well.  You can either specify a root entry in fstab, or set vfs.root.mountfrom
in loader.conf, or simply set bootfs property.  The above information also
describes precedence rules if more than one knob is used: vfs.root.mountfrom is
the most significant, fstab is after it, bootfs plays role in root filesystem
selection only if neither of the previous is set.
Thus, it's completely up to you which method to use.  Whichever is more
convenient.  I prefer to just set bootfs.

Another piece of information is that neither mountpoint nor canmount property
affects ZFS root mounting.  They of course have their usual effect in other
contexts like importing a pool on a different system or when a different
filesystem is selected to be a root filesystem.
So, again, you can set these properties to whatever is most convenient for you.

-- 
Andriy Gapon