zfs only mounting "root" after reboot

[Thomas Schweikle]

On 16.08.2014 12:19, Thomas Schweikle wrote:
> On 16.08.2014 11:12, Anton Sayetsky wrote:
>> man rc.conf
>> /zfs_enable
> 
> This is allso set -- just forgot to mention it:
> # cat /etc/rc.conf
> [...]
> zfs_enable="YES"
> 

After digging deep into how FreeBSD boots if zfs is used as root I
found the solution to this problem. It is with any special
filesystem like "linproc", "fdesc", or others you might use, as long
as these are mounted to someting *not* in zfs:zroot.

FreeBSD mount only zfs:zroot, then, after doing some work not
requiring any of /tmp, /usr or /var. Then mounts everything given in
fstab with "mount -a" except those marked with option "late". After
having done that all further zfs:zroot mountpoints are processed by
executing "zfs mount -a". Last thing done is mounting everything
marked with "late" in fstab by "mount -al".

Since some of these special filesystems require /tmp, /usr or /var
being mounted this fails and FreeBSD bails out into a single user shell.
The solution is to move mounting these later in the boot process,
after /tmp, /usr or /var are mounted. This is done by adding option
"late" to these file systems, as given below

# cat /etc/fstab
# Device       Mountpoint         FStype    Options   Dump Pass#
/dev/acd0      /cdrom             cd9660    ro,noauto    0 0
#
linproc        /compat/linux/proc linprocfs rw,auto,late 0 0
fdesc          /dev/fd            fdescfs   rw,auto,late 0 0
#
/dev/gpt/swap0 none               swap      sw           0 0

I think this shall be mentioned in the FreeBSD handbook, but
couldn't find any hint to it. Is it a bug with zfs mounting? Is it
intended to have zfs mounts happen before working on fstab mounts?
If not, it could be useful to make more clear what happens and that
you *must* mark filesystems depending on /tmp, /usr or /var to mount
"late"!

-- 
Thomas