Mount order for ZFS, jails, and nullfs

[Robert Bonomi]

> From owner-freebsd-questions at freebsd.org  Fri Sep 24 11:02:59 2010
> To: freebsd-questions at freebsd.org
> From: Torbjorn Granlund <tg at gmplib.org>
> Date: Fri, 24 Sep 2010 18:03:06 +0200
> Subject: Mount order for ZFS, jails, and nullfs
>
> In jails, I'd like a local ZFS /, a read-only nullfs-mounted /usr, and a
> local /usr/local.  (I'd also have read-only nullfs-mounted /bin, /lib,
> /libexec, but let's forget about that for now.)
>
> This way, I can upgrade the master /usr once, in one place, and have all
> jails inherit it.  And my dear jail inmates can install anything in
> /usr/local (such as their favourite packages/ports).
>
> I.e., things should look like this:
>
> /myjail/                zfs
> /myjail/usr             nullfs ro
> /myjail/usr/local       zfs
>
> There is no problem to make this happen by issuing a handful of commands
> manually after boot, but I cannot seem to get it to work automatically,
> with existing boot mechanism.  The problem is that the mount of /usr
> will be attempted before ZFS has mounted /myjail, the jail's root.
>
> ZFS maintains its own mount table.  It is possible to disable the
> automated mounting in ZFS by specifying the pseudo mountpoint "legacy",
> and then--according to the FreeBSD manual--mount it with mount(8).
> Unfortunately, FreeBSD's mount seems unable to perform mounts of ZFS
> volumes.
>
> How can I achieve the result I want after a reboot?

if the traditional way doesn't work for you, the solution is "don't use
the traditional way'.  <grin?

Seriously.

For the stuff that has to  mount _after_ zfs is running (i.e. '/myjail/usr'),
mark them as 'noauto' in the fstab.  

Do the 'equivalent' for any zfs fs that mounts _below_ those points (i.e.,
'/myjail/user/local').

That way the normal stages in the boot process will _not- mount them 
prematurely.

THEN, add a boot script that "requires zfs' and wherever the actual zfs
mounts are done. In -that- script, insert 'manual' commands, _in_the_order_
you need things to happen in.

It is advisable to drop notes in /etc/fstab and the zfs equivalent, that
these filesystems are automatically mounted by the {mumble} script.  Such
notes save _lots_ of hair-tearing in the future, when you have to trouble-
shoot something you "don't remember" the details of doing.