using zfs and unionfs together, does zfs need to be extended?

[George Hartzell]

I'd like to be able to set up a large-ish number of very similar
jails, with a minimum of fuss and take advantage of zfs' cool
features.  I'd like to use unionfs to do this, but zfs' lack of
whiteout support seems to make it impossible.  [jump to the bottom if
you want to skip the setup and get to the questions]

It seems like the most popular way to set up jails these days uses
read-only nullfs mounts of a base system and symbolic links into a
read-write nullfs mount for each jail's specific stuff (etc,
/usr/local, etc...).  These approaches are well described in:

 http://erdgeist.org/arts/software/ezjail
 http://www.freebsd.org/doc/en/books/handbook/jails-application.html

and they work fine with zfs based storage.

It's also possible to use unionfs to layer jail-specific storage over
a base system.  While this approach gives more per-jail flexibility
and avoids having to relocate various directories in the base system,
various unionfs problems seem to have pushed it out of favor.  The
ongoing work of daichi at freebsd.org et al. that fixes various problems
with unionfs,

   http://people.freebsd.org/~daichi/unionfs/

makes it look as if this approach might be now be safe, using
something like:

   mount -t unionfs -o below,noatime /usr/jails/base /usr/jails/www

The obvious zfs analog to this:

   mount -t unionfs -o below,noatime /tank/jails/base /tank/jails/www

fails with:

   mount_unionfs: /tank/jails/www: Operation not supported

A bit of digging suggests that the mount fails when the unionfs code
checks to see if /tank/jails/www supports whiteouts.  The fact that
this check doesn't occur if the uniondir is read-only provides a way
to superficially check if whiteouts are the only problem, this:

   mount -t unionfs -o ro,below,noatime /tank/jails/base /tank/jails/www

does indeed seem to lead to a working [albeit read-only] union mount.

One can work around the problem by creating a ZFS volume, building a
UFS filesystem on it, and then using that as the uniondir, e.g.:

  zfs create -V 5G tank/jail/vol1
  newfs /dev/zvol/tank/jail/vol1
  mkdir /usr/jail/zvol-www
  mount /dev/zvol/tank/jail/vol1 /usr/jail/zvol-www/
  mount -t unionfs -o below,noatime /tank/jail/base/ /usr/jail/zvol-www

The upper layer is still [presumably, I haven't tested these yet]
snapshot-able, send-able, etc.... but this approach leaves me with a
bunch of UFS filesystems that need care and feeding (fsck, etc...).

So finally, the question: How hard would it be to add whiteout support
to our ZFS?  Is it "just" a matter of understanding the places in the
UFS code that do whiteout things, locating the analogous places in the
ZFS tree and doing similar things (it seems to be a "simple" matter of
creating/destroying a whiteout vnode when necessary and checking for
it when appropriate) or is there something fundamentally harder about
it?  Has anyone already done it?  If it were doable/done cleanly,
might it get committed?

Thanks,

g.