Exposing a hierarchy of ZFS datasets inside multiple jails
jhell
jhell at DataIX.net
Tue Jun 21 20:21:11 UTC 2011
On Tue, Jun 21, 2011 at 08:51:22PM +0200, Paul Schenkeveld wrote:
> Hi,
>
> On Fri, Jun 17, 2011 at 02:46:59PM -0400, Lars Kellogg-Stedman wrote:
> > Hello all,
> >
> > Hi there,
> >
> > I am trying to expose a hierarchy of home directories to a number of
> > FreeBSD jails. The home directories are configured such that each is a
> > unique ZFS dataset. The jails are used for development work and hence
> > are created and destroyed on a regular basis.
> >
> > My first thought was simply to use nullfs to mount /home inside the
> > jail, but nullfs doesn't provide any way to access subordinate
> > filesystems.
> >
> > My second thought was to export the directories via NFS and then run
> > the automounter daemon (amd) inside each jail. This would have Just
> > Worked...if it were possible to perform NFS mounts inside a jail. But
> > it's not.
> >
> > My third thought was to run amd on the host and provision nullfs
> > mounts into the jails...but amd support for nullfs doesn't exist.
> >
> > My fourth thought was to go back to exporting the directories using
> > NFS, because of course amd works with NFS, right? Unfortunately,
> > rather than mounting a directory on the target mountpoint, amd likes
> > to mount things in a temporary location (/.amd_mnt/...) and then
> > create a symlink...which, of course, is useless inside the jail
> > environment.t
> >
> > So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to
> > the jail? No! This brings us back to my first attempt, in which we
> > find that there is no way to access subordinate filesystems using
> > nullfs.
> >
> > And then my head exploded.
> >
> > Is there a good solution for what I'm trying to do? A bad solution
> > would be to run a script after booting the jail that would create
> > multiple nullfs mountpoints for all the home directories, but this is
> > pretty clunky -- it would need to be run periodically to take into
> > account new directories or removed directories. So basically I would
> > have to write a poorly designed automounter.
> >
> > There must be a better way. How are other folks solving this?
> >
> > It looks like there are discussions going back several years about
> > setting the VFCF_JAIL on NFS filesystems, but it these haven't
> > resulted in any changes to the released code. Is this the best way to
> > go? In theory, if I build a kernel under which NFS is jail friendly I
> > can go ahead and run amd inside the jail
>
> Probably not a good solution but to stir the pool of thoughts a bit...
>
> Nullfs mounts and NFS mounts operate on filesystems (or datasets) and
> do not include subordinates. Smbfs operates on directory (sub)trees
> so have /home and /home/user[123...] datasets outside the jails, run
> samba there with a share called [home] (not to be confused with the
> [homes] share that comes with smb.conf.sample) and mount this share
> using mount_smbfs inside every jail (from fstab.<jailname>).
>
mount_smbfs needing rpcbind for the host that jails are being hosted on
and also rpcbind for each jail, your going to run into a problem here
due to the fact that rpcbind listens on the <anyaddress>*:N without
going in and making some specicial modifications for each jail and the
host itself by editing code and recompiling for each jail.
This is a known problem with RPC type services and jails and will only
escalate by the number of jails that you would have to implement this
on.
Instead of making each userdir a dataset I would suggest just creating
them as regular directories within a base dataset if this is the
approach you would like to take with nullfs.
More information about the freebsd-jail
mailing list