mount_unionfs for jails

[Matthew Seaman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Nejc S wrote:
| Hello,
| 
|> Here is what I am trying to do:
|>
|> mount_unionfs -o below /usr/jails/basejail /usr/jail/jail1
|>
|> after I do that I edit /usr/jail/jail1/etc/rc.conf and add the appropriate
|> entries to the host system rc.conf, but when I start the jail it starts
|> using the settings from /usr/jails/basejail.
| 
| I have the same setup and it works for me.
| 
|> Is my mount_unionfs syntax wrong, is this a bug in unionfs (man page says
|> unionfs is broken, but doesn't specify how its broken) or is this expected
|> behavior?
| 
| It shouldn't be wrong. I have this in my fstab:
| 
| /jail/base   /jail/spl/nejcspl       unionfs rw,noatime,below     0       0
| 
| (noatime option is completely optional, of course.)
| 
| But, if I were you, I would update the RELEASE to STABLE. This will also fix
| some bugs in unionfs. However at least some other bugs still aren't fixed in 7-STABLE
| to this day (most notably the socket bug, which prevents mysql from running in a jail
| and writing socket file to /tmp/mysql.sock), so we had to MFC the patch from
| HEAD manually. If you need the patch, let me know.
| 
| However, I don't suggest running jails on top of unionfs where you need
| decent stability (i.e. in production). I am writing thesis at the moment which
| also covers this topic. We also stumbled upon these issues:
| 
| - socket file bug, mentioned before, still present in 7-STABLE, no ideas
|   when it will be MFCed;
| 
| - "mv" bug (see freebsd-fs archives for August 2008, me and my friend posted
|   a few posts there) which causes troubles when moving directories (files
|   would appear as gone and then reappear again) which exist or don't exist
|   on lower and upper levels;
| 
| - another "mv" bug which I discovered yesterday and seems to be very strange
|   and hard to replay - I didn't even mess with the lower level, it seems that
|   also just the upper layer can behave strangely sometimes (erros like
|   "mv: invalid argument" when simply trying to move a big (>10 GB) directory -
|   the error was gone after I restarted the jail (i.e. also remounting the
|   unionfs);
| 
| - strange behaviour of some applications (apache in my case) not "seeing" the
|   lower layer (/etc/hosts most notably) - we had to do "touch" (and then copy
|   to all jails on change) on files we _really_ need to be visible. However,
|   after we "fscked" our partition with unionfs directories, we weren't able
|   to reproduce this error;
| 
| - UFS filesystem would get to inconsistent state (we don't know exactly when)
|   so some commands would behave strangely and fsck (see above) is needed in
|   single user mode;
| 
| - _most notably_: there hasn't been a single reply to our unionfs related
|   problem reports and posts to freebsd-fs list. So I guess that people who
|   are in charge for unionfs in FreeBSD aren't really responsive and that
|   the future of unionfs in FreeBSD isn't really bright. It's a pity, though,
|   since this is a very useful feature, especially for jailed systems. However,
|   hope remains, that things will be fixed at least in 8.0 if not in 7.1.

I think the problem is not so much lack of interest amongst available people,
as lack of available people interested in work on that bit of filesystem code.
Problems with unionfs and generally with VFS related stuff have been around for 
quite some time.

| So, you can see that there are (still) many issues with unionfs on FreeBSD.
| Please let me know if you are able to solve your problem. Or else we can make
| this list a little longer. :)

Having just gone through an attempt to set up a series of jails using unionfs
layering, here's another annoyance.  What I wanted to do was have a 'basejail'
+ unionfs overlay setup, but with various directories (/home, /usr/ports, /usr/src,
and so forth) shared (ie. nullfs mounted) between all the jails and the base
system.  However an fstab.jails.jail0 like this:

/jails/basejail /jails/jail0 unionfs rw,noatime,below,copymode=transparent,whiteout=whenneeded  0  0
/usr/ports	/jails/jail0/usr/ports	nulls	rw 0  0

fails to work saying 'no such file or directory: /jails/jail0/usr/ports'
irrespective of the existence of that directory in either layer of the unionfs.
Seems it's impossible to have a nullfs mountpoint on top of a unionfs 
filesystem.

At least, that's as far as I got when playing with this: press of time led
me to implement a work around.  If anyone knows how to get such a setup working
as I originally intended I'd be very glad of a pointer to any documentation.

	Cheers,

	Matthew

- -- 
Dr Matthew J Seaman MA, D.Phil.                       Flat 3
~                                                      7 Priory Courtyard
PGP: http://www.infracaninophile.co.uk/pgpkey         Ramsgate
~                                                      Kent, CT11 9PW, UK
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREDAAYFAkjclSoACgkQ3jDkPpsZ+VbP5QCfaGxXTnWIv2075a7yCseBgJQo
6IAAn3dCnTM9953sh9M54tMKA5w0p8Va
=br6C
-----END PGP SIGNATURE-----