[PATCH FOR REVIEW] Implementation of skeleton jail

[Simon L. Nielsen]

On 2006.05.20 15:21:00 +0800, Xin LI wrote:

> Here is an implementation of what I call it "skeleton jail".  The idea
> is that it is more or less to be common that we do not want to actually
> copy of the base system (sometimes even other stuff) across zillions of
> jails.

Hey,

Good to see more people thinking about this topic :-).

Some time ago I "designed" a similar system [1] which I named Service
Jails, though I did it the other way around wrt. the RO nullfs part
(which you might be aware of since I was CC'ed :-) ).

The big difference compared to what you describe, and to what ezjail
uses, is that Service Jails has a shared directory mounted RO at the
root of the jail and then a RW part mounted under /s inside the jail
and the shared root had apropriate symlinks for directories which
should be RW inside the jail.

This turned out to have some drawbacks, since it really requires a raw
partition per jail, which can be somewhat troublesome to manage.  And
probably more trouble than it's worth in most cases.  Nullfs
read-write or a vnode backed md(4) device could be use, but I'm not
too keen on either of those if I can avoid them.

(I mainly included this description to give a bit of thought to
different ways to implement light weight jails.)

[1] http://simon.nitro.dk/service-jails.html

> The skeleton jail is an approach that makes management of such jails
> easier, by making use of mount_nullfs(8) to make read-only shadow or
> read-write shadow from the so-called "skeleton root".
> 
> For instance, by default the skeleton jail would mount the following
> directories from the skeleton root (/) to the jail:

What is the advantage to making that many mounts compared to making
one mount and then symlinking, like ez-jail does?

> In order to create the environment that is suitable for the skeleton
> jail (say, create the directory hierarchy, populate the /etc/ stuff,
> etc, but not the actual installworld), I have added a new target
> "installskel" to src/Makefile which will help the work.

As Ruslan noted, installskel seems a bit redundant compared to just
using "make distribution".

> There are four variables that can be set in either system level default
> or per-jail way:
> 
>  - _skel_enable
> Whether to raise the jail from a skeleton root.  The default is NO
>  - _skel_root
> The place of skeleton root.  The default is "/"
>  - _skel_romounts
> Which directories (relative to the skeleton root) should be mounted
> read-only to the skeleton jail.  The default is shown above.
>  - _skel_rwmounts
> Which directories (relative to the skeleton root) should be mounted
> read-write to the skeleton jail.  The default is nothing, but a
> potential useful option might be "/usr/ports", except for security
> concerns.

As Dirk Engling noted there really isn't any reason to mount
/usr/ports RW.

> To try out the patch:
> 
>  - Apply the patch.
>  - Do a full "make buildworld" and potentially "make installworld" so
> that your system is fresh.
>  - Install the patched jail script into /etc/rc.d/ (e.g. can be done
> with rm /etc/rc.d/jail && mergemaster -i)
>  - Create a directory, i.e. "/vhosts/skeltest"
>  - Do a "make installskel DESTDIR=/vhosts/skeltest"
>  - Add the following stuff into /etc/rc.conf:
> jail_enable="YES"
> jail_list="skeltest"
> 
> jail_skeltest_rootdir="/vhosts/skeltest"
> jail_skeltest_hostname="skeltest.example.com"
> jail_skeltest_ip="127.0.0.1"
> jail_skeltest_devfs_enable="YES"
> jail_skeltest_exec="/bin/sh /etc/rc"
> 
>  - Do a "/etc/rc.d/jail start skeltest" or reboot to see the jail up.
> 
> Comments?

I would really like to see better support for light-weight jails in
the base systems, especially since I'm using it more and more (this
mail will bass through one such system :-) ).

I think it would be better to simply use symlinks and then include a
single RO mount like ez-jail does since you avoid having as many mount
points.  I think using symlinks could be created on demand like below,
though it is of cause a bit more work since you need to handle
creating /usr.  What do you think of this approach?

Personally I like the part of just including the list of magic shared
directories in rc.conf since I think it's simpler to manager in the
long term compared to a bunch of fstab.$jail files.  This also makes
it simpler to include per jail special mounts in fstab.$jail.

[...]

> --- etc/rc.d/jail	11 May 2006 14:23:43 -0000	1.32
> +++ etc/rc.d/jail	20 May 2006 06:53:10 -0000
> @@ -68,6 +68,16 @@
>  	eval _flags=\"\${jail_${_j}_flags:-${jail_flags}}\"
>  	[ -z "${_flags}" ] && _flags="-l -U root"
>  
> +	# Default settings for skel jail
> +	eval _skel_enable=\"\${jail_${_j}_skel_enable:-${jail_skel_enable}}\"
> +	[ -z "${_skel_enable}" ] && _skel_enable="NO"
> +	eval _skel_root=\"\${jail_${_j}_skel_root:-${jail_skel_root}}\"
> +	[ -z "${_skel_root}" ] && _skel_root="/"
> +	eval _skel_romounts=\"\${jail_${_j}_skel_romounts:-${jail_skel_romounts}}\"
> +	[ -z "${_skel_romounts}" ] && _skel_romounts="bin sbin lib libexec usr/bin usr/sbin usr/include usr/lib usr/libdata usr/libexec usr/sbin usr/share"

Shouldn't this rather be taken from defaults/rc.conf rather than being
hardcoded?

[...]

> @@ -152,6 +166,20 @@
>  		[ -f "${_fstab}" ] || warn "${_fstab} does not exist"
>  		umount -a -F "${_fstab}" >/dev/null 2>&1
>  	fi
> +	if checkyesno _skel_enable; then
> +		for _mntpt in $_skel_romounts
> +		do
> +			if [ -d "${_rootdir}/${_mntpt}" ] ; then
> +				umount -f ${_rootdir}/${_mntpt} > /dev/null 2>&1
> +			fi
> +		done
> +		for _mntpt in $_skel_rwmounts
> +		do
> +			if [ -d "${_rootdir}/${_mntpt}" ] ; then
> +				umount -f ${_rootdir}/${_mntpt} > /dev/null 2>&1
> +			fi
> +		done
> +	fi

I think these two loops could just be merged like:

		for _mntpt in $_skel_romounts $_skel_rwmounts

or?

-- 
Simon L. Nielsen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-rc/attachments/20060520/a4bd2e95/attachment.pgp