Idea about "skeleton jail"

[Justin Hopper]

On Mon, 2005-01-31 at 21:39 +0800, Xin LI wrote:
> Dear folks,
> 
> The recent discussion about whether we should have the perl port to
> touch/install /usr/bin/perl.  While I'm not interested in joining the
> discussion, it inspired me that we can make use of the fact that ports
> should not install things to "system" area and take advantage from it.
> Finally these ideas results me to hack up something that might be
> valuable to share with our users.
> 
> What I am going to proposal is a concept that I call it "skeleton jail",
> or "skeljail" for short.  A skel jail is something that shares most base
> system binaries/libraries with the host, through read-only mount_null's.
>
> I have already done some experiments.  Basically we want the following
> directories to be mount_null'ed:
> 	/bin, /sbin, /lib, /libexec, /usr/bin, /usr/sbin, /usr/include,
> 	/usr/lib, /usr/libdata, /usr/libexec, /usr/sbin, /usr/share

We had looked into this idea at one point for our hosting systems, but
what deterred us was the fact that on our systems we run several jails
per box, around 50, and to have a mount for each system directory (12 or
so) inside each jail would lead to a box loaded with mount points (600
+).  I never looked into it fully, but I assumed this would be a
resource problem, having so many mounts.  Also, at that time we were
using FreeBSD 4.4, and nullfs would sometimes cause kernel panics when
trying to umount the jails.

I'm curious if your idea for jails extends to running 50+ jails on a box
or not?  I'd definitely be interested in any feedback you have on what
problems may or may not be encountered with so many mounts and also the
stability of nullfs nowadays.

For our 5.x hosting platform, we used a single shared filesystem that
was mounted in each client jail, that contained the basic FreeBSD
distribution.  Ports are handled in a similar manner, having all the
"basic" and commonly used ports already installed in the shared
filesystem, and if the user wants to install their own ports, they go
into the user's filesystem.

We are considering open sourcing all of our stuff, to contribute back
what we can to the OS that allowed us to build our entire company.  I'd
really like to see what others have done to make jails more manageable,
as it seems like there is so much that can be done but not many people
are working on it.  It seems jails have the potential to become an
incredible way to virtually partition servers, and it would not be that
hard to implement solid tools for managing them.  We have things like
JID-aware top and tools for automated jail builds, but it would be great
to work with some FreeBSD heavies to finish up clean development of
things like jail resource restrictions (CPU,MEM,#PROCS,etc) and perhaps
a clean and universally useful way to easily configure and launch full
jail environments.

Pawel had some really interesting ideas for jails, but it seems that
he's too busy to work on them at the moment.  Speaking of which, his
multiple IPs patch for 5.3 is still broken, and I haven't been able to
find what the problem is =(

> To get most of what we want the jail to do, to work, this includes
> ssh(1) and something else.  Optionally, we may want to mount_nullfs a
> read-write /usr/ports/distfiles, a readonly /usr/ports, and something
> like /usr/game to be mounted into the skeljail.
> 
> In order to avoid having to do something magic instead of "make
> installworld", I have a patchset against src/Makefile and
> src/Makefile.incl to make the work a bit easier.  It adds a so-called
> "installskel" target that creates a skeljail that contains necessary
> directory hierarchy, and a set of /etc configuration files that will be
> useful to start the jail.  The target must be used after a ``make
> buildworld''
> 
> The two major benefits for the skeljail are:
> - Reduces the ordinary management cost because many base system files
> are shared, hence you patch only once to get all jails patched.
> - Reduces the space cost that needed for a newly created jail.  It used
> to need about 110MB and with skeljail you will only need no more than
> 3MB.
> 
> Apparantly skeljail is not suitable for those who want:
> - Run different FreeBSD releases on a single box.
> - Run ports that does touch system area.
> 
> But having it doesn't hurt the ability for you to run a full jail.
> 
> I have some handcrafted shell scripts to implement skeljail by having
> everything automatically mounted/dismounted.  However, I think it might
> be better if we can have jail_<name>_skeljail="YES" switch in our jail
> rc.d(8) startup script.  Please let me know if you are interested in the
> idea and I'll post a patch for review if there's enough people that
> wants this.
> 
> Thanks in advance!
> 
> Cheers,
-- 
Justin Hopper  <jhopper at bsdhosting.net>
UNIX Systems Engineer
BSDHosting.net
Hosting Division of Digital Oasys Inc.
http://www.bsdhosting.net