jail design

[Jeremie Le Hen]

Hi Ollivier,

On Mon, Jul 29, 2013 at 03:43:58PM +0200, Ollivier Robert wrote:
> Hello,
> 
> I have a new server I'm going to run all my services on (www, smtp/imap, and so on).  Running 9.2-BETA1, full ZFS-on-root.
> 
> What is the best practices about jails knowing that:
> - I have only one IPv4
> - I have a full /48 IPv6 to play with
> 
> I've looked at ezjail which is doing most of what I need but it does not support ip4/ip6=inherit parameters (and no jail.conf support either) so my networking setup is more complicated. All the other packages like qjail have only limited ZFS support.
> 
> Do I need to setup pf to redirect all traffic in/out for specific ports to my jails? Or do I try to shoehorn "inherit" into ezjail?  Is inherit easier to deal with?  What are the security implications?
> 
> I need something as easy as ezjail or a way to tweek it, with
> - one jail for smtp/imap
> - one for www stuff, ideally one jail per hosted domain (using nginx)
> 
> I'm a jail newbie, in case you haven't found it already :)

I've run services in jail for roughly a decade, always by hand.  This is
not a big deal if you do it correctly because, well, you don't create
jail everyday :).  Especially with recent tools.

- First I started creating a jail for each service, sharing host's
  directories like /lib, /bin, /usr/lib, /usr/share, ... with nullfs in
  each jail in order to save space.

- Then I found that it was painful because when I upgraded the host I
  had to upgrade all the jails at once.  So I began to use the setup
  described in the handbook, section 16.6 [1].  Basically you create
  a base jail which is mounted r/o in each jail; all r/w directory are
  actually symlinks to a r/w space dedicated to the jail.

  This worked pretty well because when you want to upgrade your jail,
  you just create a new base jail and you progressively switch each jail
  to the new base, applying the upgrade procedure in each of them.

  At that time, every single service ran in its own jail.  Jails were
  bound to 127.0.0.2, 127.0.0.3 and I used the right rdr rules on my
  public IP address.

- Finally as time goes on and my free time decreases, I switched to the
  current model where I have very few jails, based on security domains
  (what you describe make sense: mail and web; you could also create one
  for shell access; honestly I don't advice to multiply jails, like one
  for each www domain, this is way to much adminitration overhead).
  These jails are full-fledged systems each on their own ZFS dataset.
  The very nice thing about that is that you can run a zfs send/receive
  every minute or so to backup your jails remotely, this is extremely
  efficient.

  Each jail is updated as if you had different servers, save the kernel
  of course.  Besides, with pkgng and poudriere (kudos to the authors),
  this is extremely simple to upgrade.

  Regarding IP address, I bought one IP address for each jail
  (Kimsufi/OVH and Dedibox both provide this I think).  The jail is
  simply brought up and down along with the jail.  As you may have
  understood, I have a spare server which synchronizes the jails very
  often.  If the main server fails, I can just start them on the backup
  one and perform an "IP failover" on my provider admin console.


As for firewalling, this is not rocket science.  The only difference is
that you need to avoid using interfaces for filtering.  You need to
consider all traffic to both go out and come in on the loopback
interface successively.

Hope this helps.

Cheers,

[1] http://www.freebsd.org/doc/handbook/jails-application.html
-- 
Jeremie Le Hen

Scientists say the world is made up of Protons, Neutrons and Electrons.
They forgot to mention Morons.