Jail management

[Mark Felder]

On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote:
> I've been using FreeBSD jails (with ezjail) for many years and they work
> very well. However I'm now reaching a critical mass (30+ jails) where I
> want to be able to manage them in bulk more easily.
> 
> In this environment, each jail runs just a single application, installed
> from a package built using poudriere from a custom port. That package
> depends on Java, so lots of other packages also get pulled in. That
> application gets new versions roughly once every 4 weeks. The problems I
> have right now are:
> 
> * FreeBSD's packaging system doesn't understand the concept of installing
> a particular package version, so all my scripts will by default upgrade
> the application to the current version even if I don't want to. I can't
> easily install a new jail at an old version.
> 
> * It is hard to reproduce the environment exactly, matching the
> application to the same version of Java that was available at the time of
> deployment. Again I'm fighting against the pkg system which always wants
> the latest version.

The package system *could* handle this, but it doesn't fit our design.
We aren't like RedHat/Debian where we "freeze" packages at a certain
version at the OS release and then backport only changes. With that
method different versions of packages will just work with everything
else in the system. With FreeBSD's ports system it's really a rolling
release as the entire ports tree moves together. Mixing packages build
from different checkouts of the ports tree is dangerous and not
guaranteed to work.

You may be better served with the quarterly branch of the ports tree
where things are mostly static for 4 months at a time. Only security and
major bugfixes trickle in. Software will get upgraded to fix security
issues -- the fixes are not "backported" as that overhead is
unmaintainable and is even being criticized in the Linux world these
days. The entire software ecosystem needs to stay nimble if we are to
stay secure. Upstream projects need to either learn to not break
functionality or to provide Long Term Support releases that people can
rely on. We definitely have growing pains.


> 
> * For failover I want each jail reproduced exactly on another host, or at
> least a snapshot which could be sent to another host within a few
> seconds. The jails are quite small (< 500Mb). Most of that is just the
> openjdk pkg.
> 
> 
> As I understand, ezjail doesn't support multiple base jails. If it did,
> then I could simply install the application (and packages) to the base
> jail and have versions of the base. Then by shutting down a jail,
> switching the base to the new version and starting up, everything would
> upgrade easily. Even better would be some concept of hierarchy with
> customer_jail sitting on top of base_version_1.0 which in turn sits on
> top of base_jail.
> 
> Would I need to abandon ezjail and be able to build all the above myself
> with a combination of nullfs (basejail) and unionfs (intermediate
> versioned jail)? Does unionfs now work with ZFS?
> 
> 
> Alternatively I could simply use zfs clones to deploy a new version of
> the application by destroying the whole jail and replacing it with a new
> one. I'd need to then script (I use saltstack) deploying the 2-3 config
> files which are different in each jail.
> 
> 
> 
> Thoughts? What seems like a more robust long term approach to jail
> management?
> 
> 

I don't use ezjail. It doesn't upgrade well, and changes to the base
jail require you stop all your jails. FreeBSD fat jails are so small
(300MB?) it's not worth it in my opinion. I simply wrote a shell script
to create fat jails and another script to handle updating them all.
They're all treated like full servers/VMs, and configs/roles are managed
with Ansible/Salt/etc.



-- 
  Mark Felder
  ports-secteam member
  feld at FreeBSD.org