Developing and maintaining a rapidly deployable image of an installed system

[Bill Moran]

In response to Greg Larkin <glarkin at FreeBSD.org>:

> Bill Moran wrote:
> [...]
> > 
> > The host system is a very basic install -- mostly just give it an IP and
> > add users for the administrators.  All the ports and details of their
> > configs are in the individual jails.  It's much more efficient than using
> > something like VMware, which has horrific performance penalties.
> > 
> 
> Hi Bill,
> 
> We've been using VMware with FreeBSD guests for a few years now without
> any performance problems. In fact, jails even work well within a FreeBSD
> VM.  What kinds of problems have you run into?

Really?  Do you have any machines with more than 30 virtual systems on
them?  Even our beefiest hardware can't handle more than 10 VMWare
machines.

We use both jails and VMWare.  Each has it's pros and cons.  The big
pro of VMWare is that you can simulate an entire piece of hardware,
which is necessary for much of our lab and testing work.  The big con
of VMWare is performance and overhead.  Another big pro of VMWare is
that we can have FreeBSD, and Linux, and MS operating systems all
running on the same hypervisor, which jails can't do.

The big pro of jails is that we can put a crapload of jails on each
physical server.  Six or 8 is typical, but we have systems with more
than 30 humming along happily.  Each jail uses far less disk space
than a virtual machine, and uses far less CPU.  Also, the jail
mechanism puts fewer layers between the OS and the hardware, which
means things like network and disk performance suffer very little.

Take two equivalent machines and put a FreeBSD jail on one and
FreeBSD in a virtual machine on the other and benchmark the network
and disk performance on each.  You'll find that VMWare loses big time.
If that's not enough to convince you, then increase the number of
systems on each machine to about 10 and rerun the tests while the
systems are under load -- VMWare doesn't scale up nearly as well
as jails do.  If you're doing purely CPU usage, then the two options
appear to be roughly equivalent, although I've never done an actual
test.

Restarting a jail takes seconds, restarting a VM takes minutes.

We've frequently had to go through our dev servers and shut down
VMWare virtual machines to free up resources when a few of the VMs
were seeing heavy usage.  We've had the same problem with jails far
less frequently.

Like I said earlier, VMWare has its advantages, but performance is
not one of them.

And yes, we have VMWare VMs with FreeBSD jails inside them.  It's a
bizarre combination, but it works quite well ... right up until
someone wants to load up one of the PostgreSQL servers and the disk
issue brings the system to its knees ... that's actually a pretty
good benchmark to illustrate the problem: pgbench will show how much
VMWare hurts disk performance right out of the gate.

I guess the overall issues are more with scalability than performance,
but the two issues are linked in such a way that I frequently don't
separate them.  VMWare starts with a minor performance hit compared
to jails, that performance hit increases significantly as you add
VMs, whereas jails scale up very well.

-- 
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/