FBSD jail versus VMWare? What services do YOU run in a jail?

markham breitbach markham_breitbach at ssimicro.com
Wed Apr 23 18:32:58 UTC 2014

   I work for an ISP delivering services to most of the remote communities
   across Canada's arctic via satellite.  To mask some of the effects of
   high latencies across satellite links we pushed our services out to the
   edge, so things like email, dns, and authentication all happen nearly
   instantly locally instead of having to wait for several seconds to pass
   data back and forth across a satellite link.  We have been using jails
   for nearly a decade now quite happily.
   Jails simplified maintenance tasks, improved our uptime and reduced
   costs by creating a logical separation of services and reducing the
   cost of redundancy.  By planning our jails carefully, we were able to
   isolate related services within separate jails, so when there was a
   requirement to update one service we could easily create a patch on our
   development jail, and  push it out everywhere, and because services
   were isolated regression testing became much simpler.  I knew, for
   example, that updates to sendmail and its dependencies could not impact
   my LDAP installation in any way because they were essentially two
   completely separate servers.
   This logical separation also allowed us to maintain a simple rsync job
   between two jail host servers at each remote site for each jail.  In
   the event of a hardware failure we could recover in a matter of minutes
   by simply starting the jails on the backup server and flushing the
   arp-cache on the switch.  Because the jail is just another part of the
   file system rsync was a very efficient way of maintaining
   synchronization versus the need to copy a whole binary blob that
   represents a typical VM disk image.
   Our original design involved severely stripped down jails with nothing
   more than the bare minimum of binaries, libraries and config to provide
   a service.  I think our original mail server jail packed down to about
   25MB including sendmail, dovecot, and bind.  This works because the
   host system shares its kernel.  We have since expanded the jails to
   include a minimal installation, as we found troubleshooting to be
   somewhat awkward in an environment without basic command line tools
   like grep and tail, but this still keeps our jail systems to around
   85MB for a complete system image.
   More recently, I have setup a smallish web hosting environmet using a
   small cluster of FreeBSD servers and a jail environment.  Currently
   hosting about 60 small business and personal website, although we have
   had as many as 100 running with an average throughput of about 15Mbps
   and a peak of about 60Mbps.
   Five of the servers are running diskless with a netboot from the Master
   and mounting a common "jail" partition via NFS.  This allows for any
   particular jail to be launched on any given client, although the
   default is for the server to arbitrarily choose.  The hosting
   environment itself, is nullfs mounted into the jail as a read-only
   partition, so each web-host can only write to their own home directory
   and some local configuration files, allowing updates and patches across
   the entire cluster in a single operation.
   This has allowed me to repurpose some older hardware with some minor
   upgrades instead of investing thousands of dollars into a brand-new
   machine and gives me the advantage of redundancy (if one server dies
   for some reason, I can restart those jails on another server within
   seconds.)  It also allows some isolation without having to enforce
   strict limitations on everyone.  In the event that one host becomes
   heavily loaded it will only affect the hosts on that machine and not
   all of my web hosts.  If the problem cannot be quickly resolved, the
   other hosts can easily be moved to another server with less than 30s
   While jails constrain you to operating within the FreeBSD environment,
   and do not have all the advantages of a full virtualization solution or
   clustering/cloud system, you also gain the advantage of minimized
   overhead.  Systems like VMWare can incur significant performance
   overhead.  I have run a large database installation (> 40M records) at
   more than 4 times the speed on bare metal versus ESXi.  The jail does
   not incur any performance hits because there are not extra abstraction
   layers.  Your application is interfacing directly with the kernel the
   same way it would if it were running outside the jail.
   It is certainly possible to run any service within a jail, although
   there are often some security implications (do you use sysVIPC, does
   the application need access to /dev/kmem?) you really will need to look
   at your own situation specifically to see if the advantages of jails
   are suitable to your environment and applications.  I like to keep an
   open mind and try to apply the best tool for the job.

   [1]Markham Breitbach
   Network Operations
   SSi   People, Ideas, Technology
   - - - - - - - - - - - - - - - - - - - - -
   +1 867 669 7500 work
   +1 867 669 7510 fax
   [2]markham_breitbach at ssimicro.com
   356B Old Airport Road
   Yellowknife , NT X1A 3T4
   - - - - - - - - - - - - - - - - - - - - -
   Visit some of our other networks
   [4]www.qiniq.com   &   [5]www.airware.ca
   On 14-04-22 3:47 PM, edflecko . wrote:

I'm really interested in the comparison of using a FBSD jail rather than
VMWare in the context of virtualization.

At my business, we heavily use VMWare - you might say we consider ourselves
a VMWare "shop". 99% of our servers are virtualized.

I've heard that it's possible to run hundreds, if not thousands, of
services in FBSD jails on a given host server because of the sharing of
resources that all of your jails take advantage of. If I understand that
correctly, that's one of the HUGE advantages of running services in jails
as opposed to creating VM after VM after VM - each VM eats up disk space on
the SAN as well as memory resources, etc. Additionally, the jailed service
is far better from a security perspective?

Having said all of that, I'm curious to hear from some of you who may be
doing just this - are you running a FBSD server with some of your mission
critical services (Apache, Bind, DHCP, etc., etc.) within jails and how do
you like it versus running hundreds of VMs and VMWare?

What type of services CAN be run from within a jail?

Thank you,
[6]freebsd-questions at freebsd.org mailing list
To unsubscribe, send any mail to [8]"freebsd-questions-unsubscribe at freebsd.org"


   1. http://www.ssimicro.com/
   2. mailto:markham_breitbach at ssimicro.com
   3. http://www.ssimicro.com/
   4. http://www.qiniq.com/
   5. http://www.airware.ca/
   6. mailto:freebsd-questions at freebsd.org
   7. http://lists.freebsd.org/mailman/listinfo/freebsd-questions
   8. mailto:freebsd-questions-unsubscribe at freebsd.org

More information about the freebsd-questions mailing list