FBSD jail versus VMWare? What services do YOU run in a jail?
markham breitbach
markham_breitbach at ssimicro.com
Wed Apr 23 18:32:58 UTC 2014
I work for an ISP delivering services to most of the remote communities
across Canada's arctic via satellite. To mask some of the effects of
high latencies across satellite links we pushed our services out to the
edge, so things like email, dns, and authentication all happen nearly
instantly locally instead of having to wait for several seconds to pass
data back and forth across a satellite link. We have been using jails
for nearly a decade now quite happily.
Jails simplified maintenance tasks, improved our uptime and reduced
costs by creating a logical separation of services and reducing the
cost of redundancy. By planning our jails carefully, we were able to
isolate related services within separate jails, so when there was a
requirement to update one service we could easily create a patch on our
development jail, and push it out everywhere, and because services
were isolated regression testing became much simpler. I knew, for
example, that updates to sendmail and its dependencies could not impact
my LDAP installation in any way because they were essentially two
completely separate servers.
This logical separation also allowed us to maintain a simple rsync job
between two jail host servers at each remote site for each jail. In
the event of a hardware failure we could recover in a matter of minutes
by simply starting the jails on the backup server and flushing the
arp-cache on the switch. Because the jail is just another part of the
file system rsync was a very efficient way of maintaining
synchronization versus the need to copy a whole binary blob that
represents a typical VM disk image.
Our original design involved severely stripped down jails with nothing
more than the bare minimum of binaries, libraries and config to provide
a service. I think our original mail server jail packed down to about
25MB including sendmail, dovecot, and bind. This works because the
host system shares its kernel. We have since expanded the jails to
include a minimal installation, as we found troubleshooting to be
somewhat awkward in an environment without basic command line tools
like grep and tail, but this still keeps our jail systems to around
85MB for a complete system image.
More recently, I have setup a smallish web hosting environmet using a
small cluster of FreeBSD servers and a jail environment. Currently
hosting about 60 small business and personal website, although we have
had as many as 100 running with an average throughput of about 15Mbps
and a peak of about 60Mbps.
Five of the servers are running diskless with a netboot from the Master
and mounting a common "jail" partition via NFS. This allows for any
particular jail to be launched on any given client, although the
default is for the server to arbitrarily choose. The hosting
environment itself, is nullfs mounted into the jail as a read-only
partition, so each web-host can only write to their own home directory
and some local configuration files, allowing updates and patches across
the entire cluster in a single operation.
This has allowed me to repurpose some older hardware with some minor
upgrades instead of investing thousands of dollars into a brand-new
machine and gives me the advantage of redundancy (if one server dies
for some reason, I can restart those jails on another server within
seconds.) It also allows some isolation without having to enforce
strict limitations on everyone. In the event that one host becomes
heavily loaded it will only affect the hosts on that machine and not
all of my web hosts. If the problem cannot be quickly resolved, the
other hosts can easily be moved to another server with less than 30s
downtime.
While jails constrain you to operating within the FreeBSD environment,
and do not have all the advantages of a full virtualization solution or
clustering/cloud system, you also gain the advantage of minimized
overhead. Systems like VMWare can incur significant performance
overhead. I have run a large database installation (> 40M records) at
more than 4 times the speed on bare metal versus ESXi. The jail does
not incur any performance hits because there are not extra abstraction
layers. Your application is interfacing directly with the kernel the
same way it would if it were running outside the jail.
It is certainly possible to run any service within a jail, although
there are often some security implications (do you use sysVIPC, does
the application need access to /dev/kmem?) you really will need to look
at your own situation specifically to see if the advantages of jails
are suitable to your environment and applications. I like to keep an
open mind and try to apply the best tool for the job.
-Markham
---
[1]Markham Breitbach
Network Operations
SSi People, Ideas, Technology
- - - - - - - - - - - - - - - - - - - - -
+1 867 669 7500 work
+1 867 669 7510 fax
[2]markham_breitbach at ssimicro.com
[3]www.ssimicro.com
356B Old Airport Road
Yellowknife , NT X1A 3T4
Canada
- - - - - - - - - - - - - - - - - - - - -
Visit some of our other networks
[4]www.qiniq.com & [5]www.airware.ca
On 14-04-22 3:47 PM, edflecko . wrote:
I'm really interested in the comparison of using a FBSD jail rather than
VMWare in the context of virtualization.
At my business, we heavily use VMWare - you might say we consider ourselves
a VMWare "shop". 99% of our servers are virtualized.
I've heard that it's possible to run hundreds, if not thousands, of
services in FBSD jails on a given host server because of the sharing of
resources that all of your jails take advantage of. If I understand that
correctly, that's one of the HUGE advantages of running services in jails
as opposed to creating VM after VM after VM - each VM eats up disk space on
the SAN as well as memory resources, etc. Additionally, the jailed service
is far better from a security perspective?
Having said all of that, I'm curious to hear from some of you who may be
doing just this - are you running a FBSD server with some of your mission
critical services (Apache, Bind, DHCP, etc., etc.) within jails and how do
you like it versus running hundreds of VMs and VMWare?
What type of services CAN be run from within a jail?
Thank you,
Ed
_______________________________________________
[6]freebsd-questions at freebsd.org mailing list
[7]http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [8]"freebsd-questions-unsubscribe at freebsd.org"
References
1. http://www.ssimicro.com/
2. mailto:markham_breitbach at ssimicro.com
3. http://www.ssimicro.com/
4. http://www.qiniq.com/
5. http://www.airware.ca/
6. mailto:freebsd-questions at freebsd.org
7. http://lists.freebsd.org/mailman/listinfo/freebsd-questions
8. mailto:freebsd-questions-unsubscribe at freebsd.org
More information about the freebsd-questions
mailing list