Jails - Pseudo Interfaces/ Virtual Networks - Best Practices?

[Jeff Kletsky]

I'm in the process of planning a transition from VMWare (on CentOS host) 
hosting service-specific FreeBSD virtual machines to the seemingly more 
efficient, stable, and lower maintenance cost approach, now that I've 
figured out at least some of the "magic" of creating jails under FreeBSD.

I do have a couple questions as to what the "recommended" approaches to 
the IP interface for the jails.

The questions generally relate to being able to consistently identify 
traffic from/to the jails, inside and outside the jail host, with tools 
such as ipfw and tcpdump.

One set of concerns that I haven't looked into is discussed in the 
thread 
<http://www.freebsd.org/cgi/getmsg.cgi?fetch=1418948+0+/usr/local/www/db/text/2008/freebsd-questions/20080720.freebsd-questions> 
where it appears that

     "To sum up, if jail host running at 10.0.0.1 connects to a jail 
running at
      10.0.0.2, the traffic will occur over lo0, and BOTH endpoints of that
      connection will use the jail (10.0.0.2) address.  To my mind, that 
can be
      problematic."

With VMWare, I was used to being able to create a pseudo-interface that 
could be "wired" to a virtual "switch" (hub, from what I can tell) and 
then that switch could be bridged (or NAT-ed) to a physical interface. 
This could lead to topologies that "looked like" physical machines and 
traffic could be controlled between them relatively unambiguously.

Some places where the FreeBSD jail "bind to physical interface" model 
seems to fail for me are described below. I'd appreciate insights into a 
good strategy to deal with these and related situations:


1) Firewall/bastion host, DNS intended for that host only

Here DNS needs to be run for the host itself only, to be able to start 
up NTP and allow ssh connections for management over the "protected" 
interface. In this situation, the DNS is *only* supplied to the physical 
host itself. Binding the jail to 127.0.0.1 doesn't seem right to me. 
Binding it to the "outside" (public) interface is equally bothersome.


2) Bastion host, SMTP relay

Here the firewall/bastion host is:
* Accepting connections on its public interface for SMTP connections and 
relaying to an internal SMTP server
* Accepting connections on its public interface for "submission" 
connections and relaying internally and externally for authenticated users
* Accepting connections on an internal IP for SMTP connections (with 
different Postfix "rules") and relaying both internally and externally
* Connecting to an internal DNS server, either in another jail on the 
same machine, or in an jail on another machine

In this case (and related ones) the firewall rules would be simplified 
if the packets from the jail were actually "in recv" from an 
identifiable interface so that the packets "looked like" those from a 
physical host on the DMZ network, allowing the "basic" access control 
and filtering to be handled by the "general DMZ" sections of the rules. 
(I'm aware that there is a "jail prisionID" selector in ipfw, but (a) it 
seems to only handle TCP and UDP, and (b) I'd likely have to repeat 
topology-related rules for each jail.)


Is there a "standard" way to create jails on a virtual interface or, 
perhaps better yet, "internal" network or switch, that would allow them 
to be handled en masse by topology-driven rules? I'm hardly a netgraph 
expert and didn't find any references in the documentation to this kind 
of use of the facility.

Is there a way to hang a "virtual DMZ" off a pseudo-interface somewhere 
"inside" the host in addition to the "physical DMZ" attached to a 
external interface
- routing tables cover virtual <=> physical communications
- packets "in recv" psuedo-interface from the virtual DMZ and ipfw can 
handle control of communication between two DMZ (and other) subnets

This could either be replicated once for each service jail, or a 
topology that had a "virtual smart switch" connecting multiple 
pseudo-interfaces that could be used by multiple jails, that then 
communicated with the rest of the world through a single pseudo-interface.

Thanks for taking the time to consider this.

Jeff