VNET / netgraph jails -- Locking down?

[Jeff Kletsky]

For several years I've been using netgraph to provide connectivity for 
"service hosts" in jails on a "jail server"

Since I'm finally getting the jail server off FreeBSD 9 and solidly onto 
11, I've got the chance to rewrite the scripting of how I'm handling 
jail connectivity and am hoping that I can lock things down a bit better 
than what I have presently.


The approach I use looks similar to that now in the jail examples. Basically

                                           /---> ng_eiface_jail1
real_interface = ng_ether <---> ng_bridge <---> ng_eiface_jail2
                                           \---> ng_eiface_jail3

While this works well, it concerns me that the real interface has to be 
in promiscuous mode (and have autosrc off).

If one of the service jails is "taken over" then there isn't a way that 
I know of to lock out changing the IP address of the interface it has, 
or potentially gaining access to another VLAN through creation of a 
cloned interface, especially if the bridge is off the parent interface, 
not off a VLAN interface.


How do people manage this in practice when the jail has the risk of 
compromise?


I prefer approaches where the jail's notion of it's own IP address is 
the same as that of other hosts connecting to it, at least within my own 
little private-address-space world.


One thing that I've been considering is:
* Configure the jail's IP on the real interface (or appropriate VLAN 
interface) as an alias
* Send packets through ng_ipfw to an ng_eiface that the jail gets, using 
ipfw and a lookup table
* Tag the packets on return with ng_tag with a unique identifier for 
that jail's interface so ipfw can tell the only acceptable source IP
* Deny any so-tagged packets that don't have the proper source address

(jail ID by itself is not enough for the outbound packets, as some of 
the jails are dual homed.)


Has anyone tried this kind of method? Any other/better suggestions?


Would ng_ip_input be the appropriate way to "send" the packets coming 
from the jail?


Thanks!


Jeff