Issues with Jails/Routes/FIBs

Brandon Gooch jamesbrandongooch at gmail.com
Thu Nov 25 20:48:47 UTC 2010


On Nov 25, 2010, at 1:38 PM, Kevin Mai <kma at mrecic.gov.ar> wrote:

> Hi folks! 
> 
> I'm facing an issue here while trying to define separate routing tables for each jail and host. 
> 
> Let me show you briefly how it's done: 
> 
> The server has 3 physical NICs, each one connected to a different network (say, public network A, public network B, and LAN). 
> 
> Currently, the default gateway is set to be the LAN gateway, even though the two jails can see their own public network subnet. 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 172.16.2.1 UGS 1 3935 bce2 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 172.16.2.127 link#3 UHS 0 0 lo0 
> 100.16.97.0/24 link#1 U 0 0 bce0 
> 100.16.97.5 link#1 UHS 0 0 lo0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 100.16.98.5 link#2 UHS 0 0 lo0 
> 
> 100.16.97.0/24 and 100.16.98.0/24 are the two public networks and 172.16.2.0/24 is the LAN. 
> 
> I have already tried removing devfs rules from the jails, setting securelevel to -1 but I'm still out of luck.. 
> 
> I know setfib can define alternate routing tables, and I even created a default gateway for two fibs, 1 & 2: 
> 
> [root at mrefns09 ~]# setfib 2 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.98.100 UGS 14 906 bce1 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 39 bce0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 
> [root at mrefns09 ~]# setfib 1 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.97.100 UGS 0 1758 bce0 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 44 bce0 
> 100.16.98.0/24 link#2 U 0 4 bce1 
> 
> And i've added the proper settings in rc.conf.. 
> 
> jail_athea97_ip="100.16.97.5 netmask 255.255.255.0" 
> jail_athea97_fib=1 
> 
> 
> jail_athea98_ip="100.16.98.5 netmask 255.255.255.0" 
> jail_athea98_fib=2 
> 
> Am I missing something? because once I get into the jail the routing table is the same: 
> 
> [root at athea97 /]# netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 172.16.2.1 UGS 13 6175 bce2 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 172.16.2.127 link#3 UHS 0 0 lo0 
> 100.16.97.0/24 link#1 U 0 0 bce0 
> 100.16.97.5 link#1 UHS 0 0 lo0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 100.16.98.5 link#2 UHS 0 0 lo0 
> 
> [root at athea97 /]# setfib 1 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.97.100 UGS 15 1814 bce0 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 44 bce0 
> 100.16.98.0/24 link#2 U 0 4 bce1 
> 
> The other jail is acting the same way. I know that since I'm doing a jexec, the shell will have the host's route because, but, how can I know if it's getting the alternate routing table? 
> 
> Thanks, 
> 
> Kevin

Try ssh'ing into one of the jails from the public side. The jail should honor the FIB configuration from that perspective. Are things behaving as you expect in the jail at that point?

As you've figured out, when jexec'ing into the jail from the host machine, you inherit the FIB of your current shell.

I think this due to the design of FreeBSD's multiple routing tables -- and not a bug :)

-Brandon


More information about the freebsd-current mailing list