Multihomed system with jails routing issues

[Alan Somers]

On Fri, Apr 4, 2014 at 8:22 PM, Chris Smith <chris at nevermind.co.nz> wrote:
> Hi All,
>
> I have a system with 1 network interface with 2 extra VLANs off it and I'm
> having some trouble getting the routing working correctly with it and jails.
>
> bge0     - management - 10.71.100.0/24
> bge0.101 - LAN        - 10.71.101.0/24
> bge0.103 - DMZ        - 10.71.101.0/24
>
> Here's what I want to achieve...
>
> Host:
> I want the host system to only listen on one interface, bge0. I want NO ip
> addresses of the host on the vlan interfaces. The only service it will be
> exposing is its sshd. The management address for this system is
> 10.71.100.50.
>
> Jails:
> The system will also host a variety of jails, each with an IP either on the
> LAN or DMZ. I am using ezjail to manage the jails.
>
> Router:
> There is a router at the .254 address of every subnet that can route between
> each network.
>
> I set up jail1 on bge0.101 with the IP 10.71.101.51. Since the host does not
> have an address configured on bge0.101, I configured the jail address as /24
> instead of the default /32.
>
> My issues:
>
> * If I do not configure the jail as a /24 (e.g. /32), the LAN cannot
> communicate with the jail.
>
> * When the jail is up and 10.71.101.51/24 is active, SSHing from the LAN to
> the mgmt interface via the router fails, as the host tries to send return
> traffic via the bge0.101 interface, even though traffic arrived via the bge0
> interface.
>
> So I did a whole lot of research for people having these apparently
> problems, and decided to try the multiple routing table/fib approach. So I
> recompiled my kernel, configured fib 1 with the LAN interface route (setfib
> route add 10.71.101.0/24 -iface bge0.101), set the jail fib and set the
> tunable net.addr_all_fibs = 0. I still can't get this working correctly.
> ezjail still seems to add the interface route to fib 0 by default (but it
> won't if i run ezjail with the setfib 1 command).

Routing is very broken when  you use multiple FIBs and set
net.addr_all_fibs=0.  See bugs

http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/167947
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187549
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187550
http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/187551
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187552
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187553
http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/187699

I'm actively working on fixing the above, but getting consensus on the
solution is slow.  If you can't solve your problem without  using
multiple FIBs, I will suggest a set of patches that may help you.  I
will need
1) Your ifconfig_bge0 etc lines from /etc/rc.conf
2) The output for "setfib X netstat -rn -f inet" for each fib in  use.
3) The rc.conf lines for any manually create routes that you're using,
for example gateways.

Also you may want to look at this possibly relevant bug.  I'm not
involved with this one:
http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/181794

-Alan


>
> Using FIB 1 and trying to ping hosts on the LAN gives an error like: sendto
> failed: invalid argument.
>
> Does anybody have any best practices for doing this, or anything else I can
> try? I'm happy to share/pastebin any configuration and I've tried most
> things I've found on the internet. I'm using FreeBSD 10.0 with a custom
> kernel for multiple routing tables.
>
> Thanks in advance!
> Chris.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"