setfib, jails and loopback interfaces

[James Gritton]

On 2017-05-31 02:33, Marko Cupać wrote:
> Hi,
> 
> I'm not subscribed to the list, could you please keep me in CC?
> 
> I'm using ezjail as instructed in Handbook, assigning jails
> lo1|127.0.0.X,bce0|10.66.66.X addresses, in order to keep jails'
> loopback traffic off host's, and in order to be able to keep internal
> services on lo1 (such as redis, mongodb, mysql etc.), and external on
> bce0 (such as apache, unifi5 etc.).
> 
> Recently I got a server with multiple NICs, and I'd like to serve both
> LAN and DMZ services from it. I found some information on how to
> accomplish that with setfib:
> 
> # cat /boot/loader.conf
> net.fibs=4
> net.add_addr_allfibs=0
> 
> # cat /etc/rc.conf
> ...
> cloned_interfaces="lo1"
> static_routes="nix nixd"
> route_nix="-net 10.66.66.0/24 -interface bce0 -fib 1"
> route_nixd="default 10.66.66.254 -fib 1"
> ...
> 
> In this setup, services bound to bce0 interface work fine, but they
> can't contact internal services on lo1. I guess it has something to do
> with jail routing, but can't figure out what.
> 
> Thank you in advance for any hints.

I haven't done the lo1 trick before, but I have had jails with addresses 
on a different FIB.  Note that the jail also has an FIB.  You probably 
at least want to set the jail's fib to 1 (exec.fib in jail.conf, I 
suppose jail_*_fib or whatever in the old rc-based system ezjail still 
uses).

The part I'm not sure about is you probably also want to have lo1's 
entries in the fib=1 routing table.  I don't know the interaction 
between cloned_interfaces and fib though - that might take some 
exploring in rc, or a word or two from someone who knows that side of 
things more than I do.

- Jamie