jail fib no longer works after net.add_addr_allfibs=0

[qroxana]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, January 11, 2021 7:37 PM, Alexander V. Chernikov <melifaro at ipfw.ru> wrote:

> 11.01.2021, 14:59, "qroxana" qroxana at protonmail.com:
>
> > On Mon, 11 Jan 2021 13:25:51 +0000, Alexander V. Chernikov melifaro at ipfw.ru wrote:
> >
> > > Could you please consider clarifying the end result you want to achieve?
> > >  If you could include some more details of how it was configured earlier, it would help as well.
> >
> > Thank you for the quick reply.
> > Let's say there are two jails defined in /etc/jail.conf
> > jail1 {
> >     ...
> >     ip4.addr = 192.168.1.101;
> >     exec.fib = 1;
> >     ...
> > }
> > jail2 {
> >     ...
> >     ip4.addr = 192.168.1.102;
> >     exec.fib = 2;
> >     ...
> > }
>
> Got it, thank you for the clarification.
>
> > All the traffic in jail1 goes to the default router defined in fib 1,
> > and traffic in jail2 goes to the default router defined in fib 2.
>
> Could you describe interface&routing setup as well?
> In particular, I'm looking into details of setting up # of fibs, interface configuration and default route setup.

Sure, the interface is em0 for both host and jails:

/etc/rc.conf
ipv4_addrs_em0="192.168.1.100/24"

static_routes="jail1 jail2"
route_jail1="default 192.168.1.10 -fib 1"
route_jail2="default 192.168.1.20 -fib 2"

/etc/jail.conf
jail1 {
    ...
    interface = em0;
    ip4.addr = 192.168.1.101;
    exec.fib = 1;
    ...
}

jail2 {
    ...
    interface = em0;
    ip4.addr = 192.168.1.102;
    exec.fib = 2;
    ...
}

I noticed net.add_addr_allfibs defaults to 0 after the
commit 2d39824195933c173bbfc9b31773070202d2e30e
svn path=/head/; revision=367491

I also noted that net.add_addr_allfibs=1 needs to be added into
/etc/sysctl.conf so it can be set before running /etc/rc.d/netif.

# setfib -F 2 route add default 192.168.1.20
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.20 fib 2: Network is unreachable

# sysctl net.add_addr_allfibs=1
net.add_addr_allfibs: 0 -> 1

# setfib -F 2 route add default 192.168.1.20
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.20 fib 2: Network is unreachable

# /etc/rc.d/netif restart
# setfib -F 2 route add default 192.168.1.20
add net default: gateway 192.168.1.20 fib 2

I'm just wondering what's the best practice for using jails
with fib when net.add_addr_allfibs=0?

Thanks.