setfib (ez)jails and wierd routing

[Andrew Hotlab]

________________________________________
From: Marko Cupać <marko.cupac at mimar.rs>
Sent: Monday, October 16, 2017 4:18 PM
To: Andrew Hotlab
Cc: freebsd-jail at freebsd.org
Subject: Re: setfib (ez)jails and wierd routing

> On Sat, 30 Sep 2017 10:38:58 +0000
> Andrew Hotlab <andrew.hotlab at hotmail.com> wrote:
> 
> > I'm running releng/10.3. Which release are you working on?
> 
> sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely
> seeing packets with source addresses of my DMZ jails (fib2) exiting
> through interface on local LAN. Those are mostly icmp echo replies that
> should be coming from jails but are not due to the fact that jails
> don't have raw sockets enables. So, echo replies are returned from
> host (and not jails), whose default gateway is on internal network.
> 

I just setup a similar scenario on a FreeBSD 11.1 host. It seems that
all is working fine (172.21.10.0/24 is the DMZ, while 192.168.1.0/24
is the LAN). Please see the following transcript:

root at BSD11:~ # uname -msr
FreeBSD 11.1-RELEASE amd64

root at BSD11:~ # ifconfig | egrep '^[a-z]|inet '
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 172.21.10.100 netmask 0xffffff00 broadcast 172.21.10.255 
        inet 172.21.10.101 netmask 0xffffffff broadcast 172.21.10.101 
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000

root at BSD11:~ # netstat -rnfinet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS         em1
127.0.0.1          link#3             UH          lo0
172.21.10.0/24     link#1             U           em0
172.21.10.100      link#1             UHS         lo0
172.21.10.101      link#1             UHS         lo0
172.21.10.101/32   link#1             U           em0
192.168.1.0/24     link#2             U           em1
192.168.1.100      link#2             UHS         lo0

root at BSD11:~ # setfib 1 netstat -rfinet
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.21.10.254      UGS         em0
localhost          link#3             UH          lo0
172.21.10.0/24     link#1             U           em0
172.21.10.101/32   link#1             U           em0
192.168.1.0/24     link#2             U           em1

root at BSD11:~ # cat /etc/jail.conf 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
jtest01 {
  host.hostname = "jtest01.test.lab";
  path = /usr/jails/jtest01;
  ip4.addr = "em0|172.21.10.101/32";
  persist;
  allow.raw_sockets;
  exec.fib = "1";
}

root at BSD11:~ # jls
   JID  IP Address      Hostname                      Path
     8  172.21.10.101   jtest01.test.lab          /usr/jails/jtest01

root at BSD11:~ # ssh 172.21.10.101 'sysctl net.my_fibnum'
Password for root at jtest01.test.lab:
net.my_fibnum: 1

root at BSD11:~ # tcpdump -i em0 -n -p icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:07:19.524839 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 0, length 64
17:07:20.539686 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 1, length 64
17:07:21.551653 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 2, length 64
17:07:22.562764 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 3, length 64
^C
4 packets captured
12 packets received by filter
0 packets dropped by kernel


> Would freebsd-net be more appropriate list for this problem?

Maybe, but I would double check your jail configuration before ask to that list.
My guess is that your jail might not be associated to the right fib.