Jail unable to access some network devices

Mon Apr 9 13:07:08 UTC 2018

On 2018-04-06 1:32 pm, Dean E. Weimer wrote:
> I am trying to setup some jails on a server utilizing multiple routing
> fib tables I have a few jails already running some of them in
> different fibs, one in the same fib as the host. But this last one I
> added has caused some problems. Its the first one I am trying to build
> in this fib. Below is sections from the fib routing setup and the
> jail.
> 
> ## Set up the DMZ[1] routing table
> # Interface route(s)
> setfib 1 route add -net 10.50.0.0/16 -iface lagg0.4
> setfib 1 route add -net 10.52.0.0/16 -iface lagg0.4
> setfib 1 route add -net 172.17.0.0/25 -iface lagg0.900
> # Default route
> setfib 1 route add default 10.50.110.9
> 
> # Default Jail Options
> allow.mount.devfs;
> mount.devfs;
> devfs_ruleset = '4';
> allow.sysvipc;
> allow.dying;
> exec.start = "/bin/sh /etc/rc";
> exec.stop = "/bin/sh /etc/rc.shutdown";
> exec.consolelog = "/jails/${name}/console.log";
> 
> proxy2 {
>   exec.fib=1;
>   allow.raw_sockets = 1;
>   ip4.addr = 'lagg0.4|10.50.20.42/16';
>   ip4.addr += 'lagg0.4|10.52.20.42/16';
>   ip4.addr += 'lagg0.900|172.17.0.5/25';
> }
> 
> The jail can talk to every host on the 172.17.0.0/25 subnet, and can
> talk to every hos tin the 10.52.0.0/16 subnet. I can't talk to some
> things in the 10.50.0.0/16 subnet though. The 3 devices I can't talk
> to also have 10.52.0.0/16 addresses on the same adapter as the
> 10.50.0.0/16 I can talk to their 10.52 address but receive an invalid
> argument error when attempting to connect via 10.50 address.
> 
> Below the 10.50.20.4 and 10.52.20.4 are the same device, an older
> FreeBSD server that this jail is intending to replace.
> 
> root at fbsdhost2:/jails/proxy2 # proxy2
> root at proxy2-temp:/ # ping 10.52.20.4
> PING 10.52.20.4 (10.52.20.4): 56 data bytes
> 64 bytes from 10.52.20.4: icmp_seq=0 ttl=64 time=0.749 ms
> 64 bytes from 10.52.20.4: icmp_seq=1 ttl=64 time=0.311 ms
> ^C
> --- 10.52.20.4 ping statistics ---
> 2 packets transmitted, 2 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.311/0.530/0.749/0.219 ms
> root at proxy2-temp:/ # ping 10.50.20.4
> PING 10.50.20.4 (10.50.20.4): 56 data bytes
> ping: sendto: Invalid argument
> ping: sendto: Invalid argument
> ^C
> --- 10.50.20.4 ping statistics ---
> 2 packets transmitted, 0 packets received, 100.0% packet loss
> root at proxy2-temp:/ # ping 10.50.110.9
> PING 10.50.110.9 (10.50.110.9): 56 data bytes
> 64 bytes from 10.50.110.9: icmp_seq=0 ttl=255 time=0.599 ms
> 64 bytes from 10.50.110.9: icmp_seq=1 ttl=255 time=0.253 ms
> ^C
> --- 10.50.110.9 ping statistics ---
> 2 packets transmitted, 2 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.253/0.426/0.599/0.173 ms
> 
> root at fbsdhost2:/jails/proxy2 # setfib 1 netstat -rnf inet
> Routing tables (fib: 1)
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            10.50.110.9        UGS     lagg0.4
> 10.50.0.0/16       84:2b:2b:4d:6e:08  US      lagg0.4
> 10.52.0.0/16       84:2b:2b:4d:6e:08  US      lagg0.4
> 127.0.0.1          lo0                UHS         lo0
> 172.17.0.0/25      84:2b:2b:4d:6e:08  US     lagg0.90
> 
> I know it has to be something on the jail configuration, as it doesn't
> even attempt to send the traffic to the interface, watched on host
> adapter. If I ping this jail address from the other system the echo
> requests do show up, on that adapter. Can anyone see anything in the
> above configuration that I have missed, or got wrong?

I noticed these in the nightly emails over the weekend.

+arpresolve: can't allocate llinfo for 10.50.20.4 on lagg0.4

Didn't think to look at the dmesg output for errors last Friday. After 
searching on that I tried a reboot. The system is now talking to 
everything. Perhaps something to do with the fact that the main host is 
getting NTP currently from the devices I was having difficulty 
connecting to for a while before the other fib routing table was used.

Looks like I have a chicken before the egg issue, the host had an arp 
entry for these devices already assigned to its other interface through 
a route. when the lagg0.4 interface had the new IP address added for the 
jail (previously no IP was defined on that interface) the arp entry no 
longer matched the correct routing interface.

Short term defining an IP on the host in that subnet on the adapter will 
keep it from occurring long term though I may need to find a more secure 
solution. As this creates a path around the external firewall that is 
controlling the traffic going from the DMZ to the LAN where the host is 
located.

-- 
Thanks,
    Dean E. Weimer
    http://www.dweimer.net/