setfib (ez)jails and wierd routing

[Andrew Hotlab]

Hi Marko. I'm running an almost identical setup, but I do not have this issue: ICMP echo reply packets are sent from the right interface.
The only difference is that I didn't defined additional lo1 and lo2 interfaces, but I guess it shouldn't be the cause.

I'm running releng/10.3. Which release are you working on?

Andrew
________________________________________
From: owner-freebsd-jail at freebsd.org [owner-freebsd-jail at freebsd.org] on behalf of Marko Cupać [marko.cupac at mimar.rs]
Sent: Friday, September 29, 2017 10:32 AM
To: freebsd-jail at freebsd.org
Subject: setfib (ez)jails and wierd routing

Hi,

I notice wierd routing in my setfib (ez)jails setup.

I have a server with multiple NICs. setfib should ensure that LAN jails
(setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but
need to go through firewalls as though they were physical boxes.

pacija at warden3:~ % sudo setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
10.30.19.160/27    00:1c:c4:de:0a:86  US         bce0
127.0.0.1          lo0                UHS         lo0
127.0.1.0/24       lo1                US          lo1

pacija at warden3:~ % sudo setfib 2 netstat -rn
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            193.53.106.254     UGS        bce1
127.0.0.1          lo0                UHS         lo0
127.0.2.0/24       lo2                US          lo2
193.53.106.0/24    00:1c:c4:de:0a:84  US         bce1

Host has the same default route as fib 1:

pacija at warden3:~ % sudo netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
...

If I ssh from the Internet into DMZ jail, everything works as expected.
But if I ping DMZ jail from the Internet, I see reply packets leaving
not the interface they came from (bce1, public address space, DMZ), but
another one (bce0, private address space, LAN). This is kinda
understandable, because jail on fib2 does not have ICMP enabled, so
it is not DMZ jail, but the host (which is in fib 0) who replies to
packets via its default gateway (router on a private LAN).

Is there an easy and elegant way to solve this? Like binding IP address
to fib? I wouldn't like to have to fire up pf on host and meddle with
reply-to rules in order to achieve this, I'd rather revert to old setup
of separate physical servers for each network.

Thank you in advance,
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/
_______________________________________________
freebsd-jail at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscribe at freebsd.org"