Communication between routing domains and nat

[kaycee gb]

Hello,

I am experimenting with routing domains/fibs and I'm blocked by this situation. 

The topology
 ____________________
| Fbsd box / fib0    |
|  _10.91.0  __      |---ext link----------
| | j1 / fib1  |     |                     |
| |net 10.91.1 |     |                     |
| |__bridge1___|     |                     |
|  ____________      |                _____|_____
| | j2 / fib2  |     |    tunnel     |           |
| | net 10.91.2|     |               |192.168.1  |
| |__bridge2___|     |---------------| service1  |
|____________________|               |___________|

fib0 has a default route to reach the world and a route to join service1 via
the tunnel. fib2 has a restricted routing information and a default route via
bridge2 (renamed to jsw2). 

# netstat -rn4 -F 0
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            EXTGW	      UGS      vtnet0
10.0.0.0/8         127.0.0.1          UR1         lo0
10.91.0.254        link#3             UHS         lo0
10.91.0.254/32     link#3             U          jsw0
10.91.100.0/24     tun0               US         tun0
10.91.100.1        link#10            UHS         lo0
10.91.110.0/24     tun1               US         tun1
10.91.110.1        link#11            UHS         lo0
10.255.1.1         link#6             UHS         lo0
10.255.1.2         link#6             UH         gre0
10.255.11.1        link#7             UHS         lo0
10.255.11.2        link#7             UH         gre1
10.255.255.1       link#8             UHS         lo0
10.255.255.2       link#8             UH         gre2
127.0.0.1          link#2             UH          lo0
169.254.0.0/16     127.0.0.1          UR1         lo0
172.16.0.0/12      127.0.0.1          UR1         lo0
EXTERNALNET/22     link#1             U        vtnet0
EXTERNALIP	   link#1             UHS         lo0
192.168.0.0/16     127.0.0.1          UR1         lo0
192.168.1.0/24     10.255.1.2         UG1        gre0

# netstat -rn4 -F 2
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.91.2.254        UGS        jsw2
10.91.0.254/32     lo0                US          lo0
10.91.2.1          link#5             UHS         lo0
10.91.2.1/32       link#5             U          jsw2
10.91.2.2          link#5             UHS         lo0
10.91.2.2/32       link#5             U          jsw2
10.91.2.3          link#5             UHS         lo0
10.91.2.3/32       link#5             U          jsw2
10.91.2.5          link#5             UHS         lo0
10.91.2.5/32       link#5             U          jsw2
10.91.2.254        link#5             UHS         lo0
10.91.2.254/32     link#5             U          jsw2
127.0.0.1          lo0                UHS         lo0

With the help of pf I am able to reach service1 (which is in fib0 ) from j2
( which is in fib2) via the tunnel.
pass out log quick on jsw2 proto udp from $j2 to $rsnns port 53
rtable 0 
So it seems routing between domains works. 

I am trying to reach the same service via the external net. The rule based on
the above one.
pass out log quick   on jsw2 proto udp from $j2 to $rsnextns
port 53 rtable 0

But that is not working. The connection hang for a moment and timeouts. 

If I add EXTERNALNET and change default gateway via EXTERNALGW in fib2, I can
reach service1 via external link without changing anything in pf. 

I do not really understand why this is blocking. I am looking for some time and
can't find an explanation for that. Should I expect routing problems when NAT
is involved with fibs ? I don't know. After adding the EXTERNALs to fib2 that
is working and that uses NAT too. 

I am for sure missing something. Anyone running something similar succesfully ? 

Oh, because I forgot that, host is running on FreeBSD 11.3 amd64. 

P.S. I hope my beautilful ascii art will stay intact :x

Kaycee,