A simple routing question - SOLVED

Robert Eckardt rol at Robert-Eckardt.de
Sat Feb 11 19:28:13 UTC 2017

Hi again,

thanks to all, who helped me by pointing me to some interesting
articles on this topic.
Eventually, I found a solution for my problem by employing
policy based routing with layer-2 ipfw rules and multiple routing
tables (FIBs).
What makes things a little more complicated is the fact that both
routers are addressed via the same interface.

This summary is for the records for those who face the same problem.

Using ECMP routing with kernel option RADIX_MPATH and two default
routes proved unsuitable.
a) since the two routers do NAT, packets must be returned through
    the same router as the connection was established and not the
    one based on destination address. (This is different from the
    situation without NAT.)
b) After one router had failed for a short time (disconnected from
    power supply) a telnet to a host in the internet failed with
    "no route to host" while ping worked.
    I did not dig deeper into it as I concluded from several
    postings that the RADIX_MPATH feature is still unstable.

Instead, (thanks to Matthew) I decided for policy based routing.
Getting ipfw to filter the source MAC wasn't difficult.
Understanding the behaviour of the stateful rules and the effect
of layer-2 or layer-3 filtering was hard.

Finally the following configuration works (for me):

1) since we need another forward information base, increase available 
    in /boot/loader.conf

2) (after reboot) set alternate default routing (this needs to get set
    on every boot, so I put it in /etc/rc.local)
         setfib 1 route delete default
         setfib 1 route add default

3) Don't forget to also add routes to provider-specific infrastucture
    (e.g. name servers) via the correspondig router ('setfib 0' opt.)
    (This can go as a static route in /etc/rc.conf or in /etc/rc.local)
         setfib 0 route add

4) make sure that the logic of layer-3 rules works as before
          2000 skipto 10000 ip from any to any not layer2

5) add layer-2 rule to ipfw
          3000 allow tag 101 ip from any to any MAC any 08:9a:bc:de:ff:ff 
in recv re1

6) avoid matching of the default deny rule for layer-2 by accepting
    everything in ether_demux
          9000 allow ip from any to any layer2
         10000 <from here existing layer-3 rule-set>

7) switch on filtering of layer-2 packets
    (either in /etc/rc.local or in /etc/sysctl.conf)

8) use stateful information on layer-3
         20000 check-state
         20100 setfib 1 ip from any to any tagged 101 keep-state
    The incoming packet, identified on layer-2 by the MAC addres of
    Router2, is handled by the alternate FIB with the default route
    pointing back to Router2.

9) make sure that in the end packets get logged if some rules don't
    work as expected
         65500 deny log ip from any to any MAC any any   <-- layer-2
         65530 deny log ip from any to any               <-- layer-3

10) when accessing Router2, which forwards to Server, via Router1 and
     the internet, you will see working dynamic rules
         ## Dynamic rules (1 144):
         20100 STATE tcp 51620 <-> 643

I found (besides 'man ipfw' ;-) the following links most helpful

Comments welcome.


Am 05.02.2017 11:30, schrieb Robert Eckardt:
> Hi all,
> currently I'm trying to solve a problem that, as I see from
> studying the archives, pops up on and off again. Yet, I was
> unable to find a solution.
> A local network connects to the internet using two providers
> with routers (AVM FritzBox) that do the IPv4-NATing and a
> server (FreeBSD 11.0-RELEASE with RADIX_MPATH und ipfw) as
> part of the inner firewall (see the illustration below).
>                     Internet
>                    /        \
>                   /          \
>    Provider2--
> ns.provider1.de  |            |  ns.provider2.de
>                  |            |
>         www.domain1.de    www.domain2.de
>                  |            |
>               Router1      Router2
>                  |            |
>                  +---+--------+
>                      |
>                    Server
>                      |
>                     LAN
> The default routes of Server point to both routers and static
> routes are defined e.g. for the respective name servers.
> Requests to domain1 are answred correctly, requests to domain2
> try their way via Router1 as ECMP routing does not take into
> account, where the connection originated from.
> However, packets of connections coming in via Router2 must
> also be returned via Router2.
> (I mainly care for TCP, but UDP would also be interesting.)

More information about the freebsd-questions mailing list