A simple routing question - SOLVED

[Ian Smith]

In freebsd-questions Digest, Vol 662, Issue 7, Message: 6
On Sat, 11 Feb 2017 20:27:37 +0100 Robert Eckardt <rol at Robert-Eckardt.de> wrote:
 > Hi again,
 > 
 > thanks to all, who helped me by pointing me to some interesting
 > articles on this topic.
 > Eventually, I found a solution for my problem by employing
 > policy based routing with layer-2 ipfw rules and multiple routing
 > tables (FIBs).
 > What makes things a little more complicated is the fact that both
 > routers are addressed via the same interface.
 > 
 > This summary is for the records for those who face the same problem.
 > 
 > Using ECMP routing with kernel option RADIX_MPATH and two default
 > routes proved unsuitable.
 > a) since the two routers do NAT, packets must be returned through
 >     the same router as the connection was established and not the
 >     one based on destination address. (This is different from the
 >     situation without NAT.)
 > b) After one router had failed for a short time (disconnected from
 >     power supply) a telnet to a host in the internet failed with
 >     "no route to host" while ping worked.
 >     I did not dig deeper into it as I concluded from several
 >     postings that the RADIX_MPATH feature is still unstable.
 > 
 > Instead, (thanks to Matthew) I decided for policy based routing.
 > Getting ipfw to filter the source MAC wasn't difficult.
 > Understanding the behaviour of the stateful rules and the effect
 > of layer-2 or layer-3 filtering was hard.
 > 
 > Finally the following configuration works (for me):
 > 
 > My SOLUTION:
 > 1) since we need another forward information base, increase available 
 > FIBs
 >     in /boot/loader.conf
 >          net.fibs="2"
 > 
 > 2) (after reboot) set alternate default routing (this needs to get set
 >     on every boot, so I put it in /etc/rc.local)
 >          setfib 1 route delete default
 >          setfib 1 route add default 10.0.0.2
 > 
 > 3) Don't forget to also add routes to provider-specific infrastucture
 >     (e.g. name servers) via the correspondig router ('setfib 0' opt.)
 >     (This can go as a static route in /etc/rc.conf or in /etc/rc.local)
 >          setfib 0 route add 2.2.3.4 10.0.0.2
 > 
 > 4) make sure that the logic of layer-3 rules works as before
 >           2000 skipto 10000 ip from any to any not layer2
 > 
 > 5) add layer-2 rule to ipfw
 >           3000 allow tag 101 ip from any to any MAC any 08:9a:bc:de:ff:ff in recv re1
 > 
 > 6) avoid matching of the default deny rule for layer-2 by accepting
 >     everything in ether_demux
 >           9000 allow ip from any to any layer2
 >          10000 <from here existing layer-3 rule-set>
 > 
 > 7) switch on filtering of layer-2 packets
 >     (either in /etc/rc.local or in /etc/sysctl.conf)
 >          net.link.ether.ipfw="1"

If in /etc/rc.local, that obviously? needs preceding with 'sysctl '

 > 8) use stateful information on layer-3
 >          20000 check-state
 >          20100 setfib 1 ip from any to any tagged 101 keep-state
 >     The incoming packet, identified on layer-2 by the MAC addres of
 >     Router2, is handled by the alternate FIB with the default route
 >     pointing back to Router2.
 >
 > 9) make sure that in the end packets get logged if some rules don't
 >     work as expected
 >          65500 deny log ip from any to any MAC any any   <-- layer-2
 >          65530 deny log ip from any to any               <-- layer-3
 >
 > 
 > 10) when accessing Router2, which forwards to Server, via Router1 and
 >      the internet, you will see working dynamic rules
 >          ## Dynamic rules (1 144):
 >          20100 STATE tcp 1.1.1.123 51620 <-> 10.0.0.2 643
 > 
 > 
 > I found (besides 'man ipfw' ;-) the following links most helpful
 > http://daemonforums.org/showthread.php?t=4610
 > https://forums.freebsd.org/threads/2201/
 > https://blog.plitc.eu/2014/freebsd-10-1-pbr-ecmp-fib-und-ipfw-layer2-filtering/
 > 
 > Comments welcome.
 > 
 > Regards,
 > Robert

Very elegant.  I haven't done anything at layer2 for years, not have I 
used tagging, but it smells good; happy to take your word it's working.

Would you care to post that - including your original question and 
diagram below - to freebsd-ipfw at freebsd.org, where I think it would be 
a) well appreciated and b) scrutinised by those who know ipfw best?

cheers, Ian