Multiple default routes / Force external routing

[sthaug at nethelp.no]

> I've poked about for weeks and asked similar questions in -questions and elsewhere without avail. Probably using the wrong keys to search and ask:
> 
> I have set up a box with various vlan interfaces on it. I naively expected to be able to set individual "default" routes and route between them via an *external* router (and filter packets there etc.) but somehow all packets seem to "short-circuit" locally, and I don't seem to be able to see why this is so and how I prevent that.

I found this behavior also, and it breaks POLA pretty badly.

There are several problems with the multiple routing table support (via
setfib) that I see:

- I found I needed "options ROUTETABLES= ..." to have additional routing
tables. I could not find this option documented anywhere.

- The standard behavior when adding new routes (via ifconfig or route
command) is that the route is added to all routing tables. Coming from
a router/MPLS/L3VPN background, this is extremely counterintuitive. I
found I needed to set the sysctl net.add_addr_allfibs to 0 to avoid
this behavior.

- Having two routing tables (one default, one table number 1 via setfib)
I also expected to be able to route between these via external router.
Pinging from the default routing table to routing table 1, traffic from
the default routing table goes out to external router and in via other
interface (in routing table 1) - but the ping reply is returned via the
loopback interface on the FreeBSD host, without going out to the router.
I assume this is the "short-circuit" you're talking about, and I find
this behavior also very counterintuitive.

If I explicitly ping from routing table 1 with ping prefixed by setfib 1,
everything works as expected (traffic both ways go via external router).

Steinar Haug, Nethelp consulting, sthaug at nethelp.no