net.add_addr_allfibs - alternative usecases

From: Volodymyr Kostyrko <arcade_at_b1t.name>
Date: Mon, 04 Oct 2021 07:33:15 UTC
Hello.

First of all, I came here not to agitate for any change, I want to 
understand how my configuration is inefficient and how I can do that better.

I have two outgoing interfaces, if0 and if0. Those are different 
internet providers, I even get ipv6 through second one, and that's nice. 
I want to automatically fallback to the interface that is working in 
case of outage. Also, I want some traffic only on one of those 
interfaces. So I got 3 fibs:

fib 0: default route
fib 1: default route is if0
fib 2: default route is if1

Fibs 1-2 are used for traffic that should only pass through exactly that 
interface. Traffic pinning is done with PF:

pass out on $outside2 inet from ($outside2) queue(in_std2, in_priv2) 
modulate state rtable 2

For example, I can test connectivity to both sides via:

setfib 1 ping -qc 5 8.8.8.8
setfib 2 ping -qc 5 8.8.8.8

And in case one of them doesn't work I can switch to other one by 
changing routing on fib 0.

Everything seems to work fine with net.add_addr_allfibs enabled. But 
once it was disabled I started wondering whether I'm using the right 
tools to solve my problem, or this can be done easier. Disabling 
net.add_addr_allfibs means that only assigned interface will provide 
default route for correspondent fib, and you can't manually add them to 
the other fib. Or maybe I got that part totally wrong?

Thanks in advance, any bit of knowledge would be appreciated.

-- 
Sphinx of black quartz judge my vow.