PF - load balancing outgoing connections

[Michael Proto]

On Mon, Oct 19, 2009 at 11:48 AM, Jed Gainer <jedgainer at gmail.com> wrote:
> I wanted to setup a machine as my LAN gateway and have it load balance over
> multiple WANs. When I found http://www.openbsd.org/faq/pf/pools.html I
> choose FreeBSD as the machines OS. After getting it up and running, and
> acting as a gateway just using one WAN via
>
> *# macros
> wan1="nfe0"
> lan1="rl0"
>
> pc1="10.0.0.2"
> xb1="10.0.0.3"
>
> # options
> #set block-policy return
> #set loginterface $wan1
> set skip on lo0
>
> # scrub
> scrub in
>
> # nat/rdr
> nat on $wan1 from !($wan1) -> ($wan1:0) static-port
>
> # uTorrent
> rdr on $wan1 proto tcp from any to any port 41016 -> $pc1
>
> # Xbox Live
> rdr on $wan1 proto {tcp, udp} from any to any port 3074 -> $xb1*
>
> I decided to try the load balancing and came up with quite a few different
> pf.confs that did not work, my LAN just lost all connectivity when I loaded
> them.
> *
> lan1r = "10.0.0.0/24"
> lan1  = "rl0"
> wan1 = "nfe0"
> wan2 = "rl1"
> gw1 = "10.0.1.2"
> gw2 = "10.0.2.2"
>
> # nat outgoing connections on each internet interface
> nat on $wan1 from $lan1r to any -> ($wan1) #static-port
> nat on $wan2 from $lan1r to any -> ($wan2) #static-port
>
> # default deny
> block in from any to any
> block out from any to any
>
> # pass all outgoing packets on internal interface
> pass out on $lan1 from any to $lan1r
>
> # pass in quick any packets destined for the gateway itself
> pass in quick on $lan1 from $lan1r to $lan1
>
> # load balance outgoing tcp traffic from internal network.
> pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto
> tcp from $lan1r to any flags S/SA modulate state
>
> # load balance outgoing udp and icmp traffic from internal network
> pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto {
> udp, icmp } from $lan1r to any keep state
>
> # general "pass out" rules for external interfaces
> pass out on $wan1 proto tcp from any to any flags S/SA modulate state
> pass out on $wan1 proto { udp, icmp } from any to any keep state
> pass out on $wan2 proto tcp from any to any flags S/SA modulate state
> pass out on $wan2 proto { udp, icmp } from any to any keep state
>
> # route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
> $ext_if2 and $ext_gw2
> pass out on $wan1 route-to ($wan2 $gw2) from $wan2 to any
> pass out on $wan2 route-to ($wan1 $gw1) from $wan1 to any*
>
> ... and ...
>
> *lan = rl0
> wan1 = nfe0
> wan2 = rl1
> wan1_gw = 173.183.32.254
> wan2_gw = 10.0.1.2
>
> nat on $wan1 from any to any -> ($wan1)
> nat on $wan2 from any to any -> ($wan2)
>
> pass in quick on $lan route-to { ($wan1 $wan1_gw), ($wan2 $wan2_gw) } \
>  round-robin inet from ($lan:network) to any flags S/SA keep state*
>
> Neither of the above worked, or the many other attempts I made.
>
> No errors are reported when I `pfctl -f /etc/pf.lb.conf` and my LAN looses
> internet connectivity.
>
> Does any one see the problem? I can ping Google fine using either WAN as
> default route so it has to be my PF conf.
>
> I am at the point where I will pay someone to get it working!
> --
> ~ Jed Gainer
> _______________________________________________
> freebsd-pf at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
>

Correct me if I'm wrong, but I don't think you can do this without
running a routing protocol with your upstream ISP. The problem is,
regardless of which connection you send your traffic out, the return
traffic will always come the same route from your ISP(s). If you send
your traffic out $wan2 but your IP space is advertised by your ISP on
$wan1 the traffic will always come back in $wan1 and you'll have an
asymmetric route (as well as messed-up states in pf on the $wan1 and
$wan2 interfaces).

The only way I've been able to load-balance outbound traffic is to
have different upstream routers advertise different routes back to my
network via BGP and work the load-balancing that way.


-Proto