PF - load balancing outgoing connections

[Jed Gainer]

I wanted to setup a machine as my LAN gateway and have it load balance over
multiple WANs. When I found http://www.openbsd.org/faq/pf/pools.html I
choose FreeBSD as the machines OS. After getting it up and running, and
acting as a gateway just using one WAN via

*# macros
wan1="nfe0"
lan1="rl0"

pc1="10.0.0.2"
xb1="10.0.0.3"

# options
#set block-policy return
#set loginterface $wan1
set skip on lo0

# scrub
scrub in

# nat/rdr
nat on $wan1 from !($wan1) -> ($wan1:0) static-port

# uTorrent
rdr on $wan1 proto tcp from any to any port 41016 -> $pc1

# Xbox Live
rdr on $wan1 proto {tcp, udp} from any to any port 3074 -> $xb1*

I decided to try the load balancing and came up with quite a few different
pf.confs that did not work, my LAN just lost all connectivity when I loaded
them.
*
lan1r = "10.0.0.0/24"
lan1  = "rl0"
wan1 = "nfe0"
wan2 = "rl1"
gw1 = "10.0.1.2"
gw2 = "10.0.2.2"

# nat outgoing connections on each internet interface
nat on $wan1 from $lan1r to any -> ($wan1) #static-port
nat on $wan2 from $lan1r to any -> ($wan2) #static-port

# default deny
block in from any to any
block out from any to any

# pass all outgoing packets on internal interface
pass out on $lan1 from any to $lan1r

# pass in quick any packets destined for the gateway itself
pass in quick on $lan1 from $lan1r to $lan1

# load balance outgoing tcp traffic from internal network.
pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto
tcp from $lan1r to any flags S/SA modulate state

# load balance outgoing udp and icmp traffic from internal network
pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto {
udp, icmp } from $lan1r to any keep state

# general "pass out" rules for external interfaces
pass out on $wan1 proto tcp from any to any flags S/SA modulate state
pass out on $wan1 proto { udp, icmp } from any to any keep state
pass out on $wan2 proto tcp from any to any flags S/SA modulate state
pass out on $wan2 proto { udp, icmp } from any to any keep state

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
$ext_if2 and $ext_gw2
pass out on $wan1 route-to ($wan2 $gw2) from $wan2 to any
pass out on $wan2 route-to ($wan1 $gw1) from $wan1 to any*

... and ...

*lan = rl0
wan1 = nfe0
wan2 = rl1
wan1_gw = 173.183.32.254
wan2_gw = 10.0.1.2

nat on $wan1 from any to any -> ($wan1)
nat on $wan2 from any to any -> ($wan2)

pass in quick on $lan route-to { ($wan1 $wan1_gw), ($wan2 $wan2_gw) } \
 round-robin inet from ($lan:network) to any flags S/SA keep state*

Neither of the above worked, or the many other attempts I made.

No errors are reported when I `pfctl -f /etc/pf.lb.conf` and my LAN looses
internet connectivity.

Does any one see the problem? I can ping Google fine using either WAN as
default route so it has to be my PF conf.

I am at the point where I will pay someone to get it working!
--
~ Jed Gainer