Bug/Intentional issue with asymmetric routing?

[Andy Coates]

Hi all,

About to pull my hair out debugging this problem, which I'm left
believing is either a bug or intentional (but I can't find any
references to the behaviour).


            |--- fw1 ---|  
server  ----|  (pfsync) |---- transit isp1
            |--- fw2 -- |  

I'm using CARP on the server LAN side so it always has a gateway
(fw1/fw2) to go though, but because there are multiple internal subnets
involved I'm using OSPF on the transit router.

The transit server sees two next-hop's for server's LAN, fw1 and fw2
(not their CARP address, their interface IPs).  In this case we presume
fw1 is the next-hop.

If fw1 is carp master there are no issues, packets follow:
  server->fw1->internet->fw1->server

If fw2 is carp master the issue occurs - TCP sessions fail:
  server->fw2->internet->fw1->server

At this point if I disabled PF on fw1 everything is fine.  If I enable
PF on fw1, but leave pf.conf blank so no rules, TCP connections fail.
Confirmed no rules with 'pfctl -s rules' and nothing listed.  Even added
'pass all no state' just in case had a default block, but still fails.

I can't work out why enabling PF is breaking TCP sessions.

Am I missing something obvious?

Running 8.0-STABLE with the GENERIC kernel on AMD64.


Thanks,
Andy.