pfsync & carp problems

[Douglas K. Rand]

Doug> I'm testing a new set of firewalls using pfsync and carp to replace an
Doug> existing IP Filter firewall and I'm having occasional problems with
Doug> TCP sessions failing over.

Some more information after I discovered the -x loud option to
pfctl. When the master firewall goes down and the already established
TCP session hangs, I get these messages on the slave:

pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=21109:24835 dir=in,rev
pf: State failure on: 1       |

And after the master comes up, I see these on the master:

pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=0:0 dir=in,rev
pf: State failure on: 1       |
pf: BAD state: TCP 67.134.74.224:52173 67.134.74.224:52173 204.152.184.134:80 [lo=2943781408 high=2943846943 win=33304 modulator=0 wscale=1] [lo=3255565389 high=3255629101 win=65535 modulator=0 wscale=0] 4:4 A seq=3255634893 ack=2943781408 len=1448 ackskew=0 pkts=0:0 dir=in,rev
pf: State failure on: 1       |

The state table on the master includes:

self tcp 67.134.74.224:52173 -> 204.152.184.134:80       TIME_WAIT:TIME_WAIT
   [2943781408 + 65535] wscale 1  [3255565389 + 63712] wscale 0
   age 00:08:29, expires in 00:00:48, 0:1 pkts, 0:40 bytes
self tcp 204.152.184.134:80 <- 67.134.74.224:52173       TIME_WAIT:TIME_WAIT
   [3255565389 + 65160] wscale 0  [2943781408 + 65535] wscale 1
   age 00:08:30, expires in 00:00:48, 0:1 pkts, 0:40 bytes

And the slave has:

self tcp 67.134.74.224:52173 -> 204.152.184.134:80       ESTABLISHED:ESTABLISHED
   [2943781408 + 65535] wscale 1  [3255565389 + 63712] wscale 0
   age 00:07:10, expires in 23:56:40, 21109:24835 pkts, 1100808:37201523 bytes
self tcp 204.152.184.134:80 <- 67.134.74.224:52173       ESTABLISHED:ESTABLISHED
   [3255565389 + 65160] wscale 0  [2943781408 + 65535] wscale 1
   age 00:07:10, expires in 23:56:40, 21109:24835 pkts, 1100808:37201523 bytes