pfsync & carp problems

[Douglas K. Rand]

I'm testing a new set of firewalls using pfsync and carp to replace an
existing IP Filter firewall and I'm having occasional problems with
TCP sessions failing over. More often than not the fail over works
fine, but some times when I reboot the master firewall the TCP session
hangs, and when the backup firewall transfers from MASTER to BACKUP
the session stays hung.

The state exists on both firewalls right after the master comes back:

master# pfctl -v -s state
[...]
self tcp 67.134.74.224:58786 -> 204.152.184.134:80       ESTABLISHED:ESTABLISHED
   [69234942 + 65535] wscale 1  [1597172605 + 63712] wscale 0
   age 00:07:37, expires in 23:59:10, 0:0 pkts, 0:0 bytes
self tcp 204.152.184.134:80 <- 67.134.74.224:58786       ESTABLISHED:ESTABLISHED
   [1597172605 + 63712] wscale 0  [69234942 + 65535] wscale 1
   age 00:07:37, expires in 23:59:02, 0:0 pkts, 0:0 bytes

[...]

slave# pfctl -v -s state
[...]
self tcp 67.134.74.224:58786 -> 204.152.184.134:80       ESTABLISHED:ESTABLISHED
   [69234942 + 65535] wscale 1  [1597172605 + 63712] wscale 0
   age 00:07:01, expires in 23:57:54, 19885:23629 pkts, 1037055:35439120 bytes, rule 187
self tcp 204.152.184.134:80 <- 67.134.74.224:58786       ESTABLISHED:ESTABLISHED
   [1597172605 + 63712] wscale 0  [69234942 + 65535] wscale 1
   age 00:07:01, expires in 23:57:54, 19885:23629 pkts, 1037055:35439120 bytes, rule 187
[...]


But after a few minutes the state goes away on both firewalls. Both
systems are running FreeBSD 6.1-p2.