Problem using Carp with NAT for High Availability Firewall

[Craig Cocca]

I have been experimenting recently with using Carp on FreeBSD 6.1 to implement a high-availability firewall.  I have two FreeBSD 6.1 machines set up, each with their own static IP address, and both machines share a virtual IP (VIP), which is the gateway IP for the machines behind the firewalls.  My network topology looks like this:

                    Internet
                     Switch
                       |
       |--------------------------------|
Firewall 1                     Firewall 2
10.0.0.1                      10.0.0.2
             192.168.0.1 (VIP)
|-------------------------|-------------------|
Server 1         Server 2        Server N


I have been successful in getting the two firewall machines set up so that the slave machine takes over the VIP from the master if the master machine loses connectivity.  However, when the master comes back online and takes over the VIP again, I'm noticing something really odd, namely that traffic starts going to the master again but ends up getting "swallowed alive" by the kernel.  

In other words, I can have one of the machines behind the firewalls sending out a ping to a host on the Internet when the slave is servicing the VIP, and I will see traffic on Firewall 2's (slave's) inside and outside interfaces.  As soon as the master comes online and takes over the VIP from the slave again, I see the traffic switch to the inside interface of the master (I see this by watching tcpdump), but I don't see the traffic getting routed to the outside interface!  Either I am doing something wrong, or there is some kind of bug in Carp.  Can anyone shed some light on this?  One other interesting thing to add to the mystery is that if I wait exactly 15 minutes from when the master takes back over the VIP, the traffic starts getting routed again.

Thanks,

Craig