FreeBSD7, pf, carp...

[Giuseppe Maniscalco]

Update...

Its a strange day, today...
Yesterday I changed the nodes states, so now the master is the old
backup. And it works fine...
The new backup (the nodes that give me problems) seems to be ok... but...

The last rule of my pf ruleset is for load balancing between external
connections ...
[...]
pass out log on $ext_if1 route-to ($ext_if2 $ext_if2_gw) from $ext_if2 to any
pass out log on $ext_if2 route-to ($ext_if1 $ext_if1_gw) from $ext_if1 to any

Now I'm running out of ideas I checked all the pf rules with pfctl -sa
and... Arrrggh!...

pass out log on em2 route-to (axe0 213.x.x.254) inet from 212.52.82.27
to any flags S/SA keep state

WHAT!?! the hell and who is 212.52.82.27??? It doesn't belong to my IP
addresses!?

After a flush it becomes normal:

pass out log on em2 route-to (axe0 213.x.x.254) inet from 213.x.x.249
to any flags S/SA keep state

What do you think about that?!




2011/3/13 Giuseppe Maniscalco <peppe.maniscalco at gmail.com>:
> Hi List! I need your help!!!
>
> I've two firewalls configured in parallel (connected with a crossover
> cable) and I use pfsync+carp to failover. So one firewall (A) handles
> all traffic as MASTER and, if it dies or if some NIC interface go
> down, the second firewall (B) takes over automatically.
> Well... As usually everything works properly, but since a few days ago
> "B" takes control and "A" become backup. But "A" cannot return to be
> master until rebooting!
> After reboot, "A" is the master for a while, then I've the same problem...
>
> I identified a problem here:
>
> fwA# sysctl -a | grep arp
> net.inet.ip.same_prefix_carp_only: 0
> net.inet.carp.allow: 1
> net.inet.carp.preempt: 1
> net.inet.carp.log: 1
> net.inet.carp.arpbalance: 0
> net.inet.carp.suppress_preempt: 1
>
> From man carp:
> net.inet.carp.suppress_preempt:
> A read only value showing the status of preemption suppression.
> Preemption can be suppressed if link on an interface is down or when
> pfsync(4) interface is not synchronized.
> Value of 0 means that preemption is not suppressed, since no problems
> are detected. Every problem increments suppression counter.
>
> All my interfaces are UP... now I don't know how to check if pfsync is
> synched or not...
>
> Meanwhile, in B node:
> fwB# sysctl -a | grep arp
> net.inet.ip.same_prefix_carp_only: 0
> net.inet.carp.allow: 1
> net.inet.carp.preempt: 1
> net.inet.carp.log: 1
> net.inet.carp.arpbalance: 0
> net.inet.carp.suppress_preempt: 0
>
> I tried with a tcpdump on the interfaces, but I see just the change of
> condition (master/backup) with the advskew modification...
> This is the only strange thing on DMZ interface... :
>
> 17:01:32.397429 01:80:c2:00:00:01 (oui Unknown) > 01:80:c2:00:00:01
> (oui Unknown), ethertype Unknown (0x8808), length 60:
> 0x0000: 0001 ffff 0000 0000 0000 0000 0000 0000 ................
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
>
> I just tried to change the NIC, but nothing! "A" continue to lose
> control in 30/45 minutes...
>
> I read somewhere that the result of "pfctl -ss" must give the same
> result on both nodes:
>
> fwA# pfctl -ss | wc -l
> 5833
> fwB# pfctl -ss | wc -l
> 5507
>
> Could it be important?
>
>
> Some additional information:
>
> fwA# more /etc/rc.conf
> ifconfig_em0="inet a.a.a.12 netmask 255.255.255.0 polling" ### DMZ ###
> ifconfig_em1="inet b.b.b.2 netmask 255.255.0.0 polling" ### CROSSOVER ###
> ifconfig_em2="inet c.c.c.189 netmask 255.255.255.224 polling" ### ISP1 ###
> ifconfig_em3="inet d.d.d.249 netmask 255.255.255.0 polling" ### ISP2 ###
> defaultrouter="c.c.c.1"
>
> #Firewall
> pf_enable="YES"
> pf_rules="/etc/pf.conf"
> pf_flags=""
> pflog_enable="YES"
> pflog_logfile="/var/log/pflog"
>
> #Failover
> pfsync_enable="YES"
> pfsync_syncdev="em1"
> cloned_interfaces="carp0 carp1 carp2"
> ifconfig_carp0="a.a.a.1/24 vhid 1 pass foo"
> ifconfig_carp0_alias0="a.a.a.11/24 vhid 1 pass foo"
> ifconfig_carp1="d.d.d.14/24 vhid 2 pass bar"
> ifconfig_carp1_alias0="d.d.d.2/24 vhid 2 pass bar"
> ifconfig_carp2="c.c.c.188/27 vhid 3 pass jack"
> ifconfig_carp2_alias0="c.c.c.165/27 vhid 3 pass jack"
>
> fwB# more /etc/rc.conf
> ifconfig_ste0="inet a.a.a.13 netmask 255.255.255.0 polling"
> ifconfig_ste1="inet b.b.b.3 netmask 255.255.0.0 polling"
> ifconfig_em0="inet c.c.c.190 netmask 255.255.255.224 polling"
> ifconfig_em1="inet d.d.d.250 netmask 255.255.255.0 polling"
> defaultrouter="c.c.c.1"
>
> #Firewall
> pf_enable="YES"
> pf_rules="/etc/pf.conf"
> pf_flags=""
> pflog_enable="YES"
> pflog_logfile="/var/log/pflog"
>
> #Failover
> pfsync_enable="YES"
> pfsync_syncdev="ste1"
> cloned_interfaces="carp0 carp1 carp2"
> ifconfig_carp0="a.a.a.1/24 vhid 1 advskew 128 pass foo"
> ifconfig_carp0_alias0="a.a.a.11/24 vhid 1 advskew 128 pass foo"
> ifconfig_carp1="d.d.d.14/24 vhid 2 advskew 64 pass bar"
> ifconfig_carp1_alias0="d.d.d.2/24 vhid 2 advskew 64 pass bar"
> ifconfig_carp2="c.c.c.188/27 vhid 3 advskew 100 pass jack"
> ifconfig_carp2_alias0="c.c.c.165/27 vhid 3 advskew 100 pass jack"
>
> In each node pf.conf I added:
>
> fwA# more pf.conf | grep failover
> pass quick on { em1 } proto pfsync # failover
> pass on { em0 em2 em3 } proto carp # failover
>
> fwB# more pf.conf | grep failover
> pass quick on { ste1 } proto pfsync # failover
> pass on { em0 ste0 em1 } proto carp # failover
>
>
> I hope that someone can give me a solution please, or maybe just an
> idea, cause I'm getting crazy!!!
> Please ask me, if you need further information...
> Thank you all!
>