Understanding the interplay of ipfw, vlan, and carp

[Freddie Cash]

On March 5, 2008 12:09 pm you wrote:
> Am Mi, 5.03.2008, 20:39, schrieb Freddie Cash:
> > On March 4, 2008 03:25 pm Freddie Cash wrote:
> > Patch applied cleanly to RELENG_7.0.  However, there are a few
> > strange things happening now.
> >
> > If there are IPs on the physical devices (em0|em1) things only seem
> > to work if my ipfw rules allow traffic over em0|em1.  If there are no
> > IPs on em0|em1, then the ipfw rules work fine using carp0|carp1.  But
> > it's not consistent.  Sometimes the counters for the em rules
> > increment and sometimes the counters for the carp rules increment.
>
> I'll look into this ... it would help if you could qualify "it's not
> consistent" a bit, so that I can reproduce.

I'll have to run some more tests on this to try and narrow things down, 
and make sure I'm actually seeing what I think I'm seeing.

This might just be me misunderstanding how the network stack works, and 
how a packet travels through the physical interfaces, through the virtual 
interfaces, and through the packet filter.

> > The rc.conf entries are:
> >   cloned_interfaces="carp0 carp2"
> >   ifconfig_em0="up"
> >   ifconfig_em2="up"
> >   ifconfig_carp0="carpdev em0 vhid 100 pass whatever 
> > 192.168.0.11/24"
> >   ifconfig_carp0_alias0="192.168.0.10/32" 
> >   ifconfig_carp2="carpdev em2 vhid 102 pass whatever2 172.20.0/1/24"
> >
> > I only upgraded one of my test boxes to RELENG_7_0.  The other is
> > still RELENG_6_3.  They no longer stay in sync.  Even though
> > net.inet.carp.preempt=1 is set on both boxes, only the interface that
> > I pull the plug on or manually down will fail-over to the other box.
> >
> > The ifconfig ouput on the 6.3 box will show (unplug em2 on the 6.3
> > box): carp0: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
> >         inet 192.168.0.11 netmask 0xffffff00
> >         inet 192.168.0.10 netmask 0xffffffff
> >         carp: MASTER vhid 100 advbase 1 advskew 150
> > carp2: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
> >         inet 172.20.0.1 netmask 0xffffff00
> >         carp: BACKUP vhid 102 advbase 1 advskew 150
> >
> > And the ifconfig output on the 7.0 box will show:
> > carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> > mtu 1500
> >         ether 00:00:5e:00:01:64
> >         inet 192.168.0.10 netmask 0xffffffff
> >         inet 192.168.0.11 netmask 0xffffff00
> >         carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
> > carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> > mtu 1500
> >         ether 00:00:5e:00:01:66
> >         inet 172.20.0.1 netmask 0xffffff00
> >         carp: MASTER carpdev em2 vhid 102 advbase 1 advskew 0
>
> What does "netstat -ssp carp" say?  It seems that vhid 100 doesn't sync
> at all.  Might be a problem with the order of the address list.

FreeBSD 6.3 box:
carp:
        1649 packets received (IPv4)
                1649 discarded for bad authentication
        6871 packets sent (IPv4)

FreeBSD 7.0 box:
carp:
	1138 packets received (IPv4)
		1138 discarded for bad authentication
	1797 packets sent (IPv4)

The rc.conf entries from the 6.3 box:
ifconfig_carp0="vhid 100 pass nexus-carp-pass  advskew 150 
192.168.0.11/24"

"ifconfig carp0" lists 192.168.0.11/24 first and 192.168.0.10/32 second.


The rc.conf entry from the 7.0 box:
ifconfig_carp0="carpdev em0 vhid 100 pass nexus-carp-pass 192.168.0.11/24"

"ifconfig carp0" lists 192.168.0.10/32 first and 192.168.0.11/24 second.


If I create the carp devices in the exact same order on each box, using 
the exact same commands (but with carpdev added on the 7.0 box), with 
only 1 IP on each carp interface, then things almost work.  If I down 
carp2 on the 7.0 box, carp2 on the 6.3 box becomes the master, but carp0 
remains as BACKUP on the 6.3 box.  And vice versa when I down carp0 on 
the 7.0 box.

Changing the advskew option on the 7.0 box to be 200 causes both carp 
devices switch.  6.3 becomes master and 7.0 becomes backup.  BUT, downing 
one interface still only causes that one to failover.  
net.inet.carp_preempt is still set to 1 on both boxes.

If I create two IPs on the carp interface, even if created in the exact 
same order on box boxes, then they won't failover at all.  Both boxes 
show all the carp interfaces set to MASTER.  And the discarded counters 
in "netstat -ssp carp" increment on both boxes every second.


Thanks for your help on this.  If needed, I can upgrade the other 6.3 box 
to 7.0.

-- 
Freddie Cash
fjwcash at gmail.com