Understanding the interplay of ipfw, vlan, and carp

[Freddie Cash]

On March 4, 2008 03:25 pm Freddie Cash wrote:
> On March 4, 2008 02:20 pm Max Laier wrote:
> > Am Di, 4.03.2008, 22:51, schrieb Freddie Cash:
> > ...
> >
> > > The lack of a "carpdev" option to directly link a carp device to an
> > > interface (similar to "vlandev" for vlan(4)) is what's really
> > > tripping me up.  It appears the carp(4) driver looks at all the
> > > interfaces in the box to find one with an IP in the same subnet as
> > > the carp IP and then uses that as the physical device.
> >
> > You could try the attached patch.  It adds carpdev support.  You'll
> > have to recompile ifconfig to make use of it.
> >
> > This patch has some shortcomings that I wanted to address for a long
> > time now, but never found the time to do so.  Mostly that IPv6 over
> > CARP is broken with this patch.  Everything else is supposed to work
> > and I'd like to hear if you experience otherwise (success stories
> > welcome, too).  This is from back in early January, but should apply
> > to RELENG_7 and HEAD w/o too much trouble.

Patch applied cleanly to RELENG_7.0.  However, there are a few strange 
things happening now.

If there are IPs on the physical devices (em0|em1) things only seem to 
work if my ipfw rules allow traffic over em0|em1.  If there are no IPs on 
em0|em1, then the ipfw rules work fine using carp0|carp1.  But it's not 
consistent.  Sometimes the counters for the em rules increment and 
sometimes the counters for the carp rules increment.

If there are no IPs on the physical devices, and I configure rc.conf to 
put two IPs onto carp0 (one with /24, one with /32) it loses the route 
for the /24, can't find the default router, and traffic doesn't go 
through.  Manually adding the route via "route add -net 
192.168.0.0/24 -iface carp0" allows traffic to flow again.

The rc.conf entries are:
  cloned_interfaces="carp0 carp2"
  ifconfig_em0="up"
  ifconfig_em2="up"
  ifconfig_carp0="carpdev em0 vhid 100 pass whatever  192.168.0.11/24"
  ifconfig_carp0_alias0="192.168.0.10/32"
  ifconfig_carp2="carpdev em2 vhid 102 pass whatever2 172.20.0/1/24"



I only upgraded one of my test boxes to RELENG_7_0.  The other is still 
RELENG_6_3.  They no longer stay in sync.  Even though 
net.inet.carp.preempt=1 is set on both boxes, only the interface that I 
pull the plug on or manually down will fail-over to the other box.

The ifconfig ouput on the 6.3 box will show (unplug em2 on the 6.3 box):
carp0: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
        inet 192.168.0.11 netmask 0xffffff00
        inet 192.168.0.10 netmask 0xffffffff
        carp: MASTER vhid 100 advbase 1 advskew 150
carp2: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
        inet 172.20.0.1 netmask 0xffffff00
        carp: BACKUP vhid 102 advbase 1 advskew 150

And the ifconfig output on the 7.0 box will show:
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
1500
        ether 00:00:5e:00:01:64
        inet 192.168.0.10 netmask 0xffffffff
        inet 192.168.0.11 netmask 0xffffff00
        carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
1500
        ether 00:00:5e:00:01:66
        inet 172.20.0.1 netmask 0xffffff00
        carp: MASTER carpdev em2 vhid 102 advbase 1 advskew 0



And, finally, if I try to create two carp devices using the same physical 
device, with IPs in the same subnet, the box crashes.  The first time, it 
locked up with the kernel panic.  Every other time it just locks the box.

The commands to do this are reproducable:
  ifconfig em0 up
  ifconfig carp0 create
  ifconfig carp0 carpdev em0 vhid 1 192.168.0.1/24
  ifconfig carp1 create
  ifconfig carp1 carpdev em0 vhid 2 192.168.0.2/24

It will complain once that it can't assign the requested address.  If you 
try the ifconfig command again, the box locks up.  Might take two or 
three tries if you're lucky.  :)

-- 
Freddie Cash
fjwcash at gmail.com