kern/132107: carp(4) advskew setting ignored when carp IP used on a gif(4) interface

[Andrew Daugherity]

>Number:         132107
>Category:       kern
>Synopsis:       carp(4) advskew setting ignored when carp IP used on a gif(4) interface
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 25 19:00:09 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Andrew Daugherity
>Release:        7.1
>Organization:
Texas A&M University
>Environment:
FreeBSD imslb4.tamu.edu 7.1-RELEASE-p1 FreeBSD 7.1-RELEASE-p1 #0: Wed Jan 14 15:03:53 CST 2009     root at imslb4.tamu.edu:/usr/obj/usr/src/sys/IMS  amd64
>Description:
For background, see my post to the freebsd-pf mailing list:
http://docs.freebsd.org/cgi/mid.cgi?38ce25da0901271659m2b6d8a1fg2e425df93781f6f5

When an IP configured on a carp(4) interface is used on a gif(4) tunnel, in order that the endpoint of the tunnel fails over, the advskew setting for this carp interface is ignored and the interface is brought up with advskew 0.  This obviously prevents proper operation of a "preferred master" configuration.

After configuring devd to ignore attach events for carp interfaces (devd automatically runs "/etc/rc.d/netif start carp0" after you do an "ifconfig carp0 create", and does so needlessly after carp0 is already configured at boot time; that's a separate bug I shall file shortly), I have determined that the "start all interfaces" mode of /etc/rc.d/netif (i.e. "rc.d/netif start", without specifying an interface) configures the advskew correctly.  It's the "rc.d/netif start carp0" mode that loses the advskew setting.

In the standard configuration, "rc.d/netif start" is called and carp interfaces are brought up (correctly).  A bit later in the bootup, devd starts, and thinks it needs to configure carpN, so it calls "rc.d/netif start carp0", which resets advskew to 0.

I'm thinking it's a kernel bug since enabling rc_debug and logging the outputs of "sh -x /etc/rc.d/netif start" vs. "ifconfig carp0 create; sh -x /etc/rc.d/netif start carp0" shows that the exact same sequence of "ifconfig carp0 ..." commands is run.  The only difference is that in the global start mode, other interfaces (including gif0) are ifconfig'd between "ifconfig carp0 create" and the rest of the "ifconfig carp0 ..." commands.

I have observed this problem on both 7.1/amd64 and 6.2/i386.
>How-To-Repeat:
Assign the same IP to a carp interface and gif interface, e.g.:
cloned_interfaces="carp0 carp1"
gif_interfaces="gif0"
carp0_advskew="advskew 100"
ifconfig_carp0="vhid 16 $carp_advskew pass XXXXXX 10.95.8.1/21"
ifconfig_gif0="10.95.8.1 10.95.0.1 netmask 255.255.255.255"

The name "carp0" is nothing special; I can call it carp3 and the behavior is the same.  Only the 'ifconfig_gif0' setting matters; the gifconfig_gif0 setting doesn't (an IP I used for that is configured on carp1, which has no problems).  Also, the mere presence of gif0 doesn't matter -- when gif0 is configured with IPs that are NOT assigned to a carp interface, carp0 comes up with the correct advskew setting.
>Fix:
A workaround is to configure devd to ignore attach events for carp interfaces, like so:
====
# don't have devd call network_start on carp0; for some reason this
#   loses the advskew setting (besides being entirely superfluous).

notify 100 {
	match "system" "IFNET";
	match "subsystem" "carp[0-9]+";
	match "type"	"ATTACH";
	action "echo devd called on $subsystem, type $type";
};
====
This overrides the following section of devd.conf:
notify 0 {
        match "system"          "IFNET";
        match "type"            "ATTACH";
        action "/etc/pccard_ether $subsystem start";
};


After doing this, carp interfaces are brought up correctly by "rc.d/netif start" on boot, and devd does not call "rc.d/netif start carp0", so the advskew setting remains correct.  This doesn't fix the underlying problem but it is an acceptable workaround to me.

>Release-Note:
>Audit-Trail:
>Unformatted: