ARP table entries / ifconfig needs to be issued twice when moving IP

[Michael Gmelin]

Hi,

I'm not sure if it's just me being tired, but I'm facing the following
problem on 10.3-RELEASE when moving an IPv4 alias from one host to
the other. This is an example of what I'm seeing:

Configuration:

Box 1:
ifconfig_bge0_name="uplink"
ifconfig_uplink="inet 10.1.1.253/24 description 'uplink'"
ifconfig_uplink_ipv6="inet6 fd8e:1111:1111:1f::fd"
mac: 14:18:77:11:22:33

Box 2:
ifconfig_bge0_name="uplink"
ifconfig_uplink="inet 10.1.1.254/24 description 'uplink'"
ifconfig_uplink_ipv6="inet6 fd8e:1111:1111:1f::fe"
mac: 14:18:77:44:55:66

Test:

Box 1: Configure alias address
# ifconfig uplink alias 10.1.1.33/32
# arp -an | grep 10.1.1.33
? (10.1.1.33) at 14:18:77:11:22:33 on uplink permanent [ethernet]

Box 2: Ping, check it's in the ARP table
# ping 10.1.1.33
...
# arp -an | grep 10.1.1.33
? (10.1.1.33) at 14:18:77:11:22:33 on uplink expires in 1188 seconds
[ethernet]

Box 1: Remove alias
# ifconfig uplink -alias 10.1.1.33   

Box 2: Add alias
# ifconfig uplink alias 10.1.1.33/32
# arp -an | grep 33
? (10.1.1.33) at 14:18:77:11:22:33 on uplink expires in 1156 seconds
[ethernet]

It's still in the arp table as a non-permanent entry, pointing to the
Box 1.

Box 2: Issue ifconfig once more:
# ifconfig uplink alias 10.1.1.33/32
# arp -an | grep 33
? (10.1.1.33) at 14:18:77:44:55:66 on uplink permanent [ethernet]

Now it's set to the local ethernet address in the arp table
permanently.

This is not a big deal in case of a simple setup (packets still flow
correctly), but as soon as you add multiple FIBs and interface routes
into the mix, things get problematic[1].

Is this intended behaviour, am I missing something?

- Michael

[1] While the example above was done on two separate boxes, the FIB
setup in production is something like this:

sysctl net.add_addr_allfibs=0
vlan1: fib 1
vlan2: fib 2
route add -host 10.1.1.33 -interface vlan2 -fib 1
route add -host 10.2.1.32 -interface vlan1 -fib 2
...

-- 
Michael Gmelin