netif restart on VMWare loses carp

[Stefan Thiel]

<originally posted on freebsd-questions, but I think freebsd-net is appropriate>
>    Hi, 
>    
>    I’m an application developer using FreeBSD for gateways on a ESXi 6 VMWare cluster.
>    
>    After rebooting the gateway all is fine, but upon ‘/etc/rc.d/netif restart’ some of the carp definitions/aliases on the vlan interfaces are lost (different ones but reproducible for each network configuration – see below a list of configurations tried). It looks like ‘netif restart’ takes the carp alias up and then immediately down again (see /var/log/messages below).
>    
>    I tried different NIC emulations/drivers (E1000 and VMX3) and 
>    different network configurations:
>    • 2 active interfaces (1 tagged mgmt. and 1 untagged trunk with just one vlan having a carp alias – current state of /etc/rc.conf.d/network below) => losing the carp alias upon ‘netif restart’
>    • 5 interfaces (2 tagged ones for mgmt. and internet access and 3 untagged trunk interfaces each having 3 vlans with each having a carp alias (see commented lines in /etc/rc.conf.d/network)) => losing the FIRST carp alias on the first trunk interface 
>    • 2 interfaces (1 tagged mgmt. and 1 untagged trunk interface with 3 vlans each having a carp alias) => loosing ALL 3 carp aliases upon ‘netif restart’
>    
>    I have also seen other configurations losing one of multiple carp aliases on 2 different interfaces. But for each configuration the carp alias lost is reproducible and stays the same – different configuration -> different effect (= other alias(es) lost).
>    

Meanwhile I tested a similar config on hardware (a pc engine box, that was formerly used for a pf sense installation), where this effect WAS NOT reproducible.

Back on VMware: Also replacing open-vm-tools with the original ones didn't help losing carp upon '/etc/rc.d/netif restart'

Whether the interfaces are connected or disconnected doesn't matter. (VMWare: virtually disconnected, Hardware: physically disconnected)

So far the summary:
- seems to be VMWare (currently version 6 - newest would be 6.5) related only
- changing NIC emulation type, drivers or network connection state didn't help

this issue is still OPEN and I'm lacking ideas how to proceed.

>    Any hints on how to fix or what and how to debug would be greatly appreciated.
>    
>    Thx 
>    Stef
>    
>    --- version of the test machine on a ESXi – should be newest stable --- 
>    root at chxnz001:~ # uname -a
>    FreeBSD chxnz001.scaling.ch 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016     root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
>    root at chxnz001:~ # freebsd-version
>    11.0-RELEASE-p7
>    
>    --- /boot/loader.conf --- 
>    # MY SETTINGS 
>    carp_load="YES"
>    
>    --- /etc/rc.conf --- 
>    clear_tmp_enable="YES"
>    syslogd_flags="-ss"
>    sendmail_enable="NONE"
>    hostname="chxnz001.scaling.ch"
>    sshd_enable="YES"
>    # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
>    dumpdev="NO"
>    # MY SETTINGS
>    vmware_guest_vmmemctl_enable="YES"
>    vmware_guest_vmxnet_enable="YES"
>    vmware_guestd_enable="YES"
>    gateway_enabled="yes"
>    
>    --- /etc/rc.conf.d/network --- 
>    ifconfig_vmx0="inet 10.1.1.254/24"
>    #defaultrouter="10.1.1.1"
>    # ifconfig_vmx1="inet 10.3.1.254/24"
>    ifconfig_vmx2="up"
>    # vlans_vmx2="11 21 31" # trying more complex setups -> effect of losing carp aliases upon netif restart can be even worse … losing them all for specific examples
>    vlans_vmx2="11"
>    ifconfig_vmx2_11="inet 10.254.241.21/24"
>    ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/32 vhid 47 advskew 100 pass test11"
>    # ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/24 vhid 47 advskew 100 pass test11"  # trying different subnet mask on alias -> no improvement
>    # ifconfig_vmx2_21="inet 10.254.221.21/24"
>    # ifconfig_vmx2_21_alias0="inet vhid 47 advskew 100 pass test21 alias 10.254.221.1/24"
>    # ifconfig_vmx2_31="inet 10.254.231.21/24"
>    # ifconfig_vmx2_31_alias0="inet vhid 47 advskew 100 pass test31 alias 10.254.231.1/24"
>    # ifconfig_vmx3="up"
>    # vlans_vmx3="12 22 32"
>    # ifconfig_vmx3_12="inet 10.254.242.22/24"
>    # ifconfig_vmx3_12_alias0="inet vhid 47 advskew 100 pass test12 alias 10.254.242.2/24"
>    # ifconfig_vmx3_22="inet 10.254.222.22/24"
>    # ifconfig_vmx3_22_alias0="inet vhid 47 advskew 100 pass test22 alias 10.254.222.2/24"
>    # ifconfig_vmx3_32="inet 10.254.232.22/24"
>    # ifconfig_vmx3_32_alias0="inet vhid 47 advskew 100 pass test32 alias 10.254.232.2/24"
>    # ifconfig_vmx4="up"
>    # vlans_vmx4="19 29 39"
>    # ifconfig_vmx4_19="inet 10.254.249.29/24"
>    # ifconfig_vmx4_19_alias0="inet vhid 47 advskew 100 pass test19 alias 10.254.249.9/24"
>    # ifconfig_vmx4_29="inet 10.254.229.29/24"
>    # ifconfig_vmx4_29_alias0="inet vhid 47 advskew 100 pass test29 alias 10.254.229.9/24"
>    # ifconfig_vmx4_39="inet 10.254.239.29/24"
>    # ifconfig_vmx4_39_alias0="inet vhid 47 advskew 100 pass test39 alias 10.254.239.9/24"
>    
>    --- note: no other files in /etc/rc.conf.d --- 
>