netif restart looses carp on vlan alias
Stefan Thiel
Stefan.Thiel at flynt.io
Tue Feb 7 12:02:32 UTC 2017
Hi,
I’m an application developer using FreeBSD for gateways on a ESXi 6 VMWare cluster.
After rebooting the gateway all is fine, but upon ‘/etc/rc.d/netif restart’ some of the carp definitions/aliases on the vlan interfaces are lost (different ones but reproducible for each network configuration – see below a list of configurations tried). It looks like ‘netif restart’ takes the carp alias up and then immediately down again (see /var/log/messages below) .
I tried different NIC emulations/drivers (E1000 and VMX3) and
different network configurations:
· 2 active interfaces (1 tagged mgmt. and 1 untagged trunk with just one vlan having a carp alias – current state of /etc/rc.conf.d/network below) => losing the carp alias upon ‘netif restart’
· 5 interfaces (2 tagged ones for mgmt. and internet access and 3 untagged trunk interfaces each having 3 vlans with each having a carp alias (see commented lines in /etc/rc.conf.d/network)) => loosing the FIRST carp alias on the first trunk interface
· 2 interfaces (1 tagged mgmt. and 1 untagged trunk interface with 3 vlans each having a carp alias) => loosing ALL 3 carp aliases upon ‘netif restart’
I have also seen other configurations losing one of multiple carp aliases on 2 different interfaces. But for each configuration the carp alias lost is reproducible and stays the same – different configuration -> different effect (=other alias(es) lost).
Any hints on how to fix or what and how to debug would be greatly appreciated.
Thx
Stef
--- version of the test machine on a ESXi – should be newest stable ---
root at chxnz001:~ # uname -a
FreeBSD chxnz001.scaling.ch 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
root at chxnz001:~ # freebsd-version
11.0-RELEASE-p7
--- /boot/loader.conf ---
# MY SETTINGS
carp_load="YES"
--- /etc/rc.conf ---
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="chxnz001.scaling.ch"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
# MY SETTINGS
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"
gateway_enabled="yes"
--- /etc/rc.conf.d/network ---
ifconfig_vmx0="inet 10.1.1.254/24"
#defaultrouter="10.1.1.1"
# ifconfig_vmx1="inet 10.3.1.254/24"
ifconfig_vmx2="up"
# vlans_vmx2="11 21 31" # trying more complex setups -> effect of losing carp aliases upon netif restart can be even worse … losing them all for specific examples
vlans_vmx2="11"
ifconfig_vmx2_11="inet 10.254.241.21/24"
ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/32 vhid 47 advskew 100 pass test11"
# ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/24 vhid 47 advskew 100 pass test11" # trying different subnet mask on alias -> no improvement
# ifconfig_vmx2_21="inet 10.254.221.21/24"
# ifconfig_vmx2_21_alias0="inet vhid 47 advskew 100 pass test21 alias 10.254.221.1/24"
# ifconfig_vmx2_31="inet 10.254.231.21/24"
# ifconfig_vmx2_31_alias0="inet vhid 47 advskew 100 pass test31 alias 10.254.231.1/24"
# ifconfig_vmx3="up"
# vlans_vmx3="12 22 32"
# ifconfig_vmx3_12="inet 10.254.242.22/24"
# ifconfig_vmx3_12_alias0="inet vhid 47 advskew 100 pass test12 alias 10.254.242.2/24"
# ifconfig_vmx3_22="inet 10.254.222.22/24"
# ifconfig_vmx3_22_alias0="inet vhid 47 advskew 100 pass test22 alias 10.254.222.2/24"
# ifconfig_vmx3_32="inet 10.254.232.22/24"
# ifconfig_vmx3_32_alias0="inet vhid 47 advskew 100 pass test32 alias 10.254.232.2/24"
# ifconfig_vmx4="up"
# vlans_vmx4="19 29 39"
# ifconfig_vmx4_19="inet 10.254.249.29/24"
# ifconfig_vmx4_19_alias0="inet vhid 47 advskew 100 pass test19 alias 10.254.249.9/24"
# ifconfig_vmx4_29="inet 10.254.229.29/24"
# ifconfig_vmx4_29_alias0="inet vhid 47 advskew 100 pass test29 alias 10.254.229.9/24"
# ifconfig_vmx4_39="inet 10.254.239.29/24"
# ifconfig_vmx4_39_alias0="inet vhid 47 advskew 100 pass test39 alias 10.254.239.9/24"
--- note: no other files in /etc/rc.conf.d ---
--- /var/log/messages – for ‘netif restart’ ---
Feb 7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface lo0: 48
Feb 7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx0: 3
Feb 7 08:51:02 chxnz001 kernel: carp: 47 at vmx2.11: BACKUP -> INIT (hardware interface down)
Feb 7 08:51:02 chxnz001 kernel: carp: demoted by 240 to 240 (interface down)
Feb 7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx2.11: 3
Feb 7 08:51:02 chxnz001 last message repeated 3 times
Feb 7 08:51:02 chxnz001 kernel: carp: demoted by -240 to 0 (vhid removed)
Feb 7 08:51:02 chxnz001 kernel: vmx2: promiscuous mode disabled
Feb 7 08:51:02 chxnz001 kernel: vmx2.11: promiscuous mode disabled
Feb 7 08:51:02 chxnz001 admin: /etc/rc.d/netif: WARNING: vmx2.11 does not exist. Skipped.
Feb 7 08:51:02 chxnz001 kernel: arp: 00:50:56:98:1d:9a is using my IP address 10.254.241.21 on vmx2.11!
Feb 7 08:51:02 chxnz001 kernel: vmx2: promiscuous mode enabled
Feb 7 08:51:02 chxnz001 kernel: vmx2.11: promiscuous mode enabled
Feb 7 08:51:02 chxnz001 kernel: carp: 47 at vmx2.11: INIT -> BACKUP (initialization complete)
Feb 7 08:51:03 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx2.11: 3
Feb 7 08:51:03 chxnz001 last message repeated 2 times
Feb 7 08:51:03 chxnz001 kernel: carp: 47 at vmx2.11: BACKUP -> INIT (hardware interface up)
Feb 7 08:51:03 chxnz001 kernel: vmx2: promiscuous mode disabled
Feb 7 08:51:03 chxnz001 kernel: vmx2.11: promiscuous mode disabled
--- ifconfig – ok after reboot ---
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:94:a1:13
inet 10.1.1.254 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:94:88:cf
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:94:17:0a
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:94:06:99
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx4: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:94:ca:65
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
vmx2.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3<RXCSUM,TXCSUM>
ether 00:50:56:94:17:0a
inet 10.254.241.21 netmask 0xffffff00 broadcast 10.254.241.255
inet 10.254.241.1 netmask 0xffffffff broadcast 10.254.241.1 vhid 47
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 11 vlanpcp: 0 parent interface: vmx2
carp: BACKUP vhid 47 advbase 1 advskew 100
groups: vlan
--- ifconfig – NOK after ‘netif restart’ (only snipped that changed – see diff below) – see vmx2.11: missing carp alias ---
. . . <snipped>
vmx2.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3<RXCSUM,TXCSUM>
ether 00:50:56:94:17:0a
inet 10.254.241.21 netmask 0xffffff00 broadcast 10.254.241.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 11 vlanpcp: 0 parent interface: vmx2
groups: vlan
--- ifconfig diff before and after ‘netif restart’ ---
root at chxnz001:~ # diff ifconfig_ok_after_reboot.out ifconfig_nok_after_netif_restart.out
14c14
< vmx2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
---
> vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
39c39
< vmx2.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
---
> vmx2.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
43d42
< inet 10.254.241.1 netmask 0xffffffff broadcast 10.254.241.1 vhid 47
48d46
< carp: BACKUP vhid 47 advbase 1 advskew 100
More information about the freebsd-questions
mailing list