netif restart looses carp on vlan alias

Tue Feb 7 13:53:16 UTC 2017

<Sorry the first version was encoded, probably due to its HTML formatting – so this try is plain text>

Hi,

I’m an application developer using FreeBSD for gateways on a ESXi 6 VMWare cluster.

After rebooting the gateway all is fine, but upon ‘/etc/rc.d/netif restart’ some of the carp definitions/aliases on the vlan interfaces are lost (different ones but reproducible for each network configuration – see below a list of configurations tried). It looks like ‘netif restart’ takes the carp alias up and then immediately down again (see /var/log/messages below) .

I tried different NIC emulations/drivers (E1000 and VMX3) and 
different network configurations:
• 2 active interfaces (1 tagged mgmt. and 1 untagged trunk with just one vlan having a carp alias – current state of /etc/rc.conf.d/network below) => losing the carp alias upon ‘netif restart’
• 5 interfaces (2 tagged ones for mgmt. and internet access and 3 untagged trunk interfaces each having 3 vlans with each having a carp alias (see commented lines in /etc/rc.conf.d/network)) => loosing the FIRST carp alias on the first trunk interface 
• 2 interfaces (1 tagged mgmt. and 1 untagged trunk interface with 3 vlans each having a carp alias) => loosing ALL 3 carp aliases upon ‘netif restart’

I have also seen other configurations losing one of multiple carp aliases on 2 different interfaces. But for each configuration the carp alias lost is reproducible and stays the same – different configuration -> different effect (=other alias(es) lost).

Any hints on how to fix or what and how to debug would be greatly appreciated.

Thx
Stef

--- version of the test machine on a ESXi – should be newest stable ---
root at chxnz001:~ # uname -a
FreeBSD chxnz001.scaling.ch 11.0-RELEASE-p2 FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016     root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
root at chxnz001:~ # freebsd-version
11.0-RELEASE-p7

--- /boot/loader.conf ---
# MY SETTINGS 
carp_load="YES"

--- /etc/rc.conf ---
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="chxnz001.scaling.ch"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
# MY SETTINGS
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"
gateway_enabled="yes"

--- /etc/rc.conf.d/network ---
ifconfig_vmx0="inet 10.1.1.254/24"
#defaultrouter="10.1.1.1"
# ifconfig_vmx1="inet 10.3.1.254/24"
ifconfig_vmx2="up"
# vlans_vmx2="11 21 31" # trying more complex setups -> effect of losing carp aliases upon netif restart can be even worse … losing them all for specific examples
vlans_vmx2="11"
ifconfig_vmx2_11="inet 10.254.241.21/24"
ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/32 vhid 47 advskew 100 pass test11"
# ifconfig_vmx2_11_alias0="inet alias 10.254.241.1/24 vhid 47 advskew 100 pass test11"  # trying different subnet mask on alias -> no improvement
# ifconfig_vmx2_21="inet 10.254.221.21/24"
# ifconfig_vmx2_21_alias0="inet vhid 47 advskew 100 pass test21 alias 10.254.221.1/24"
# ifconfig_vmx2_31="inet 10.254.231.21/24"
# ifconfig_vmx2_31_alias0="inet vhid 47 advskew 100 pass test31 alias 10.254.231.1/24"
# ifconfig_vmx3="up"
# vlans_vmx3="12 22 32"
# ifconfig_vmx3_12="inet 10.254.242.22/24"
# ifconfig_vmx3_12_alias0="inet vhid 47 advskew 100 pass test12 alias 10.254.242.2/24"
# ifconfig_vmx3_22="inet 10.254.222.22/24"
# ifconfig_vmx3_22_alias0="inet vhid 47 advskew 100 pass test22 alias 10.254.222.2/24"
# ifconfig_vmx3_32="inet 10.254.232.22/24"
# ifconfig_vmx3_32_alias0="inet vhid 47 advskew 100 pass test32 alias 10.254.232.2/24"
# ifconfig_vmx4="up"
# vlans_vmx4="19 29 39"
# ifconfig_vmx4_19="inet 10.254.249.29/24"
# ifconfig_vmx4_19_alias0="inet vhid 47 advskew 100 pass test19 alias 10.254.249.9/24"
# ifconfig_vmx4_29="inet 10.254.229.29/24"
# ifconfig_vmx4_29_alias0="inet vhid 47 advskew 100 pass test29 alias 10.254.229.9/24"
# ifconfig_vmx4_39="inet 10.254.239.29/24"
# ifconfig_vmx4_39_alias0="inet vhid 47 advskew 100 pass test39 alias 10.254.239.9/24"

--- note: no other files in /etc/rc.conf.d ---

--- /var/log/messages – for ‘netif restart’ ---
Feb  7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface lo0: 48
Feb  7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx0: 3
Feb  7 08:51:02 chxnz001 kernel: carp: 47 at vmx2.11: BACKUP -> INIT (hardware interface down)
Feb  7 08:51:02 chxnz001 kernel: carp: demoted by 240 to 240 (interface down)
Feb  7 08:51:02 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx2.11: 3
Feb  7 08:51:02 chxnz001 last message repeated 3 times
Feb  7 08:51:02 chxnz001 kernel: carp: demoted by -240 to 0 (vhid removed)
Feb  7 08:51:02 chxnz001 kernel: vmx2: promiscuous mode disabled
Feb  7 08:51:02 chxnz001 kernel: vmx2.11: promiscuous mode disabled
Feb  7 08:51:02 chxnz001 admin: /etc/rc.d/netif: WARNING: vmx2.11 does not exist.  Skipped.
Feb  7 08:51:02 chxnz001 kernel: arp: 00:50:56:98:1d:9a is using my IP address 10.254.241.21 on vmx2.11!
Feb  7 08:51:02 chxnz001 kernel: vmx2: promiscuous mode enabled
Feb  7 08:51:02 chxnz001 kernel: vmx2.11: promiscuous mode enabled
Feb  7 08:51:02 chxnz001 kernel: carp: 47 at vmx2.11: INIT -> BACKUP (initialization complete)
Feb  7 08:51:03 chxnz001 kernel: ifa_maintain_loopback_route: deletion failed for interface vmx2.11: 3
Feb  7 08:51:03 chxnz001 last message repeated 2 times
Feb  7 08:51:03 chxnz001 kernel: carp: 47 at vmx2.11: BACKUP -> INIT (hardware interface up)
Feb  7 08:51:03 chxnz001 kernel: vmx2: promiscuous mode disabled
Feb  7 08:51:03 chxnz001 kernel: vmx2.11: promiscuous mode disabled

--- ifconfig – ok after reboot ---
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:50:56:94:a1:13
        inet 10.1.1.254 netmask 0xffffff00 broadcast 10.1.1.255 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
vmx1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:50:56:94:88:cf
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
vmx2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:50:56:94:17:0a
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
vmx3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:50:56:94:06:99
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
vmx4: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:50:56:94:ca:65
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo 
vmx2.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:50:56:94:17:0a
        inet 10.254.241.21 netmask 0xffffff00 broadcast 10.254.241.255 
        inet 10.254.241.1 netmask 0xffffffff broadcast 10.254.241.1 vhid 47 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 11 vlanpcp: 0 parent interface: vmx2
        carp: BACKUP vhid 47 advbase 1 advskew 100
        groups: vlan 

--- ifconfig – NOK after ‘netif restart’ (only snipped that changed – see diff below) – see vmx2.11: missing carp alias ---
. . . <snipped>
vmx2.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:50:56:94:17:0a
        inet 10.254.241.21 netmask 0xffffff00 broadcast 10.254.241.255 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 11 vlanpcp: 0 parent interface: vmx2
        groups: vlan 

--- ifconfig diff before and after ‘netif restart’ ---
root at chxnz001:~ # diff ifconfig_ok_after_reboot.out ifconfig_nok_after_netif_restart.out 
14c14
< vmx2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
---
> vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
39c39
< vmx2.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
---
> vmx2.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
43d42
<       inet 10.254.241.1 netmask 0xffffffff broadcast 10.254.241.1 vhid 47 
48d46
<       carp: BACKUP vhid 47 advbase 1 advskew 100