[Bug 221146] [ixgbe] Problem with second laggport

Wed Apr 10 18:49:46 UTC 2019

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221146

Johan Ström <johan at stromnet.se> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |johan at stromnet.se

--- Comment #26 from Johan Ström <johan at stromnet.se> ---
Hi,

not sure if this is fully related, but I've had issues with carp and lagg too.
If changing the carp status, i.e. plugging in one of the configured interfaces
(that was how I first noticed it), then the lagg0 interface went down and up,
but carp failed to catch up on this. This happens on reboot too (but not sure
it has happened *every time*, this is a new setup).

The net.inet.carp.demotion counter went to 2160 ( /240 = 9, which is the number
of VLANs with CARP on the lagg), but got stuck there and never came back down
to 0:

Apr 10 20:23:00 gw1 kernel: ix1: link state changed to UP
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 240 (send error 50 on
vlan14)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 480 (send error 50 on
vlan11)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 720 (send error 50 on
vlan17)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 960 (send error 50 on
vlan16)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 1200 (send error 50 on
vlan13)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 1440 (send error 50 on
vlan15)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 1680 (send error 50 on
vlan10)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 1920 (send error 50 on
vlan1)
Apr 10 20:23:07 gw1 kernel: carp: demoted by 240 to 2160 (send error 50 on
vlan18)
Apr 10 20:23:08 gw1 kernel: carp: 10 at vlan10: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan10: 3
Apr 10 20:23:08 gw1 kernel: carp: 13 at vlan13: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan13: 3
Apr 10 20:23:08 gw1 kernel: carp: 5 at vlan15: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan15: 3
Apr 10 20:23:08 gw1 kernel: carp: 18 at vlan18: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan18: 3
Apr 10 20:23:08 gw1 kernel: carp: 11 at vlan11: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: arp: 172.28.2.1 moved from 00:22:4d:6b:b1:5b to
00:00:5e:00:01:0b on vlan11
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan11: 3
Apr 10 20:23:08 gw1 kernel: carp: 17 at vlan17: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan17: 3
Apr 10 20:23:08 gw1 kernel: carp: 1 at vlan1: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan1: 3
Apr 10 20:23:08 gw1 kernel: carp: 14 at vlan14: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan14: 3
Apr 10 20:23:08 gw1 kernel: carp: 16 at vlan16: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:23:08 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan16: 3

Then I manually removed the demote via sysctl:

Apr 10 20:23:21 gw1 kernel: carp: demoted by -2160 to 0 (sysctl)
Apr 10 20:23:23 gw1 kernel: carp: 11 at vlan11: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 17 at vlan17: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 14 at vlan14: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 1 at vlan1: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 13 at vlan13: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 10 at vlan10: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: arp: 172.28.4.129 moved from 00:00:5e:00:01:11 to
00:22:4d:6b:b1:5b on vlan17
Apr 10 20:23:23 gw1 kernel: carp: 5 at vlan15: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 16 at vlan16: BACKUP -> MASTER (preempting a
slower master)
Apr 10 20:23:23 gw1 kernel: carp: 18 at vlan18: BACKUP -> MASTER (preempting a
slower master)

(Also a bit interesting that it mentions those ARP changes..  Why would either
of the nodes announce the CARPed IP on the nic mac rather than the CARP ip, at
any time?)

The "other" carp node (not using lagg) is 11.2-RELEASE-p7, this node with lagg
is 11.2-RELEASE-p8.
The lagg'ed nic's are ix0-ix4 "<Intel(R) PRO/10GbE PCI-Express Network Driver,
Version - 3.2.12-k> mem 0xddc00000-0xdddfffff,0xdde04000-0xdde07fff at device
0.0 on pci6" on a Supermicro A2SDi-4C-HLN4F.

On both nodes, net.inet.carp.preempt=1, and advbase 1, advskew 100 on this
node, 200 on the other.

To add another dimension to this. If I set net.inet.carp.preempt=0 (which I had
initially), I cannot get the interfaces out of BACKUP at all:
...
Apr 10 20:45:23 gw1 kernel: carp: 1 at vlan1: MASTER -> BACKUP (more frequent
advertisement received)
Apr 10 20:45:23 gw1 kernel: ifa_maintain_loopback_route: deletion failed for
interface vlan1: 3
Apr 10 20:45:36 gw1 kernel: carp: demoted by -2160 to 0 (sysctl)

and then nothing more.. Setting it to 1 again, immediately makes it master.

Anyway, not sure if this is related to the ixgbe 3.2.12-k driver, or lagg, or
carp..  But I thought I'd write it down here anyway.

-- 
You are receiving this mail because:
You are the assignee for the bug.