Unbalanced LACP link
Vitalii Duk
mlevel.ars at gmail.com
Mon Mar 16 09:37:18 UTC 2015
I've changed use_flowid to 0 and it helped! But isn't it setting
significant? In a description it says "Shift flowid bits to prevent
multiqueue collisions".
On 16 March 2015 at 09:50, Konstantin Kulikov <k.kulikov2 at gmail.com> wrote:
> Hello, Vitalii.
>
> Make sure following sysctl are set to zero:
> sysctl net.link.lagg.default_use_flowid=0
> sysctl net.link.lagg.0.use_flowid=0
> sysctl net.link.lagg.1.use_flowid=0
>
> Then adjust lagghash depending on your configuration.
> For example if you freebsd machine acts as router with single, default,
> route,
> you should set lagghash to l3, since for l2 both src and dst MACs will
> always be same and you'd get unbalanced load.
>
> On Sun, Mar 15, 2015 at 1:38 PM, Vitalii Duk <mlevel.ars at gmail.com> wrote:
> > Hello, guys.
> > I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link
> > DGS-3610-26G switch. This link worked fine last two years on FreeBSD
> > 9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic
> > started to go mostly through one interface.
> >
> > lagg0 - input interface (no IP-address, only VLANs).
> > lagg1 - output interface.
> >
> > [router]# ifconfig lagg0
> > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> >
> > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
> > ether 90:e2:ba:02:d0:ae
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet autoselect
> > status: active
> > laggproto lacp lagghash l2
> > laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> > laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> >
> > [#router]# ifconfig lagg1
> > lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> >
> > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
> > ether 00:1b:21:93:52:d4
> > inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet autoselect
> > status: active
> > laggproto lacp lagghash l2
> > laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> > laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> >
> > [router]# netstat -hw1 -I igb0
> > input igb0 output
> > packets errs idrops bytes packets errs bytes colls
> > 27K 0 0 12M 5 0 350 0
> > 26K 0 0 12M 7 0 693 0
> > 29K 0 0 12M 8 0 838 0
> > 29K 0 0 11M 10 0 1.0K 0
> > 27K 0 0 10M 7 0 770 0
> >
> > As you can see most traffic is going through igb1 (lagg0) and igb3
> (lagg1)
> > interfaces. In peak times there are more that 1 Gbps of traffic mostly
> > through one 1GbE interface, and it cause an errors. So this link is
> > assymetric.
> >
> > [router]# netstat -hw1 -I igb1
> > input igb1 output
> > packets errs idrops bytes packets errs bytes colls
> > 22K 0 0 8.5M 63K 0 69M 0
> > 20K 0 0 7.7M 58K 0 62M 0
> > 20K 0 0 7.3M 57K 0 61M 0
> > 19K 0 0 6.8M 56K 0 61M 0
> > 18K 0 0 6.5M 57K 0 62M 0
> >
> > [router]# netstat -hw1 -I igb2
> > input igb2 output
> > packets errs idrops bytes packets errs bytes colls
> > 40K 0 0 49M 1 0 93 0
> > 41K 0 0 51M 1 0 88 0
> > 38K 0 0 45M 1 0 88 0
> > 33K 0 0 39M 2 0 174 0
> > 31K 0 0 35M 1 0 89 0
> >
> > [router]# netstat -hw1 -I igb3
> > input igb3 output
> > packets errs idrops bytes packets errs bytes colls
> > 32K 0 0 34M 44K 0 18M 0
> > 29K 0 0 30M 45K 0 18M 0
> > 32K 0 0 34M 48K 0 17M 0
> > 28K 0 0 29M 42K 0 15M 0
> > 25K 0 0 26M 40K 0 16M 0
> >
> > [router]# netstat -hw1 -I lagg0
> > input lagg0 output
> > packets errs idrops bytes packets errs bytes colls
> > 45K 0 0 17M 62K 0 69M 0
> > 46K 0 0 17M 66K 0 75M 0
> > 43K 0 0 17M 62K 0 69M 0
> > 50K 0 0 19M 71K 1 82M 0
> > 46K 0 0 18M 66K 0 75M 0
> >
> > [router]# netstat -hw1 -I lagg1
> > input lagg1 output
> > packets errs idrops bytes packets errs bytes colls
> > 70K 0 0 79M 49K 0 18M 0
> > 71K 0 0 79M 49K 0 19M 0
> > 79K 0 0 91M 54K 0 20M 0
> > 65K 0 0 72M 45K 0 17M 0
> > 65K 0 0 72M 45K 0 17M 0
> >
> > All igb interfaces are the same:
> > igb0 at pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086
> > rev=0x01 hdr=0x00
> > vendor = 'Intel Corporation'
> > device = '82576 Gigabit Network Connection'
> > class = network
> > subclass = ethernet
> >
> > I've tried to disable/enable strict_mode, put switch into active/passive
> > mode and trying to set duplex/speed manyally. Nothing helps.
> > Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because
> > switch supports only l2 load-balancing, but it also doesn't helps.
> >
> > net.link.lagg.lacp.debug: 0
> > net.link.lagg.0.lacp.lacp_strict_mode: 0
> > net.link.lagg.0.lacp.debug.rx_test: 0
> > net.link.lagg.0.lacp.debug.tx_test: 0
> > net.link.lagg.1.lacp.lacp_strict_mode: 0
> > net.link.lagg.1.lacp.debug.rx_test: 0
> > net.link.lagg.1.lacp.debug.tx_test: 0
> >
> > I've read that there were some problems with LACP on FreeBSD 10.
> >
> > Information from the switch:
> >
> > # show lacp summary
> > Aggregate port 6:
> >
> > Local information:
> > LACP port Oper Port Port
> > Port Flags State Priority Key Number State
> > ----------------------------------------------------------------------
> > Gi0/18 SP bndl 4096 0x6 0x12 0x3c
> > Gi0/22 SP bndl 4096 0x6 0x16 0x3c
> >
> > Partner information:
> > LACP port Oper Port Port
> > Port Flags Priority Dev ID Key Number State
> > ---------------------------------------------------------------------
> > Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d
> > Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d
> >
> > Aggregate port 7:
> >
> > Local information:
> > LACP port Oper Port Port
> > Port Flags State Priority Key Number State
> > ----------------------------------------------------------------------
> > Gi0/19 SP bndl 4096 0x7 0x13 0x3c
> > Gi0/21 SP bndl 4096 0x7 0x15 0x3c
> >
> > Partner information:
> > LACP port Oper Port Port
> > Port Flags Priority Dev ID Key Number State
> > ---------------------------------------------------------------------
> > Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d
> > Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d
> > _______________________________________________
> > freebsd-net at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list