NFS over LAGG / lacp poor performance

Marek Salwerowicz marek_sal at wp.pl
Fri Apr 25 07:48:54 UTC 2014 

Hi list,

I have two FreeBSD boxes (both based on SuperMicro X9DRD-7LN4F-JBOD
motherboard,  with 32GB RAM, 1 CPU :Intel(R) Xeon(R) CPU E5-2640 v2)

storage1% uname -a                                         
FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan 12
20:11:23 UTC 2014    
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


storage2% uname
-a                                                                  
FreeBSD storage2 10.0-RELEASE-p1 FreeBSD 10.0-RELEASE-p1 #0: Tue Apr  8
06:45:06 UTC 2014    
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64



They work as NFS storages for VMware ESXi.

On both there are installed 4 igb 1Gbit NICs, with LACP and 2 VLANs
(vlan900 is for NFS, vlan14 is for management/general purpose):

on storage1:
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at
device 0.0 on pci5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at
device 0.1 on pci5
igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at
device 0.2 on pci5
igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at
device 0.3 on pci5

on storage2:
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at
device 0.0 on pci5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at
device 0.1 on pci5
igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at
device 0.2 on pci5
igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at
device 0.3 on pci5


storage1% ifconfig -a                      
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:c1:1d:18
        inet6 fe80::225:90ff:fec1:1d18%igb0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:c1:1d:18
        inet6 fe80::225:90ff:fec1:1d19%igb1 prefixlen 64 scopeid 0x2
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:c1:1d:18
        inet6 fe80::225:90ff:fec1:1d1a%igb2 prefixlen 64 scopeid 0x3
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:c1:1d:18
        inet6 fe80::225:90ff:fec1:1d1b%igb3 prefixlen 64 scopeid 0x4
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:c1:1d:18
        inet6 fe80::225:90ff:fec1:1d18%lagg0 prefixlen 64 scopeid 0x8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:25:90:c1:1d:18
        inet 192.168.1.65 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::225:90ff:fec1:1d18%vlan14 prefixlen 64 scopeid 0x9
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 14 parent interface: lagg0
vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
        options=103<RXCSUM,TXCSUM,TSO4>
        ether 00:25:90:c1:1d:18
        inet 172.25.25.65 netmask 0xffffff00 broadcast 172.25.25.255
        inet6 fe80::225:90ff:fec1:1d18%vlan900 prefixlen 64 scopeid 0xa
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 900 parent interface: lagg0


storage2% ifconfig
-a                                                              
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:ca:3b:e0
        inet6 fe80::225:90ff:feca:3be0%igb0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:ca:3b:e0
        inet6 fe80::225:90ff:feca:3be1%igb1 prefixlen 64 scopeid 0x2
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:ca:3b:e0
        inet6 fe80::225:90ff:feca:3be2%igb2 prefixlen 64 scopeid 0x3
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:ca:3b:e0
        inet6 fe80::225:90ff:feca:3be3%igb3 prefixlen 64 scopeid 0x4
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:ca:3b:e0
        inet6 fe80::225:90ff:feca:3be0%lagg0 prefixlen 64 scopeid 0x6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:25:90:ca:3b:e0
        inet 192.168.1.72 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::225:90ff:feca:3be0%vlan14 prefixlen 64 scopeid 0x7
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 14 parent interface: lagg0
vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:25:90:ca:3b:e0
        inet 172.25.25.72 netmask 0xffffff00 broadcast 172.25.25.255
        inet6 fe80::225:90ff:feca:3be0%vlan900 prefixlen 64 scopeid 0x8
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 900 parent interface: lagg0


Both boxes are  connected to the same switch (HP 1910-48G)

I need to transfer around 10 TB of data from storage1 to storage2
I obeserve that during copying, only one NIC (instead of all 4) is used.

Boxes are not stressed during copying

What's more, apart from having 1 NIC saturated (transfer around 120
MB/s), I observe transfer rate on level of 70-80 MB/s

I haven't changed any kernel configuration, nor sysctl's - am I missing
sth ?

Regards,
Marek

-- 
Marek Salwerowicz

More information about the freebsd-net mailing list