Horrendous upload network performance with VLAN (download seems OK)

[Patrick Proniewski]

Hello,

I'm running FreeBSD 9.3-RELEASE up-to-date, on two HP Proliant G6 server blades in the same enclosure. One with VLANs in the uplink, the other without VLANs. They use bxe driver.

	bxe0: <QLogic NetXtreme II BCM57711E 10GbE (A0) BXE v:1.78.78
	> mem 0xfb000000-0xfb7fffff,0xfa800000-0xfaffffff irq 28 at device 0.0 on pci2
	bxe0: PCI BAR0 [10] memory allocated: 0xfb000000-0xfb7fffff (8388608) -> 0xfffffe00fb000000
	bxe0: PCI BAR2 [18] memory allocated: 0xfa800000-0xfaffffff (8388608) -> 0xfffffe00fa800000
	bxe0: Found 10GBase-CX4 media.
	bxe0: Ethernet address: 00:17:a4:77:04:10
	bxe0: MSI-X vectors Requested 5 and Allocated 5
	../..
	(and so on up to bxe7)

Blade A is configured to access the network through a connection without VLANs (the link provided by Network team comes with no VLAN tagging). Its transfer rate is perfect, both up and down.

	ifconfig_bxe0="inet x.y.z.141/24"
	defaultrouter="x.y.z.1"

	# ifconfig bxe0
	bxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	    options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
	    ether 00:17:a4:77:04:00
	    inet x.y.z.141 netmask 0xffffff00 broadcast x.y.z.255
	    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	    media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
	    status: active


Blade B is configured to access the network through a link sporting multiple VLANs, so I've created a network interface that uses one of these VLANs. Ping is OK, I can ssh to this server, transfer rate to the server (down) is not fantastic but OK, enough to perform pkg installation or FreeBSD update. Transfer rate from the server to the rest of the world is abysmal, often stalling after few 100's KB.

	ifconfig_bxe0="UP"
	vlans_bxe0="161"
	ifconfig_bxe0_161="inet x.y.z.142/24"
	defaultrouter="x.y.z.1"

	# ifconfig bxe0
	bxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	    options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
	    ether 00:17:a4:77:04:10
	    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	    media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
	    status: active

	# ifconfig bxe0.161
	bxe0.161: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	    options=303<RXCSUM,TXCSUM,TSO4,TSO6>
	    ether 00:17:a4:77:04:10
	    inet x.y.z.142 netmask 0xffffff00 broadcast x.y.z.255
	    inet6 fe80::217:a4ff:fe77:410%bxe0.161 prefixlen 64 scopeid 0x10
	    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	    media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
	    status: active
	    vlan: 161 parent interface: bxe0


The same switch is used by those two blade servers running FreeBSD, and by about 14 other blade servers running VMware ESXi 5.x.
ESXi blades use multiple VLANs and work perfectly. A blade running FreeBSD with no VLAN in its uplink (i.e. not sharing the same uplink as ESXi blades) has very good network performances. The only blade with problem is the one running FreeBSD and sharing the uplink of ESXi blades.

scp of a 347MB file from an ESXi blade server to FreeBSD blade (same switch, same blade chassis, same uplink):

	#    scp esxi11.domain.tld:/tardisks/FILE /dev/null
	Password:
	FILE                                   100%  347MB  38.6MB/s   00:09  

scp of a 347MB file from an FreeBSD blade to ESXi blade (same switch, same blade chassis, same uplink):

	#    scp FILE  esxi11.domain.tld:/dev/null
	Password:
	FILE                                     0%  400KB 172.2KB/s - stalled -

traceroute from FreeBSD blade to ESXi blade:

	#    traceroute esxi11.domain.tld
	traceroute to esxi11.domain.tld (x.y.z.151), 64 hops max, 52 byte packets
	1  * * *
	2  * * *
	^C

traceroute from ESXi blade to FreeBSD blade:

	# traceroute freebsdB.domain.tld
	traceroute to freebsdB.domain.tld (x.y.z.142), 30 hops max, 40 byte packets
	1  freebsdB (x.y.z.142)  0.123 ms  0.090 ms  0.078 ms

I'm quite lost here.
Any idea that would explain the problem, and help me resolve it?

Original thread posted on <https://forums.freebsd.org/threads/horrendous-network-performance-with-vlan.49211/>

Patrick