Gigabit Ethernet performance with Realtek 8111E

[Michael Laß]

Hi!

I've got a small NAS with Intel D525MW (Atom) board inside using FreeBSD
9.0-RC1 as operating system. It has an onboard Realtek 8111E ethernet
adapter. I'm experiencing heavy performance problems when transfering
files from a specific PC in my network to that NAS. I did the following
tests by transfering large amount of data between the diferrent machines
(using dd and nc):

NAS -> Linux1:    ~ 400Mbit/s
NAS -> Linux2:    ~ 400Mbit/s
Linux1 -> NAS:    heavy fluctuation, between 700Mbit/s and 0bit/s
Linux2 -> NAS:    ~ 400Mbit/s
Linux1 -> Linux2: ~ 400Mbit/s
Linux2 -> Linux1: ~ 400Mbit/s

As you can see everythink works fine except for transfering data from
Linux1 to that NAS box. The following graph shows the problem:
http://dl.dropbox.com/u/25455527/network-problems.png

While the transfer rate drops to zero the NAS also has a very bad ping
up to one second. Ping of Linux1 is perfectly fine during these outages.

I also had a quick look on the data stream with wireshark on Linux1 and
it shows a lot of TCP Dup ACK (up to 263 Dup ACKs created by NAS for one
frame).

What can be eliminated as a cause is:
- Switch (I tried connecting Linux1 and NAS directly)
- Cable (I changed that a few times)
- Harddisk I/O (I'm only writing from /dev/zero to /dev/null)

The sevirity of that problem varies from one minute to another but can
always be reproduced with a few tries.

When limiting either NAS or Linux1 to 100Mbit I'm getting a steady
transfer rate of about 90Mbit/s.
When decreasing the MTU on NAS to 1200 the problem seems to disappear,
getting a transfer rate of about 160Mbit/s.

ifconfig re0:
> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=388b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
> 	ether 38:60:77:3e:af:a5
> 	inet 192.168.178.54 netmask 0xffffff00 broadcast 192.168.178.255
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 	media: Ethernet autoselect (1000baseT <full-duplex>)
> 	status: active

pciconf -lv:
> re0 at pci0:1:0:0:	class=0x020000 card=0xd6258086 chip=0x816810ec rev=0x06 hdr=0x00
>     vendor     = 'Realtek Semiconductor Co., Ltd.'
>     device     = 'RTL8111/8168B PCI Express Gigabit Ethernet controller'
>     class      = network
>     subclass   = ethernet

Because Linux1 seems to be involved in that problem: It's running Linux
3.0 and it has an "Atheros Communications AR8121/AR8113/AR8114" onboard.

Does anyone have an idea what could be the problem here? Decreasing the
MTU is some kind of solution but the performance is still not optimal
and a MTU of 1500 should be no problem.

Greetings,
Michael Laß