Low Tx-Rx performance with 10Gb NICs

[Axel Fischer]

Hi Igor,

my name is Axel Fischer. working at Marvell SC.

In addition to your reply to my colleague Lino
Sanfilippo I did some performance measurements
on FreeBSD 9 with a commercial 10 GBit network
card.

Unlike on other OS the FreeBSD performance
for duplex rx/tx operation never exceeded the
limit about 9.5 GBit/s. Normally a performance
of at least 16 GBit (up to line speed 20 GBit
in duplex mode) is expected.
As Lino already mentioned the CPU/bus system
(in general the HW) does not set a limit.
Furthermore I noticed that the CPU(s) load
is not very high, about 30 %.

Here is an overview of the measurements:

netperf rx-tx 4 streams / 60s

1768.16 Mb/s Port=2001 RX
999.33 Mb/s Port=2002 RX
72.16 Mb/s Port=1001 TX
61.49 Mb/s Port=1002 TX
2302.76 Mb/s Port=2003 RX
73.48 Mb/s Port=1003 TX
2416.23 Mb/s Port=2004 RX
76.02 Mb/s Port=1004 TX
==== >  RX+TX Total Result: Mb/s 7769.63 


CPU load:

last pid:  1739;  load averages:  0.97,  0.49,  0.21    up 0+00:02:26
11:02:52
46 processes:  2 running, 44 sleeping
CPU 0:  2.0% user,  0.0% nice, 23.2% system,  0.4% interrupt, 74.4% idle
CPU 1:  1.2% user,  0.0% nice, 19.7% system,  0.4% interrupt, 78.7% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system, 80.7% interrupt, 19.3% idle
CPU 3:  0.0% user,  0.0% nice,  0.8% system,  1.6% interrupt, 97.6% idle
CPU 4:  2.4% user,  0.0% nice, 25.6% system,  0.0% interrupt, 72.0% idle
CPU 5:  3.1% user,  0.0% nice, 25.6% system,  0.0% interrupt, 71.3% idle
CPU 6:  0.0% user,  0.0% nice,  0.4% system, 32.7% interrupt, 66.9% idle
CPU 7:  0.4% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.0% idle
Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free
Swap: 4096M Total, 4096M Free

Additionally I noticed the following TCP errors
with netstat -s ...:

1186 data packets (1717328 bytes) retransmitted
6847875 window update packets
2319 duplicate acks
25831 out-of-order packets (37403288 bytes)
3733 discarded due to memory problems (drops)
1186 segment rexmits in SACK recovery episodes
1717328 byte rexmits in SACK recovery episodes


My questions: 

- What is the max. performance (duplex) on
  FreeBSD 9 that you have measured with a 10 GBit
  NIC ? 
  (Expected > 16 GBit/s on appropriate HW)

Thank you in advance,
Axel


-------- Weitergeleitete Nachricht --------
Von: Igor Mozolevsky <igor at hybrid-lab.co.uk>
An: Lino Sanfilippo <lsanfil at marvell.com>
Kopie: Hackers freeBSD <freebsd-hackers at freebsd.org>, Axel Fischer
<afischer at marvell.com>, Ralf Assmann <rassmann at marvell.com>, Markus
Althoff <malthoff at marvell.com>
Betreff: Re: Low Tx-Rx performance with 10Gb NICs
Datum: Thu, 23 May 2013 11:21:08 -0700

On 23 May 2013 19:00, Lino Sanfilippo <lsanfil at marvell.com> wrote:

> Is there a known issue concerning high traffic on Tx and Rx paths?  Are there any system
> settings I could adjust to get the expected performance? Any hints are very appreciated.

check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting
ierrs on em chips, but none on fxp chips (connected to the same
wire/switch); might be unrelated to yours, but worth a check...



-- 
Axel Fischer | R&D Software / SW Engineer | Marvell Semiconductor
Germany GmbH
Office +49 (7243) 502 370 | Fax +49 (7243) 502 982
afischer at marvell.com

M A R V E L L |www.marvell.com

This communication, together with any attachments hereto or links
contained herein, is for the sole use of the intended recipient(s) and
may contain information that is confidential or legally protected. If
you are not the intended recipient, you are hereby notified that any
review, disclosure, copying, dissemination, distribution or use of this
communication is STRICTLY PROHIBITED. If you have received this
communication in error, please notify the sender immediately by return
e-mail message and delete the original and all copies of the
communication, along with any attachments hereto or links herein, from
your system.

Marvell Semiconductor Germany GmbH, Siemensstr. 23, 76275 Ettlingen,
Amtsgericht Mannheim HRB 361620

Geschäftsführer: Dipl.-Volksw. Mathias Horak