Low Tx-Rx performance with 10Gb NICs

[Andre Oppermann]

On 24.05.2013 16:46, Axel Fischer wrote:
> Hi Igor,
>
> my name is Axel Fischer. working at Marvell SC.

Hi Axel,

> In addition to your reply to my colleague Lino
> Sanfilippo I did some performance measurements
> on FreeBSD 9 with a commercial 10 GBit network
> card.

Which driver?

> Unlike on other OS the FreeBSD performance
> for duplex rx/tx operation never exceeded the
> limit about 9.5 GBit/s. Normally a performance
> of at least 16 GBit (up to line speed 20 GBit
> in duplex mode) is expected.
> As Lino already mentioned the CPU/bus system
> (in general the HW) does not set a limit.
> Furthermore I noticed that the CPU(s) load
> is not very high, about 30 %.

Your RX/TX is probably serialized in the driver and you can only
make use of one core.

> Here is an overview of the measurements:
>
> netperf rx-tx 4 streams / 60s
>
> 1768.16 Mb/s Port=2001 RX
> 999.33 Mb/s Port=2002 RX
> 72.16 Mb/s Port=1001 TX
> 61.49 Mb/s Port=1002 TX
> 2302.76 Mb/s Port=2003 RX
> 73.48 Mb/s Port=1003 TX
> 2416.23 Mb/s Port=2004 RX
> 76.02 Mb/s Port=1004 TX
> ==== >  RX+TX Total Result: Mb/s 7769.63
>
>
> CPU load:
>
> last pid:  1739;  load averages:  0.97,  0.49,  0.21    up 0+00:02:26
> 11:02:52
> 46 processes:  2 running, 44 sleeping
> CPU 0:  2.0% user,  0.0% nice, 23.2% system,  0.4% interrupt, 74.4% idle
> CPU 1:  1.2% user,  0.0% nice, 19.7% system,  0.4% interrupt, 78.7% idle
> CPU 2:  0.0% user,  0.0% nice,  0.0% system, 80.7% interrupt, 19.3% idle
> CPU 3:  0.0% user,  0.0% nice,  0.8% system,  1.6% interrupt, 97.6% idle
> CPU 4:  2.4% user,  0.0% nice, 25.6% system,  0.0% interrupt, 72.0% idle
> CPU 5:  3.1% user,  0.0% nice, 25.6% system,  0.0% interrupt, 71.3% idle
> CPU 6:  0.0% user,  0.0% nice,  0.4% system, 32.7% interrupt, 66.9% idle
> CPU 7:  0.4% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.0% idle
> Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free
> Swap: 4096M Total, 4096M Free
>
> Additionally I noticed the following TCP errors
> with netstat -s ...:
>
> 1186 data packets (1717328 bytes) retransmitted

This may happen and is typically not cause for concern on
a loaded system.

> 6847875 window update packets

Normal.

> 2319 duplicate acks

Related to the retransmits.

> 25831 out-of-order packets (37403288 bytes)

This is unusual.  What kind of test setup do you have,
back-to-back cards or a switch in between?  Out of order
normally shouldn't happen unless over the internet.

> 3733 discarded due to memory problems (drops)
> 1186 segment rexmits in SACK recovery episodes
> 1717328 byte rexmits in SACK recovery episodes

Again related to retransmits.

> My questions:
>
> - What is the max. performance (duplex) on
>    FreeBSD 9 that you have measured with a 10 GBit
>    NIC ?
>    (Expected > 16 GBit/s on appropriate HW)

Certain large CDN are known to push more than 20Gbit/s production
traffic per machine.

Please also my other message from today to hackers@ "Re: preemptive
kernel" with Message-ID: <51A4B991.3070805 at freebsd.org>.

-- 
Andre


> Thank you in advance,
> Axel
>
>
> -------- Weitergeleitete Nachricht --------
> Von: Igor Mozolevsky <igor at hybrid-lab.co.uk>
> An: Lino Sanfilippo <lsanfil at marvell.com>
> Kopie: Hackers freeBSD <freebsd-hackers at freebsd.org>, Axel Fischer
> <afischer at marvell.com>, Ralf Assmann <rassmann at marvell.com>, Markus
> Althoff <malthoff at marvell.com>
> Betreff: Re: Low Tx-Rx performance with 10Gb NICs
> Datum: Thu, 23 May 2013 11:21:08 -0700
>
> On 23 May 2013 19:00, Lino Sanfilippo <lsanfil at marvell.com> wrote:
>
>> Is there a known issue concerning high traffic on Tx and Rx paths?  Are there any system
>> settings I could adjust to get the expected performance? Any hints are very appreciated.
>
> check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting
> ierrs on em chips, but none on fxp chips (connected to the same
> wire/switch); might be unrelated to yours, but worth a check...
>
>
>