Unstable local network throughput

[Ben RUBSON]

> On 02 Aug 2016, at 21:35, Hans Petter Selasky <hps at selasky.org> wrote:
> 
> Hi,

Thank you for your answer Hans Petter !

> The CX-3 driver doesn't bind the worker threads to specific CPU cores by default, so if your CPU has more than one so-called numa, you'll end up that the bottle-neck is the high-speed link between the CPU cores and not the card. A quick and dirty workaround is to "cpuset" iperf and the interrupt and taskqueue threads to specific CPU cores.

My CPUs : 2x E5-2620v3 with DDR4 at 1866.
What is strange is that even without using the card (iPerf on localhost), as my results show, I have very low and unstable random throughput (compared to Linux on the same host).

> Are you using "options RSS" and "options PCBGROUP" in your kernel config?

I only installed FreeBSD 10.3 and updated it, so I use the GENERIC kernel.
RSS and PCBGROUP are not defined in /usr/src/sys/amd64/conf/GENERIC, so I think I do not use them.

> Are you also testing CX-4 cards from Mellanox?

No, I only have CX-3 at my disposal :)

Ben

PS : in my previous mail I sometimes used GB/s, of course you must read Gb/s everywhere.