Unstable local network throughput

[Hans Petter Selasky]

On 08/02/16 20:43, Ben RUBSON wrote:
> Hello,
>
> I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter.
>
> FreeBSD 10.3 just installed, last updates performed.
> Network adapters running last firmwares / last drivers.
> No workload at all, just iPerf as the benchmark tool.
>

Hi,

The CX-3 driver doesn't bind the worker threads to specific CPU cores by 
default, so if your CPU has more than one so-called numa, you'll end up 
that the bottle-neck is the high-speed link between the CPU cores and 
not the card. A quick and dirty workaround is to "cpuset" iperf and the 
interrupt and taskqueue threads to specific CPU cores.

Are you using "options RSS" and "options PCBGROUP" in your kernel config?

Are you also testing CX-4 cards from Mellanox?

--HPS