Unstable local network throughput

[Adrian Chadd]

Hi!

mlx4_core0: <mlx4_core> mem
0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0
numa-domain 1 on pci16
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6
(Aug 11 2016)

so the NIC is in numa-domain 1. Try pinning the worker threads to
numa-domain 1 when you run the test:

numactl -l first-touch-rr -m 1 -c 1 ./test-program

You can also try pinning the NIC threads to numa-domain 1 versus 0 (so
the second set of CPUs, not the first set.)

vmstat -ia | grep mlx (get the list of interrupt thread ids)
then for each:

cpuset -d 1 -x <irq id>

Run pcm-memory.x each time so we can see the before and after effects
on local versus remote memory access.

Thanks!



-adrian