Unstable local network throughput

[Ben RUBSON]

> On 11 Aug 2016, at 19:51, Ben RUBSON <ben.rubson at gmail.com> wrote:
> 
> 
>> On 11 Aug 2016, at 18:36, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>> 
>> Hi!
> 
> Hi Adrian,
> 
>> mlx4_core0: <mlx4_core> mem
>> 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0
>> numa-domain 1 on pci16
>> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6
>> (Aug 11 2016)
>> 
>> so the NIC is in numa-domain 1. Try pinning the worker threads to
>> numa-domain 1 when you run the test:
>> 
>> numactl -l first-touch-rr -m 1 -c 1 ./test-program
> 
> # numactl -l first-touch-rr -m 1 -c 1 /usr/local/bin/iperf -c 192.168.2.1 -l 128KB -P 16 -i 2 -t 6000  
> Could not parse policy: '128KB'
> 
> I did not manage to give arguments to command. Any idea ?

I answer to myself, this should do the trick :
numactl -l first-touch-rr -m 1 -c 1 -- /usr/local/bin/iperf -c 192.168.2.1 -l 128KB -P 16 -i 2 -t 6000

However of course it still gives the error below :

> # numactl -l first-touch-rr -m 1 -c 1 /usr/local/bin/iperf                                           
> numactl: numa_setaffinity: Invalid argument
> 
> And sounds like -m is not allowed with first-touch-rr.
> What should I use ?
> 
> Thank you !
> 
>> You can also try pinning the NIC threads to numa-domain 1 versus 0 (so
>> the second set of CPUs, not the first set.)
>> 
>> vmstat -ia | grep mlx (get the list of interrupt thread ids)
>> then for each:
>> 
>> cpuset -d 1 -x <irq id>
>> 
>> Run pcm-memory.x each time so we can see the before and after effects
>> on local versus remote memory access.
>> 
>> Thanks!
>> 
>> 
>> 
>> -adrian
>