Unstable local network throughput

[Ben RUBSON]

> On 11 Aug 2016, at 18:36, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> 
> Hi!
> 
> mlx4_core0: <mlx4_core> mem
> 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0
> numa-domain 1 on pci16
> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6
> (Aug 11 2016)
> 
> so the NIC is in numa-domain 1. Try pinning the worker threads to
> numa-domain 1 when you run the test:
> 
> numactl -l first-touch-rr -m 1 -c 1 ./test-program
> 
> You can also try pinning the NIC threads to numa-domain 1 versus 0 (so
> the second set of CPUs, not the first set.)
> 
> vmstat -ia | grep mlx (get the list of interrupt thread ids)
> then for each:
> 
> cpuset -d 1 -x <irq id>
> 
> Run pcm-memory.x each time so we can see the before and after effects
> on local versus remote memory access.
> 
> Thanks!

Adrian, here are the results :



Idle system :
http://pastebin.com/raw/K1iMVHVF



No pinning :
http://pastebin.com/raw/w5KuexQ3
CPU : http://pastebin.com/raw/8zgRaazN

numactl -l fixed-domain-rr -m 1 -c 1 :
http://pastebin.com/raw/VWweYF9H
CPU : http://pastebin.com/raw/QjaVH32X

numactl -l fixed-domain-rr -m 0 -c 0 :
http://pastebin.com/raw/71hfGJdw
CPU : http://pastebin.com/raw/hef058Na

numactl -l fixed-domain-rr -m 1 -c 1
+ cpuset -l <DOM1_CPU> -x <IRQ> :
http://pastebin.com/raw/nEQkgMK2
CPU : http://pastebin.com/raw/R652KAdJ

numactl -l fixed-domain-rr -m 0 -c 0
+ cpuset -l <DOM0_CPU> -x <IRQ> :
http://pastebin.com/raw/GdYJHyae
CPU : http://pastebin.com/raw/Ggfx9uF9



No pinning, default kernel (no NUMA option) :
http://pastebin.com/raw/iQ2u8d8k
CPU : http://pastebin.com/raw/Xr77KpcM

default kernel (no NUMA option)
+ cpuset -l <DOM0_CPU>
+ cpuset -l <DOM0_CPU> -x <IRQ> :
http://pastebin.com/raw/VBWg4SZs

default kernel (no NUMA option)
+ cpuset -l <DOM1_CPU>
+ cpuset -l <DOM1_CPU> -x <IRQ> :
http://pastebin.com/raw/SrJLZxuT



No pinning, default kernel (no NUMA option), NUMA BIOS disabled :
http://pastebin.com/raw/P5LrUASN



I would say :
- FreeBSD <= 10.3 : disable NUMA in BIOS
- FreeBSD >= 11   : disable NUMA in BIOS or enable NUMA in kernel.
But let's wait your analysis :)



Ben