Unstable local network throughput

[Adrian Chadd]

Which ones of these hit the line rate comfortably?



-a


On 11 August 2016 at 15:35, Ben RUBSON <ben.rubson at gmail.com> wrote:
>
>> On 11 Aug 2016, at 18:36, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>>
>> Hi!
>>
>> mlx4_core0: <mlx4_core> mem
>> 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0
>> numa-domain 1 on pci16
>> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6
>> (Aug 11 2016)
>>
>> so the NIC is in numa-domain 1. Try pinning the worker threads to
>> numa-domain 1 when you run the test:
>>
>> numactl -l first-touch-rr -m 1 -c 1 ./test-program
>>
>> You can also try pinning the NIC threads to numa-domain 1 versus 0 (so
>> the second set of CPUs, not the first set.)
>>
>> vmstat -ia | grep mlx (get the list of interrupt thread ids)
>> then for each:
>>
>> cpuset -d 1 -x <irq id>
>>
>> Run pcm-memory.x each time so we can see the before and after effects
>> on local versus remote memory access.
>>
>> Thanks!
>
> Adrian, here are the results :
>
>
>
> Idle system :
> http://pastebin.com/raw/K1iMVHVF
>
>
>
> No pinning :
> http://pastebin.com/raw/w5KuexQ3
> CPU : http://pastebin.com/raw/8zgRaazN
>
> numactl -l fixed-domain-rr -m 1 -c 1 :
> http://pastebin.com/raw/VWweYF9H
> CPU : http://pastebin.com/raw/QjaVH32X
>
> numactl -l fixed-domain-rr -m 0 -c 0 :
> http://pastebin.com/raw/71hfGJdw
> CPU : http://pastebin.com/raw/hef058Na
>
> numactl -l fixed-domain-rr -m 1 -c 1
> + cpuset -l <DOM1_CPU> -x <IRQ> :
> http://pastebin.com/raw/nEQkgMK2
> CPU : http://pastebin.com/raw/R652KAdJ
>
> numactl -l fixed-domain-rr -m 0 -c 0
> + cpuset -l <DOM0_CPU> -x <IRQ> :
> http://pastebin.com/raw/GdYJHyae
> CPU : http://pastebin.com/raw/Ggfx9uF9
>
>
>
> No pinning, default kernel (no NUMA option) :
> http://pastebin.com/raw/iQ2u8d8k
> CPU : http://pastebin.com/raw/Xr77KpcM
>
> default kernel (no NUMA option)
> + cpuset -l <DOM0_CPU>
> + cpuset -l <DOM0_CPU> -x <IRQ> :
> http://pastebin.com/raw/VBWg4SZs
>
> default kernel (no NUMA option)
> + cpuset -l <DOM1_CPU>
> + cpuset -l <DOM1_CPU> -x <IRQ> :
> http://pastebin.com/raw/SrJLZxuT
>
>
>
> No pinning, default kernel (no NUMA option), NUMA BIOS disabled :
> http://pastebin.com/raw/P5LrUASN
>
>
>
> I would say :
> - FreeBSD <= 10.3 : disable NUMA in BIOS
> - FreeBSD >= 11   : disable NUMA in BIOS or enable NUMA in kernel.
> But let's wait your analysis :)
>
>
>
> Ben
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"