Unstable local network throughput

[Adrian Chadd]

On 10 August 2016 at 12:50, Ben RUBSON <ben.rubson at gmail.com> wrote:
>
>> On 10 Aug 2016, at 21:47, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>>
>> hi,
>>
>> yeah, I'd like you to do some further testing with NUMA. Are you able
>> to run freebsd-11 or -HEAD on these boxes?
>
> Hi Adrian,
>
> Yes I currently have 11 BETA3 running on them.
> I could also run BETA4.

hi,

ok, lets start by getting the NUMA bits into the kernel so you can
mess with things.

add this to the kernel

options MAXMEMDOM=8
(which hopefully is enough)
options VM_NUMA_ALLOC
options DEVICE_NUMA

Then reboot and post your 'dmesg' output to the list. This should show
exactly which domain devices are in.

Install the 'intel-pcm' package. There's a 'pcm-numa.x' command - do
kldload cpuctl, then run pcm-numa.x and see if it works. It should
give us some useful information about NUMA.
(Same as pcm-memory.x, pcm-pcie.x, etc.)

Then next is playing around with interrupt thread / userland cpuset
and memory affinity. We can look at that next. Currently the kernel
doesn't know about NUMA local memory for device driver memory, kernel
allocations for mbufs, etc, but we should still get a "good enough"
idea about things. We can talk about that here once the above steps
are done.

Thanks!



-adrian