Unstable local network throughput

[Adrian Chadd]

On 16 August 2016 at 02:58, Ben RUBSON <ben.rubson at gmail.com> wrote:
>
>> On 16 Aug 2016, at 03:45, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>>
>> Hi,
>>
>> ok, can you try 5) but also running with the interrupt threads pinned to CPU 1?
>
> What do you mean by interrupt threads ?
>
> Perhaps you mean the NIC interrupts ?
> In this case see 6) and 7) where NIC IRQs are pinned to CPUs 0-11 (6) and 11-23 (7) ?

Hm, interesting. ok. So, I wonder what the maximum per-domain memory
throughput is.

I don't have any other easy things to instrument right now - the
"everything disabled" method likely works best because of how the
system is interleaving memory for you (instead of the OS trying to do
it). Not pinning things means latency can be kept down to work around
lock contention (ie, if a lock is held by thread A, and thread B needs
to make some progress, it can make progress on another CPU , keeping
CPU A held for a shorter period of time.)

Would you mind compiling in LOCK_PROFILING and doing say, these tests
with lock profiling enabled? It'll impact performance, sure, but I'd
like to see what the locking looks like.

sysctl debug.lock.prof.reset=1
sysctl debug.lock.prof.enable=1
(run test for a few seconds)
sysctl debug.lock.prof.enable=0
sysctl debug.lock.prof.stats (and capture)

* interrupts - domain 0, work - domain 1
* interrupts - domain 1, work - domain 1
* interrupts - domain 1, work - domain 0

Thanks!



-adrian