Unstable local network throughput

[Adrian Chadd]

[snip]

ok, so this is what I was seeing when I was working on this stuff last.

The big abusers are:

* so_snd lock, for TX'ing producer/consumer socket data
* tcp stack pcb locking (which rss tries to work around, but it again
doesn't help producer/consumer locking, only multiple sockets)
* for some of the workloads, the scheduler spinlocks are pretty
heavily contended and that's likely worth digging into.

Thanks! I'll go try this on a couple of boxes I have with
intel/chelsio 40g hardware in it and see if I can reproduce it. (My
test boxes have the 40g NICs in NUMA domain 1...)



-adrian