Advice on a multithreaded netisr patch?

[Robert Watson]

On Sun, 5 Apr 2009, Barney Cordoba wrote:

> I'm curious as to your assertion that hardware transmit queues are a big 
> win. You're really just loading a transmit ring well ahead of actual 
> transmission; there's no need to force a "start" for each packet queued. You 
> then have more overheard managing the multiple queues; more memory used, 
> more cpu cache needed, more interrupts (perhaps), overhead generating the 
> flowid. It seems to me that a more efficient method of transmitting, such as 
> offloading the transmit workload to a kernel task, would be more effective 
> than using multiple transmit queues. All the source thread has to do is 
> queue the packet and get out.

When using multiple cores, we've observed significant contention on the 
transmit-side locks protecting a single output queue; when multiple queues are 
used, that contention is avoided.  The lock only coveres the queue, but the 
overhead of a single high contention lock twice for every packet (enqeueu, 
later dequeue) is significant at high pps and with many cores.

> As an aside, why is Kip doing development on a Chelsio card rather than a 
> more mainstream product such as Intel or Broadcom that would generate more 
> widespread interest?

Because they paid him to to write their driver?  :-)

Robert N M Watson
Computer Laboratory
University of Cambridge