New optimized soreceive_stream() for TCP sockets, proof of concept

[Andrew Gallatin]

Robert Watson writes:
 > On Mon, 5 Mar 2007, Andrew Gallatin wrote:
 > 
 > > With the patch, we finally seem to be performance competative on the receive 
 > > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of 
 > > those OSes do much better (saturate the link with jumbos) when CPU affinity 
 > > is used to bind the interrupt handler and netserver process to different 
 > > cores on the same socket.  I imagine FreeBSD may be able to do even better 
 > > if it ever grows CPU affinity support for both interrupt handlers and 
 > > processes.  With the patch, it performs at least as well, if not better 
 > > than, Solaris and Linux do without CPU affinity.
 > 
 > I don't have numbers in front of me, and am currently packing for a trip to 
 > Tokyo so won't find them before traveling, but my experience has been that 
 > binding the ithread to a specific CPU is very helpful in improving receive 
 > performance.  You can slap a sched_bind(0) into the interrupt handler the 
 > first time it runs and it should stick appropriately, and add a sysctl to 
 > sched_bind() for a user process as a hack to test it out.

OK, So I did a hack which binds anything which calls accept() to
CPU1, and then hacked the intr handler of my driver to bind
it to CPU0.  I saw no improvement.  Darn.

BTW, doing binding like this seems to entirely eliminate the
out-of-order packets I see when net.isr.direct=0.

Drew