Call for performance evaluation: net.isr.direct (fwd)

[Andrew Gallatin]

Robert Watson writes:
 > 
 > On Wed, 12 Oct 2005, Andrew Gallatin wrote:
 > 
 > > Speaking of net.isr, is there any reason why if_simloop() calls 
 > > netisr_queue() rather than netisr_dispatch()?
 > 
 > Yes -- it's basically to prevent recursion for loopback traffic, which can 
 > result in both lock orders and general concerns regarding reentrance.  To 
 > be specific: if you send a packet on a loopback TCP socket, it gets 
 > processes asynchronously in the netisr rather than immediately walking 
 > back into the TCP code again.  Right now WITNESS would warn about this, 
 > but there were also quite bad things that could happen before we did the 
 > locking work -- for example, when connections are torn down.  It also 
 > avoids Really Deep Stacks.

Right now, at least, it seems to work OK.  I haven't tried witness,
but a non-debug kernel shows a big speedup from enabling it.  Do
you think there is a chance that it could be made to work in FreeBSD?

For what its worth, our TCP RR loopback latency on a dual-core AMD64
3800+ is roughly 2x as much as the solaris and linux latency.  Heck,
we're even slower than Darwin.  And I didn't think it was *possible*
to be slower than Darwin:

---------------------------------------------------------------------
Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                        ctxsw       UNIX         UDP         TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
rome      FreeBSD 7.0-C 7.220  15.7 18.1              36.1           
rome.sw.m  Darwin 8.2.0 1.250 6.531 19.9  33.7  48.4  33.0  48.0  63.
rome         SunOS 5.11 5.140  12.8 15.4  22.0        17.0  43.1 141.
rome      Linux 2.6.9-1 3.760 9.780 11.7  17.0  20.5  18.9  23.2  32.


Using net.isr.direct over the loopback brings the latency down
to 25us.  

vmstat 1 with direct disabled:
 procs      memory      page                   disk   faults      cpu
 r b w     avm    fre  flt  re  pi  po  fr  sr ad0   in   sy  cs us sy id
 2 0 0   33216 457016    0   0   0   0   0   0   4 53593 107075 214030  5 68 27
 2 0 0   33216 457016    0   0   0   0   0   0   0 53694 107279 214356  5 72 24
<...>

And  with it enabled:

 2 0 0   33796 456620    0   0   0   0   0   0   0  106 153987 150570  2 59 39
 2 0 0   33796 456620    0   0   0   0   0   0   0  107 154100 150945  3 61 36



Drew