Call for performance evaluation: net.isr.direct (fwd)

[Robert Watson]

On Wed, 12 Oct 2005, Andrew Gallatin wrote:

> Speaking of net.isr, is there any reason why if_simloop() calls 
> netisr_queue() rather than netisr_dispatch()?

Yes -- it's basically to prevent recursion for loopback traffic, which can 
result in both lock orders and general concerns regarding reentrance.  To 
be specific: if you send a packet on a loopback TCP socket, it gets 
processes asynchronously in the netisr rather than immediately walking 
back into the TCP code again.  Right now WITNESS would warn about this, 
but there were also quite bad things that could happen before we did the 
locking work -- for example, when connections are torn down.  It also 
avoids Really Deep Stacks.

At some point, someone needs to look at some scheduler traces and make 
sure we're not seeing anything silly like the following:

- Socket output delivers to TCP, which outputs to loopback, which inserts
   the packet into the netisr queue, waking up the netisr thread.

- The netisr, running at a lower priority, preempts the running thread,
   which may still hold TCP locks, causing it to hit to the lock and yield
   to the user thread, which will now run briefly with depressed priority
   due to priority propagation.

I.e., it may be that we're taking untimely context switches on UP for 
loopback traffic.  I've not actually seen this, but we should make sure 
we're not seeing it.

Robert N M Watson