Network card interrupt handling

[John Baldwin]

On Wednesday, August 26, 2015 09:30:48 AM Sean Bruno wrote:
> We've been diagnosing what appeared to be out of order processing in
> the network stack this week only to find out that the network card
> driver was shoveling bits to us out of order (em).
> 
> This *seems* to be due to a design choice where the driver is allowed
> to assert a "soft interrupt" to the h/w device while real interrupts
> are disabled.  This allows a fake "em_msix_rx" to be started *while*
> "em_handle_que" is running from the taskqueue.  We've isolated and
> worked around this by setting our processing_limit in the driver to
> -1.  This means that *most* packet processing is now handled in the
> MSI-X handler instead of being deferred.  Some periodic interference
> is still detectable via em_local_timer() which causes one of these
> "fake" interrupt assertions in the normal, card is *not* hung case.
> 
> Both functions use identical code for a start.  Both end up down
> inside of em_rxeof() to process packets.  Both drop the RX lock prior
> to handing the data up the network stack.
> 
> This means that the em_handle_que running from the taskqueue will be
> preempted.  Dtrace confirms that this allows out of order processing
> to occur at times and generates a lot of resets.
> 
> The reason I'm bringing this up on -arch and not on -net is that this
> is a common design pattern in some of the Ethernet drivers.  We've
> done preliminary tests on a patch that moves *all* processing of RX
> packets to the rx_task taskqueue, which means that em_handle_que is
> now the only path to get packets processed.

It is only a common pattern in the Intel drivers. :-/  We (collectively)
spent quite a while fixing this in ixgbe and igb.  Longer (hopefully more
like medium) term I have an update to the interrupt API I want to push in
that allows drivers to manually schedule interrupt handlers using an
'hwi' API to replace the manual taskqueues.  This also ensures that
the handler that dequeues packets is only ever running in an ithread
context and never concurrently.

-- 
John Baldwin