ixgbe & if_igb RX ring locking

[John Baldwin]

On Monday, October 15, 2012 9:04:27 am John Baldwin wrote:
> On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote:
> > On 13.10.2012 23:24, Jack Vogel wrote:
> > > On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo<rizzo at iet.unipi.it>  wrote:
> > 
> > >>
> > >> one option could be (same as it is done in the timer
> > >> routine in dummynet) to build a list of all the packets
> > >> that need to be sent to if_input(), and then call
> > >> if_input with the entire list outside the lock.
> > >>
> > >> It would be even easier if we modify the various *_input()
> > >> routines to handle a list of mbufs instead of just one.
> > 
> > Bulk processing is generally a good idea we probably should implement.
> > Probably starting from driver queue ending with marked mbufs 
> > (OURS/forward/legacy processing (appletalk and similar))?
> > 
> > This can minimize an impact for all
> > locks on RX side:
> > L2
> > * rx PFIL hook
> > L3 (both IPv4 and IPv6)
> > * global IF_ADDR_RLOCK (currently commented out)
> > * Per-interface ADDR_RLOCK
> > * PFIL hook
> > 
> >  From the first glance, there can be problems with:
> > * Increased latency (we should have some kind of rx_process_limit), but 
> > still
> > * reader locks being acquired for much longer amount of time
> > 
> > >>
> > >> cheers
> > >> luigi
> > >>
> > >> Very interesting idea Luigi, will have to get that some thought.
> > >
> > > Jack
> > 
> > Returning to original post topic:
> > 
> > Given
> > 1) we are currently binding ixgbe ithreads to CPU cores
> > 2) RX queue lock is used by (indirectly) in only 2 places:
> > a) ISR routine (msix or legacy irq)
> > b) taskqueue routine which is scheduled if some packets remains in RX 
> > queue and rx_process_limit ended OR we need something to TX
> > 
> > 3) in practice taskqueue routine is a nightmare for many people since 
> > there is no way to stop "kernel {ix0 que}" thread eating 100% cpu after 
> > some traffic burst happens: once it is called it starts to schedule 
> > itself more and more replacing original ISR routine. Additionally, 
> > increasing rx_process_limit does not help since taskqueue is called with 
> > the same limit. Finally, currently netisr taskq threads are not bound to 
> > any CPU which makes the process even more uncontrollable.
> 
> I think part of the problem here is that the taskqueue in ixgbe(4) is
> bogusly rescheduled for TX handling.  Instead, ixgbe_msix_que() should
> just start transmitting packets directly.
> 
> I fixed this in igb(4) here:
> 
> http://svnweb.freebsd.org/base?view=revision&revision=233708
> 
> You can try this for ixgbe(4).  It also comments out a spurious taskqueue 
> reschedule from the watchdog handler that might also lower the taskqueue 
> usage.  You can try changing that #if 0 to an #if 1 to test just the txeof 
> changes:

Is anyone able to test this btw to see if it improves things on ixgbe at all?
(I don't have any ixgbe hardware.)

-- 
John Baldwin