ixgbe & if_igb RX ring locking

[Alexander V. Chernikov]

On 17.10.2012 18:06, John Baldwin wrote:
> On Monday, October 15, 2012 9:04:27 am John Baldwin wrote:
>> On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote:
>>> On 13.10.2012 23:24, Jack Vogel wrote:
>>>> On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo<rizzo at iet.unipi.it>  wrote:
>>>
>>>>>
>>>>> one option could be (same as it is done in the timer
>>>>> routine in dummynet) to build a list of all the packets
>>>>> that need to be sent to if_input(), and then call
>>>>> if_input with the entire list outside the lock.
>>>>>
>>>>> It would be even easier if we modify the various *_input()
>>>>> routines to handle a list of mbufs instead of just one.
>>>
>>> Bulk processing is generally a good idea we probably should implement.
>>> Probably starting from driver queue ending with marked mbufs
>>> (OURS/forward/legacy processing (appletalk and similar))?
>>>
>>> This can minimize an impact for all
>>> locks on RX side:
>>> L2
>>> * rx PFIL hook
>>> L3 (both IPv4 and IPv6)
>>> * global IF_ADDR_RLOCK (currently commented out)
>>> * Per-interface ADDR_RLOCK
>>> * PFIL hook
>>>
>>>   From the first glance, there can be problems with:
>>> * Increased latency (we should have some kind of rx_process_limit), but
>>> still
>>> * reader locks being acquired for much longer amount of time
>>>
>>>>>
>>>>> cheers
>>>>> luigi
>>>>>
>>>>> Very interesting idea Luigi, will have to get that some thought.
>>>>
>>>> Jack
>>>
>>> Returning to original post topic:
>>>
>>> Given
>>> 1) we are currently binding ixgbe ithreads to CPU cores
>>> 2) RX queue lock is used by (indirectly) in only 2 places:
>>> a) ISR routine (msix or legacy irq)
>>> b) taskqueue routine which is scheduled if some packets remains in RX
>>> queue and rx_process_limit ended OR we need something to TX
>>>
>>> 3) in practice taskqueue routine is a nightmare for many people since
>>> there is no way to stop "kernel {ix0 que}" thread eating 100% cpu after
>>> some traffic burst happens: once it is called it starts to schedule
>>> itself more and more replacing original ISR routine. Additionally,
>>> increasing rx_process_limit does not help since taskqueue is called with
>>> the same limit. Finally, currently netisr taskq threads are not bound to
>>> any CPU which makes the process even more uncontrollable.
>>
>> I think part of the problem here is that the taskqueue in ixgbe(4) is
>> bogusly rescheduled for TX handling.  Instead, ixgbe_msix_que() should
>> just start transmitting packets directly.
>>
>> I fixed this in igb(4) here:
>>
>> http://svnweb.freebsd.org/base?view=revision&revision=233708
>>
>> You can try this for ixgbe(4).  It also comments out a spurious taskqueue
>> reschedule from the watchdog handler that might also lower the taskqueue
>> usage.  You can try changing that #if 0 to an #if 1 to test just the txeof
>> changes:
>
> Is anyone able to test this btw to see if it improves things on ixgbe at all?
> (I don't have any ixgbe hardware.)
Yes. I'll try to to this next week (since ixgbe driver from at least 9-S 
fails to detect twinax cable which works in 8-S....)).
>


-- 
WBR, Alexander