cvs commit: src/sys/dev/bge if_bge.c

[Oleg Bulyzhin]

On Sat, Dec 23, 2006 at 09:36:33PM +0000, Robert Watson wrote:
> 
> On Sat, 23 Dec 2006, John Polstra wrote:
> 
> >>That said, dropping and regrabbing the driver lock in the rxeof routine 
> >>of any driver is bad.  It may be safe to do, but it incurs horrible 
> >>performance penalties.  It essentially allows the time-critical, high 
> >>priority RX path to be constantly preempted by the lower priority 
> >>if_start or if_ioctl paths.  Even without this preemption and priority 
> >>inversion, you're doing an excessive number of expensive lock ops in the 
> >>fast path.
> >
> >We currently make this a lot worse than it needs to be by handing off the 
> >received packets one at a time, unlocking and relocking for every packet. 
> >It would be better if the driver's receive interrupt handler would harvest 
> >all of the incoming packets and queue them locally. Then, at the end, hand 
> >off the linked list of packets to the network stack wholesale, unlocking 
> >and relocking only once.  (Actually, the list could probably be handed off 
> >at the very end of the interrupt service routine, after the driver has 
> >already dropped its lock.)  We wouldn't even need a new primitive, if 
> >ether_input() and the other if_input() functions were enhanced to deal 
> >with a possible list of packets instead of just a single one.
> 
> I try this experiement every few years, and generally don't measure much 
> improvement.  I'll try it again with 10gbps early next year once back in 
> the office again.  The more interesting transition is between the link 
> layer and the network layer, which is high on my list of topics to look 
> into in the next few weeks.  In particular, reworking the ifqueue handoff.  
> The tricky bit is balancing latency, overhead, and concurrency...
> 
> FYI, there are several sets of patches floating around to modify if_em to 
> hand off queues of packets to the link layer, etc.  They probably need 
> updating, of course, since if_em has changed quite a bit in the last year.  
> In my implementaiton, I add a new input routine that accepts mbuf packet 
> queues.

I'm just curious, do you remember average length of mbuf queue in your
tests? While experimenting with bge(4) driver (taskqueue, interrupt moderation,
converted bge_rxeof() to above scheme), i've found it's quite easy to exhaust
available mbuf clusters under load (trying to queue hundreids of received
packets). So i had to limit rx queue to rather low length.

> 
> Robert N M Watson
> Computer Laboratory
> University of Cambridge

-- 
Oleg.

================================================================
=== Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg at rinet.ru ===
================================================================