nfe taskq performance issues

[Pyun YongHyeon]

On Thu, Jul 23, 2009 at 08:58:07AM -0700, Peter Steele wrote:
> We've been hitting serious nfe taskq performance issues during stress
> tests and in doing some research on the problem we came across this old
> email:
> 
>  
> 
> From: Ivan Voras <ivoras at freebsd.org>
> Date: April 28, 2009 3:53:14 AM PDT
> To: freebsd-threads at freebsd.org
> Cc: freebsd-net at freebsd.org, freebsd-performance at freebsd.org
> Subject: Re: FreeBSD 7.1 taskq em performance
> >
> > I have been hitting some barrier with FreeBSD 7.1 network performance.
> I
> > have written an application which contains two kernel threads that
> takes
> > mbufs directly from a network interface and forwards to another
> network
> > interface. This idea is to simulate different network environment.
> >
> > I have been using FreeBSD 6.4 amd64 and tested with an Ixia box
> > (specialised hardware firing very high packet rate). The PC was a
> Core2 2.6
> > GHz with dual ports Intel PCIE Gigabit network card. It can manage up
> to 1.2
> > million pps.
> >
> > I have a higher spec PC with FreeBSD 7.1 amd64 and Quadcore 2.3 GHz
> and
> > PCIE Gigabit network card. The performance can only achieve up to 600k
> pps.
> > I notice the 'taskq em0' and 'taskq em1' is solid 100% CPU but it is
> not in
> > FreeBSD 6.4. 
> 
>  
> 
> In our case we are running FreeBSD 7.0, but we are seeing our boxes
> experience serious thread starvation issues as the nfe0 cpu percentage
> climbs steadily while cpu idle time drops at times to 0 percent. This
> email thread mentioned a patch for the em driver here:
> 
>  
> 
> http://people.yandex-team.ru/~wawa/ 
> <http://people.yandex-team.ru/%7Ewawa/> 
> 
>  
> 
> Does anyone know if this patch will work with the nfe driver?
> 

That's for em(4).

AFAIK all nfe(4) controllers lacks intelligent interrupts
moderation so driver should be prepared to handle excessive
interrupt loads. I'm not sure whether NVIDIA ethernet controllers
really lacks efficient interrupt mitigation mechanism but it
seems Linux also faces the same hardware problem.
As you might know there is no publicly available data sheet for
NVIDIA controllers so setting it right looks very hard to me.