cvs commit: src/sys/dev/em if_em.c if_em.h

[Scott Long]

Andrew Gallatin wrote:
> Scott Long writes:
>  > 
>  > Touching the APIC is tricky.  First, you have to pay the cost of a 
>  > spinlock.  Then you have to may the cost of at least one read and write 
>  > across the FSB.  Even though the APIC registers are memory mapped, they 
>  > are still uncached.  It's not terribly expensive, but it does add up.
>  > Bypassing this and using a fast interrupt means that you pay the cost of
>  > 1 PCI read, which you would have to do anyways with either method, and 1 
>  > PCI write, which will be posted at the host-pci bridge and thus only as 
>  > expensive as an FSB write.  Overall, I don't think that the cost 
>  > difference is a whole lot, but when you are talking about thousands of
>  > interrupts per second, especially if multiple interfaces are running 
>  > under load, it might be important.  And the 750x and 752x chipsets are
>  > so common that it is worthwhile to deal with them (and there are reports
>  > that the aliasing problem is happening on more chipsets than just these 
>  > now).
> 
> I think you've sold me.
> 
> I'm in the process of trying to justify time to write a FreeBSD driver
> for our PCIe 10GbE cards, and I find what you're doing fascinating.
> I'd like to use some of your techniques for the driver I'm writing.
> 
>  > As for latency, the taskqueue runs at the same PI_NET priority as an the
>  > ithread would.  I thought that there was an optimization on some 
>  > platforms to encourage quick preemption for ithreads when they are 
>  > scheduled, but I can't find it now.  So, the taskqueue shouldn't be all
>  > that different from an ithread, and it even means that there won't be
>  > any sharing between instances even if the interrupt vector is shared.
> 
> OK that (a taskqueue not getting the same fast preemption an ithread
> would) was just what I was afraid of.  I'm glad that it is not a
> problem.

The only problems I saw here were early on when I had the taskqueue 
running at a normal thread priority.  It was constantly being preempted
by the netisr, resulting in really bad performance.  Moving it up to
ithread-equivalent priority fixed this.  If you copy this, make sure
you pay close attention to the sched_prio() call that is made in the 
driver.  I don't like exposing the driver to the low-level scheduling 
interface, so the long term plan is to either wrap it into the taskqueue
API, or implement the two-stage interrupt API.

> 
> <...>
> 
>  > However, taskqueues are really just a proof of concept for what I really
>  > want, which is to allow drivers to register both a fast handler and an
>  > ithread handler.  For drivers doing this, the ithread would be private
> 
> Ah, the darwin / MacOSX model.  That would be very cool. 
> 

Yep.  Working in IOKit was very interesting, and this is one of the few
things that transfers well to FreeBSD.  C++ does have a certain elagence
for drivers, but the cost of virtual methods in the fast path of the
driver and stack is still far too high to justify using it.

> Anyway, keep up the good work.  It is inspiring!

Thanks!

Scott