PCI interrupts passing DMA

[Andrew Gallatin]

I was toying with a programmable PCI card and wrote some code 
which DMAs a small block of data to the host, and then interrupts the
host.   The host checks the end of the block, and sees if it gets the
value it expects.  

On an SMP P4 (hyperthreaded, with ServerWorks chipset) FreeBSD 4.8 UP,
and on Linux 2.4.18, there is a huge delay between the interrupt being
handled, and the DMA finally completing (from the host's perspective).
Time enough for the interrupt handler to be triggered 3 or 4 times,
and to print "foo" to a serial console line each time it notices
that the DMA has not completed.

The interesting thing is that on FreeBSD 4.8SMP, and FreeBSD
5.1-current (SMP), the data has arrived by the time the interrupt
handler is called.

This would be easy to explain if the interrupt latency were vastly
different between the FreeBSD SMP kernel and the other kernels, but it
does not seem to be.  It actually seems to be about 5us faster
(interrupt to wakeup of user-level process, so some fat is in there)
than the FreeBSD UP kernel, possibly due to APIC io.  *measurement done
without console printf*

My question is: What the heck could the SMP kernel be doing which
causes the DMA to "complete" faster?   

Thanks,

Drew