PCI interrupts passing DMA

[Andrew Gallatin]

Aaro Koskinen writes:

 > > My question is: What the heck could the SMP kernel be doing which
 > > causes the DMA to "complete" faster?
 > 
 > The chipset probably uses PCI bus (MSI-like mechanism) to deliver the
 > interrupt from the IO APIC to the local APIC, which means that the PCI
 > bridge(s) must complete the DMA transfer before the interrupt is
 > delivered to preserve the write order.

AHA!  I think you hit it on the nose.  It turns out that the FreeBSD
SMP kernel sets up all IOAPIC interrupts as IOART_DELLOPRI.   But
linux doesn't set the IOART_DELLOPRI bit.  This seems account for the
difference in behaviour between FreeBSD & linux.

The following diff seems to make SMP FreeBSD behave the same as linux,
and the same as UP FreeBSD:

Index: i386/i386/mpapic.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/mpapic.c,v
retrieving revision 1.63
diff -u -r1.63 mpapic.c
--- i386/i386/mpapic.c  23 Jul 2003 18:59:38 -0000      1.63
+++ i386/i386/mpapic.c  18 Sep 2003 14:07:38 -0000
@@ -134,7 +134,7 @@
        ((u_int32_t)            \
         (IOART_INTMSET |       \
          IOART_DESTPHY |       \
-         IOART_DELLOPRI))
+         IOART_DELFIXED))

 #define DEFAULT_ISA_FLAGS      \
        ((u_int32_t)            \



 > In PIC mode, the interrupt is delivered by the wire and it has no
 > effect on pending writes. A common solution is that the interrupt
 > handler must perform a read from the device to the force flushing of
 > buffers.

Yep.  I was trying to avoid that because PIO reads are so horribly
expensive..   I guess I'll have to do it after all.  I wish MSIs had
been around from the beginning & were more widely used.

Thanks for your help,

Drew