Enabling MSI-X on -CURRENT for New Network Driver

[David Christensen]

I'm working on adding MSI-X support for a new network driver
and having some difficulty in actually getting an interrupt.
Does this look right?

        /* Select and configure the IRQ. */
        sc->bxe_msix_count = pci_msix_count(dev);
        rid = 1;

        /* Try allocating MSI-X interrupts. */
        if ((sc->bxe_cap_flags & BXE_MSIX_CAPABLE_FLAG) &&
                (bxe_msi_enable >= 2) && (sc->bxe_msix_count > 0)) {

                int msix_needed = sc->bxe_msix_count;

                if (pci_alloc_msix(dev, &sc->bxe_msix_count) == 0) {
                        if (sc->bxe_msix_count == msix_needed) {
                                DBPRINT(sc, BXE_INFO_LOAD, "%s(): Using %d MSI-X "
                                        "vector(s).\n", __FUNCTION__, sc->bxe_msix_count);
                                sc->bxe_flags |= BXE_USING_MSIX_FLAG;
                        } else {
                                pci_release_msi(dev);
                                sc->bxe_flags &= ~BXE_USING_MSIX_FLAG;
                                sc->bxe_msix_count = 0;
                        }
                }
        }

        /* Try allocating MSI interrupts if we didn't get MSI-X. */

        ...

        /* Try legacy interrupt. */
        ...

        /* Allocate the interrupt and report any errors. */
        sc->bxe_res_irq = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid,
                RF_ACTIVE);

        /* Report any IRQ allocation errors. */
        if (sc->bxe_res_irq == NULL) {
                BXE_PRINTF("%s(%d): PCI map interrupt failed!\n",
                        __FILE__, __LINE__);
                rc = ENXIO;
                goto bxe_attach_fail;
        }

        sc->bxe_irq_rid = rid;
        sc->bxe_intr = bxe_intr;


The allocation doesn't fail and I usually see an IRQ allocated to
the driver using "vmstat -i" (though not always):

===[root] /usr/src/sys/modules/bxe # vmstat -i
interrupt                          total       rate
irq1: atkbd0                           1          0
irq4: sio0                         46432          6
irq6: fdc0                            10          0
irq14: ata0                           58          0
irq17: atapci1                     42684          5
cpu0: timer                     15331063       1999
irq256: em0                          917          0
cpu3: timer                     15330811       1999
cpu1: timer                     15330808       1999
cpu2: timer                     15330811       1999
cpu5: timer                     15330811       1999
cpu6: timer                     15330810       1999
cpu4: timer                     15330806       1999
cpu7: timer                     15330811       1999
irq258: bxe0                           2          0
Total                          122736835      16010

But my interrupt handler doesn't seem to be called.
The goal is to get a single interrupt working first,
multiple queue support comes next.  Any ideas?

Dave