Poor USB performance on ASUS 520 motherboard (no IRQ?)

From: Daniel O'Connor via freebsd-stable <freebsd-stable_at_freebsd.org>
Date: Wed, 24 Nov 2021 06:54:57 UTC
Hi,
We bought a few low end motherboards to drive our hardware which is controlled by a custom USB (Cypress FX2 interface) board.

I found that the Gigabyte GA-A320M-H one works fine but the ASUS A520M-A/CSM one is significantly slower for tests where there are many back and forth messages (streaming data seems fine).

I've tried updating the BIOS and tested FreeBSD 12 (same as the Gigabyte) and FreeBSD 13 with no change.

One thing I did notice is this dmesg output:
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfcfa0000-0xfcfa7fff at device 0.0 on pci1
xhci0: 32 bytes context size, 64-bit DMAxhci1: <XHCI (generic) USB 3.0 controller
...
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfcb00000-0xfcbfffff at device 0.3 on pci7
xhci1: 64 bytes context size, 64-bit DMA
..
xhci2: <XHCI (generic) USB 3.0 controller> mem 0xfca00000-0xfcafffff at device 0.4 on pci7
xhci2: 64 bytes context size, 64-bit DMA

vs the working one:
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfcea0000-0xfcea7fff irq 28 at device 0.0 on pci1
xhci0: 32 bytes context size, 64-bit DMA
...
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfca00000-0xfcafffff irq 55 at device 0.3 on pci7
xhci1: 64 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table
...
xhci2: <XHCI (generic) USB 3.0 controller> mem 0xfc900000-0xfc9fffff irq 52 at device 0.4 on pci7
xhci2: 64 bytes context size, 64-bit DMA
xhci2: Unable to map MSI-X table

I think the MSI-X warning is a difference between 12 & 13 but the lack of IRQ appears to be common to the brokenness.

I tried setting these with no change:
hw.pci.honor_msi_blacklist="0"
hw.pci.msix_rewrite_table="1"

And also without success:
hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

pciconf -lcvb shows:
xhci0@pci0:1:0:0:       class=0x0c0330 card=0x11421b21 chip=0x43ec1022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfcfa0000, size 32768, enabled
    cap 05[50] = MSI supports 8 messages, 64 bit
    cap 11[68] = MSI-X supports 8 messages
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) RO NS
                 link x4(x4) speed 8.0(8.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0019[200] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1
xhci1@pci0:7:0:3:       class=0x0c0330 card=0x876b1043 chip=0x15e01022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Raven USB 3.1'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfcb00000, size 1048576, enabled
    cap 09[48] = vendor (length 8)
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 10[64] = PCI-Express 2 endpoint max data 256(256) RO NS
                 link x16(x16) speed 8.0(8.0) ASPM disabled(L0s/L1)
    cap 05[a0] = MSI supports 8 messages, 64 bit
    cap 11[c0] = MSI-X supports 8 messages
                 Table in map 0x10[0xfe000], PBA in map 0x10[0xff000]
    ecap 000b[100] = Vendor 1 ID 1
xhci2@pci0:7:0:4:       class=0x0c0330 card=0x876b1043 chip=0x15e11022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Raven USB 3.1'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xfca00000, size 1048576, enabled
    cap 09[48] = vendor (length 8)
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 10[64] = PCI-Express 2 endpoint max data 256(256) RO NS
                 link x16(x16) speed 8.0(8.0) ASPM disabled(L0s/L1)
    cap 05[a0] = MSI supports 8 messages, 64 bit
    cap 11[c0] = MSI-X supports 8 messages
                 Table in map 0x10[0xfe000], PBA in map 0x10[0xff000]
    ecap 000b[100] = Vendor 1 ID 1

I'm happy to try a newer kernel or patches if anyone has suggestions :)

Thanks.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum