RPi4B boot -v "pci0: failed to allocate bus number" and "device_attach: pci0 attach returned 6"; more
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 18 Dec 2022 19:10:45 UTC
For reference, the example context used is
(long line output split for readability):
# uname -apKU
FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0
stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022
root@releng3.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC
arm64 aarch64 1301509 1301509
(So: not my build.)
From the boot -v capture:
pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2
pcib0: parsing FDT for ECAM0:
pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: Bus is not cache-coherent
pcib0: hardware identifies as revision 0x304.
pcib0: note: reported link speed is 5.0 GT/s.
pci1: <PCI bus> on pcib0
pci1: domain=0, physical bus=0
found-> vendor=0x14e4, dev=0x2711, revid=0x00
domain=0, bus=0, slot=0, func=0
class=06-04-00, hdrtype=0x01, mfdev=0
cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=0
powerspec 3 supports D0 D3 current D0
secbus=1, subbus=1
pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci1
pcib0: rman_reserve_resource: start=0xc0000000, end=0xc00fffff, count=0x100000
pcib1: domain 0
pcib1: secondary bus 1
pcib1: subordinate bus 1
pcib1: memory decode 0xc0000000-0xc00fffff
pci2: <PCI bus> on pcib1
pcib1: allocated bus range (1-1) for rid 0 of pci2
pci2: domain=0, physical bus=1
found-> vendor=0x1106, dev=0x3483, revid=0x01
domain=0, bus=1, slot=0, func=0
class=0c-03-30, hdrtype=0x00, mfdev=0
cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=0
powerspec 3 supports D0 D3 current D0
MSI supports 4 messages, 64 bit
map[10]: type Memory, range 64, base 0, size 12, memory disabled
pcib1: slot 0 INTA is routed to irq 92
bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 92 at device 0.0 on pci2
bcm_xhci0: note: xhci firmware not found.
bcm_xhci0: note: installing xhci firmware.
bcm_xhci0: note: xhci firmware detected; firmware is revision 138a1.
pcib1: allocated memory range (0xc0000000-0xc0000fff) for rid 10 of bcm_xhci0
bcm_xhci0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xc0000000
bcm_xhci0: 32 bytes context size, 64-bit DMA
bcm_xhci0: attempting to allocate 1 MSI vectors (4 supported)
bcm_xhci0: using IRQ 93 for MSI
bcm_xhci0: MSI enabled
bcm_xhci0: (New XHCI DeviceId=0x34831106)
usbus0 on bcm_xhci0
bcm_xhci0: usbpf: Attached
bcm_xhci0: note: switched to 32-bit DMA.
pci0: <PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6
In order to get more directly comparable output, I tried
using lspci port and use of lspci on raspiOS64 (my
abbreviation) and fedora 37.
raspiOS64:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 10) (prog-if 00 [Normal decode])
Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: c0000000-c00fffff [size=1M]
Prefetchable memory behind bridge: [disabled]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240] L1 PM Substates
01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
Flags: bus master, fast devsel, latency 0, IRQ 63
Memory at 600000000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
fedora:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) (prog-if 00 [Normal decode])
Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
Flags: bus master, fast devsel, latency 0, IRQ 36
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: 00000000-000fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240] L1 PM Substates
Kernel driver in use: pcieport
01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller
Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
Flags: bus master, fast devsel, latency 0, IRQ 38
Memory at 600000000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
(00:00.0's "Memory behind bridge" displayed based on
different spaces, unfortunately. Not as uniform as I
was intending. Still. . .)
FreeBSD:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 91
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000-0fff [size=4K] [16-bit]
Memory behind bridge: c0000000-c00fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240] L1 PM Substates
01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller
Flags: bus master, fast devsel, latency 0, IRQ 92
Memory at c0000000 (64-bit, non-prefetchable)
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Note the 01:00.0's "Memory at": c0000000 vs. 600000000 for
the two linux exmaples. (Also: No size shown for FreeBSD.)
According to "PhilE Raspberry Pi Engineer & Forum Moderator" at:
https://forums.raspberrypi.com/viewtopic.php?t=288902&sid=792675eb2c6513e936b1cd9fe385a038&start=75
QUOTE
It's worth stating a few facts:
(1) The PCIe controller in BCM2711 has a single outbound window. This means that all addresses for PCIe devices have to be mapped in a single block, such that the spacing in PCIe space matches the spacing in host physical address space.
(2) The size of the outbound window must be a power of two, and the base address of the outbound window must be aligned to that size.
END QUOTE
According to bcm2711-peripherals.pdf figure 1 the standard
placement of PCIe space is 0x600000000..0x7ffffffff (avoiding
overlaps with anything else in any space).
For reference, the live Device Tree for the FreeBSD context
has for pcie@7d500000 :
ranges = <0x02000000 0x00000000 0xc0000000 0x00000006 0x00000000 0x00000000 0x40000000>;
The "0x00000006 0x00000000" part agrees with that "figure 1".
(More recent vintages agree for that part as well.)
I'm far from expert for such things but the 01:00.0 having
"Memory at c0000000 (64-bit, non-prefetchable)" looks
risky for PCI'e outbound memory mapping due to overlaps
having potential consequences.
An additional note of something that might be involved:
The RPi* firmware version that FreeBSD uses still has:
pcie@7d500000 {
. . .
pci@1,0 {
. . .
usb@1,0 {
. . .
but modern firmware (such as in the raspiOS64 and
fedora I used) has:
pcie@7d500000 {
. . .
pci@0,0 {
. . .
usb@0,0 {
. . .
I do not know if that is somehow related to
what I'm reporting or not.
===
Mark Millard
marklmi at yahoo.com