Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: <pcib1 memory window> request" leads to panic
- Reply: John Baldwin : "Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: <pcib1 memory window> request" leads to panic"
- In reply to: Mark Millard : "Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: <pcib1 memory window> request" leads to panic"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Wed, 14 Feb 2024 18:16:02 UTC
Top posting a related but separate item: I looked up some old (2022-Dec-17) lspci -v output from a Linux boot. Note the "Memory at" value 600000000 (in the 35 bit BCM2711 address space) and the "(64-bit, non-prefetchable)" (and "[size=4K]"). 01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0 Flags: bus master, fast devsel, latency 0, IRQ 51 Memory at 600000000 (64-bit, non-prefetchable) [size=4K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+ Capabilities: [c4] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd "Memory at 600000000 (64-bit, non-prefetchable)": Violation of a PCIe standard? On Feb 14, 2024, at 09:57, Mark Millard <marklmi@yahoo.com> wrote: > On Feb 14, 2024, at 08:08, John Baldwin <jhb@FreeBSD.org> wrote: > >> On 2/12/24 5:57 PM, Mark Millard wrote: >>> On Feb 12, 2024, at 16:36, Mark Millard <marklmi@yahoo.com> wrote: >>>> On Feb 12, 2024, at 16:10, Mark Millard <marklmi@yahoo.com> wrote: >>>> >>>>> On Feb 12, 2024, at 12:00, Mark Millard <marklmi@yahoo.com> wrote: >>>>> >>>>>> [Gack: I was looking at the wrong vintage of source code, predating >>>>>> your changes: wrong system used.] >>>>>> >>>>>> >>>>>> On Feb 12, 2024, at 10:41, Mark Millard <marklmi@yahoo.com> wrote: >>>>>> >>>>>>> On Feb 12, 2024, at 09:32, John Baldwin <jhb@freebsd.org> wrote: >>>>>>> >>>>>>>> On 2/9/24 8:13 PM, Mark Millard wrote: >>>>>>>>> Summary: >>>>>>>>> pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>>>>> . . . >>>>>>>>> rman_manage_region: <pcib1 memory window> request: start 0x600000000, end 0x6000fffff >>>>>>>>> panic: Failed to add resource to rman >>>>>>>> >>>>>>>> Hmmm, I suspect this is due to the way that bus_translate_resource works which is >>>>>>>> fundamentally broken. It rewrites the start address of a resource in-situ instead >>>>>>>> of keeping downstream resources separate from the upstream resources. For example, >>>>>>>> I don't see how you could ever release a resource in this design without completely >>>>>>>> screwing up your rman. That is, I expect trying to detach a PCI device behind a >>>>>>>> translating bridge that uses the current approach should corrupt the allocated >>>>>>>> resource ranges in an rman long before my changes. >>>>>>>> >>>>>>>> That said, that doesn't really explain the panic. Hmm, the panic might be because >>>>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the bus_translate_resource >>>>>>>> hack only kicks in the activate_resource method of pci_host_generic.c. >>>>>>>> >>>>>>>>> Detail: >>>>>>>>> . . . >>>>>>>>> pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>>>> >>>>>>>> This indicates this is a translating bus. >>>>>>>> >>>>>>>>> pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci0 >>>>>>>>> rman_manage_region: <pcib1 bus numbers> request: start 0x1, end 0x1 >>>>>>>>> pcib0: rman_reserve_resource: start=0xc0000000, end=0xc00fffff, count=0x100000 >>>>>>>>> rman_reserve_resource_bound: <PCIe Memory> request: [0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1 >>>>>>>>> rman_reserve_resource_bound: trying 0xffffffff <0xc0000000,0xfffff> >>>>>>>>> considering [0xc0000000, 0xffffffff] >>>>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 (requested 0x100000) >>>>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000 >>>>>>>>> allocating from the beginning >>>>>>>>> rman_manage_region: <pcib1 memory window> request: start 0x600000000, end 0x6000fffff >>>>> >>>>> What you later typed does not match: >>>>> >>>>> 0x600000000 >>>>> 0x6000fffff >>>>> >>>>> You later typed: >>>>> >>>>> 0x60000000 >>>>> 0x600fffffff >>>>> >>>>> This seems to have lead to some confusion from using the >>>>> wrong figure(s). >>>>> >>>>>>>> The fact that we are trying to reserve the CPU addresses in the rman is because >>>>>>>> bus_translate_resource rewrote the start address in the resource after it was allocated. >>>>>>>> >>>>>>>> That said, I can't see why rman_manage_region would actually fail. At this point the >>>>>>>> rman is empty (this is the first call to rman_manage_region for "pcib1 memory window"), >>>>>>>> so only the check that should be failing are the checks against rm_start and >>>>>>>> rm_end. For the memory window, rm_start is always 0, and rm_end is always >>>>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new (0x60000000 - 0x600fffffff) >>>>>>>> ranges are within those bounds. >>>>> >>>>> No: >>>>> >>>>> 0xffffffff >>>>> >>>>> .vs (actual): >>>>> >>>>> 0x600000000 >>>>> 0x6000fffff >> >> Ok, then this explains the failure if the "raw" addresses are above 4G. I have >> access to an emag I'm currently using to test fixes to pci_host_generic.c to >> avoid corrupting struct resource objects. I'll post the diff once I've got >> something verified to work. >> >>> It looks to me like in sys/dev/pci/pci_pci.c the: >>> static void >>> pcib_probe_windows(struct pcib_softc *sc) >>> { >>> . . . >>> pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, 0xffffffff); >>> . . . >>> is just inappropriately restrictive about where in the system >>> address space a PCIe can validly be mapped to on the high end. >>> That, in turn, leads to the rejection on the RPi4B now that >>> the range use is checked. >> >> No, the physical register in PCI-PCI bridges is only 32-bits. Only the >> prefetchable BAR supports 64-bit addresses. > > Just for my edification . . . > > As I understand, SYS_RES_MEMORY for the BCM2711 > means the 35 bit addressing space in the BCM2711, > not a PCIe device internal address range that > corresponds. Am I wrong about that? > > If I'm wrong, what does identify the 35 bit > addressing space in the BCM2711? > > If I'm correct, then the 0..0xffffffff > seems to be from the wrong address space up > front. Or, may be, the SYS_RES_MEMORY and the > 0xffffffff argments are not related as I > expected and the 0xffffffff is not a > SYS_RES_MEMORY value? > >> This is why the host bridge >> is doing a translation from the CPU side (0x600000000) to the PCI BAR >> addresses (0xc0000000) so that the BAR addresses are down in the 32-bit >> address range. It's also true that many PCI devices only support 32-bit >> addresses in memory BARs. 64-bit BARs are an optional extension not >> universally supported. >> >> The translation here is somewhat akin to a type of MMU where the CPU >> addresses are mapped to PCI addresses. The problem here is that the >> PCI BAR resources need to "stay" as PCI addresses since we depend on >> being able to use rman_get_start/end to get the PCI addresses of >> allocated resources, but pci_host_generic.c currently rewrites the >> addresses. >> >> Probably I should remove rman_set_start/end entirely (Warner added them >> back in 2004) as the methods don't do anything to deal with the fallout >> that the rman.rm_list linked-list is no longer sorted by address once >> some addresses get rewritten, etc. >> === Mark Millard marklmi at yahoo.com