From nobody Wed Feb 14 18:16:02 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZmbn2ynBz51hYk for ; Wed, 14 Feb 2024 18:16:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-54.consmr.mail.gq1.yahoo.com (sonic315-54.consmr.mail.gq1.yahoo.com [98.137.65.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZmbm1rGVz4R53 for ; Wed, 14 Feb 2024 18:16:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=jnlLwFf9; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1707934578; bh=STGOR7uwlL32Djb90xxLmXVD4396T33A28E2CKIUUiA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=jnlLwFf956VErlaD5eg2zmji0xilni35r1lTBDgpORVFrxWhGdklGgZ7GbGxpJ+2LqQqw/C08smgogeLBRBxbNg7dIhkxi2WhMnsE2VAB0yC0R+FJMdpr+LLLS4P3MH1/RBiKqST85tVWsT0nYVQsoqzfKXZKVirxpcg6TRgzx+pWZu8Lo7vChW3ld2Gyc9bG6kkG8vBz7FyMu/tQiRXxB5gternbUnc4S++ncgNyPmrOnOGO54jikus0917FFhCD9JKAXI33voDotcIQQbty5NWcQstE1OKpyJqvjGRS9fXGcl+JSeBmyHW1m1+9LVoY+t+HByrV3wp+JJwrPFynA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1707934578; bh=YWkWVdzVaGbxA38ur6vxKcaSvb0o6IjTw2Cd08kox5V=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=kD5tTLvB0g5ld5mAnxNqAwi7a9VYzb62aBeaVRbX0Y7HYB4X7RuDou1aAQHammI7uDA98E2agLyHOqJeM9eq1SbPBhwyEnZQpu1Z6xjuafHxv+GCcMdWh6neKxTnFUj7cUuboUOugtwTD8dVfIGjdKBoigjNfZAuZzNqWzeLSJeY4tKRtI1KeNsc8O4OEjxH/qtHle+PBuqT8BKBaBRB+472Tk1E4JuW6spQ1vh14ldkoG5noJCfivIr5IZLhVrVZUac0wmrCUfw65Yfex+f+XdiQQnFxtKZzgqAcWUQG9rD6Kn2TSfSukrENPfAqyfT+Zp/nUGTxJtqdKXQqBSO4A== X-YMail-OSG: qX7yZVUVM1ki2loZzACq3_QDIMR8yV4n2qYJDLeLGqBA79ALqRiojnCCbQywhtr 9.OmEH1SIgDl_bNVIG.gyCNZ_PBJtKa1bvMqPUqJxjy4I2agSU8k1w1_xFc9pds9azyCORktWtYS ada5jVmrCos0AENmnyH4yS.21_f3px6zirSM2_KQQecPA7mitEq4AY3seKZ_3mxhPR3nFdT7tvY5 3JVsDeggwAj5jlTK7I1KJ8mJ4dYuBYiVQXUSCM_AP5JH8iAjeqw3R3nOUJfkRHMf9ykKPogm.cCM ap3mo7Go7vyoiyhfRAW0eoIgn8YjPxNy0fok4w5Yb0Yelhw8HTTNrieDHmKL0vrOeaMVlyHV7ZsJ ovqOznhyz5RtnTZOmuEveMpdsMRi3C.AeJ0ngnZBNl0o6q8Gp4Y5gLaizc_O7ECO4yJZy79w9Zbz D4hP_BGjOUsIHsWJhivgsnfP1q0E19oSuttrMW8VYBTR5CkHPrhLL6IqYLlobEXIB22rr601OoWx hq5A27.QxKfDJSme4YN7EUXqZU50nsEh2hseWWlO9zM23O.Nw0cE8306TLzfGpfalkHOmUuEwyCi qHdLfjj8xK4FFEwCGsBeZENrZsuQvB47uXPKleDD7D7dqiXF.hY3s1zDqoooWdVkJ7oH2Boqxfw5 kZY0mk2HmY8WmOthvYF.eDXLYe.a1jm3kQ4a8zatqTymj5mL2S5ZFswWDTfgdeTpwcg7BJ2Qnz4I H3shA058piGmgMWyF1tsdIit2UwZ98U99AWXAU4cN0r5GKeSECinjyLOosnVFcSjiQynru6c9uBp vKLD3HiQtwHj5tg2BgcPuYHdQTZypNZfhLzdk6AN8.mrlM1BGjXy6_UhIiACF2e7pNvNIz6PmPWT tKtfbtyOx6OAfNJlzBZUOb9rVj4k0._pHrk5on94YOgjqHIEtNHqtObs44lpZgntUrWJq33RmsF6 4ax0UivhZPRhMRpM9PYf22z5T0wzPoujfuP51.itKSZ2VKAowLg0MysNknwq_EsxeSUaieEQUenU kWEmnpEtcHIs3hONv5sDJeQc3nD53PFp4ZkndgUy3M72g7Em06rH5vI5OXWNO2sqIaufEjtecF.3 M9H6xwMxO0KfgisA1kR.QNwtunzq12tRL.En_j__oNg4JwuPFcuq_.vIBxrCVBJvgUoIpTztZZA7 9Q4eSfmUNxLp8Pder65N620UBhU5g2PgI9fRCQm5bmuj2LUHQoxyVQ6uj8roE26GifO.e201jySq crXezzuAg_3b5NkCPrDR36RmXCIZXDQGAzIbr4HkPb4Y0LWvQgAfwxggDzIk6lcQ8LbAWa1coA_A o2lSEUeoWe4RxXToLMgmN_Wr_CNZjJDgQRVet0Rv0Yl0P3fqxKBI5rDzOHmGsNBX5gFVX3op9O.z 04nkVP.NLKPMOl561SAYTTvn3KEQSKrHnPpScwfgnRncsbepYw5u46c.McRbAt3BwfNgOgexSozo dLXvoYcfuLFN3qvBHUvR7maxL8SvqKnezGxFv.D3At6WtekvbqUPrdj_kc1A49.Wp3S.8SvEqA0i bDjAICLfOLveKGx006lUmpD0ou1PCNwiwffOx3_hny.yRDAIc0oILzCH_wGSEYb2woJkHhmc4Qgq BTwXVIaU5M.W.NzZ6O_.eMunhV3TI6rJNUNiMVLnB2d_4JE0h0jY29ne1BmrovqJEJS9vbGrtiz7 p9A5h10D.oSZ3oGVBP5kCXMLS5pt8O4WkGckfkwNZSSowQT0dkSTevmrwguoPfmo..0QRce1DWf4 qrzoSD7OrLvs2lo2iu8zeIZM1.vkSDu00ANinznflOH.vmAS4_5ISwdgo_NBkywcPIKFwwqkCh43 9HLe657WJeODJ4XLkT9ZZ2o98XsXGIL7cdMhl2xBhhMQQ.wRwonmJWhaV5DIYAsZl0SyXJDfdcYs wNtYYKIRvHgscAyWP9a5oXi2B4MIcvALHRmw2KAuHmUhRnUIWASmN8kcmXVlxy58oFcWRR_zneR8 nswZvsZUAzwiDL9KNbupkWa6vHqwUQ37csE_NhRqq8XqKbec2qY3N_Rblao0zXC2kdiBe4.7uIt7 Oy3DR8a6iPdvsHH4w59iD_VaccxLxDLZTiV9UPwnQ93NxASX.WZYptyEPbNtlkqy.5csDyI..Lr0 2zzVEk4w6CljfnUJlFvLXcal4ctCmQJ96Hv1d5JSb93j_aKnM2ltq2QGdqqLTzveSztoFRqj7aMZ rAayWhVjJ4I8LotDcvIa7F5o5iDIYY76jK8i5ne0Ty7g.9.0VCKac.E6yw3TiUF5IxvfhrogjZYq q564- X-Sonic-MF: X-Sonic-ID: 0d52a574-75f6-4567-869f-8a3446ac7e69 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Wed, 14 Feb 2024 18:16:18 +0000 Received: by hermes--production-gq1-5c57879fdf-tnlsv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 75bd2ec618e725ea0b4cd6d8312a2072; Wed, 14 Feb 2024 18:16:13 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: request" leads to panic From: Mark Millard In-Reply-To: <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> Date: Wed, 14 Feb 2024 10:16:02 -0800 Cc: FreeBSD ARM List , Current FreeBSD , Warner Losh Content-Transfer-Encoding: quoted-printable Message-Id: References: <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5.ref@yahoo.com> <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5@yahoo.com> <1F704317-FDB8-4BDA-8A67-61CF48794DFE@yahoo.com> <9AFDF067-96E4-4E67-90D2-F40DAF3F138F@yahoo.com> <4C279710-5F88-4295-B1A4-7C395F3587E5@yahoo.com> <3A145420-399D-4EBD-9FF4-18924908AB1D@yahoo.com> <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> To: John Baldwin X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_THREE(0.00)[4]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.30:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.30:from] X-Rspamd-Queue-Id: 4TZmbm1rGVz4R53 Top posting a related but separate item: I looked up some old (2022-Dec-17) lspci -v output from a Linux boot. Note the "Memory at" value 600000000 (in the 35 bit BCM2711 address space) and the "(64-bit, non-prefetchable)" (and "[size=3D4K]"). 01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller Device tree node: = /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0 Flags: bus master, fast devsel, latency 0, IRQ 51 Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+ Capabilities: [c4] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd "Memory at 600000000 (64-bit, non-prefetchable)": Violation of a PCIe standard? On Feb 14, 2024, at 09:57, Mark Millard wrote: > On Feb 14, 2024, at 08:08, John Baldwin wrote: >=20 >> On 2/12/24 5:57 PM, Mark Millard wrote: >>> On Feb 12, 2024, at 16:36, Mark Millard wrote: >>>> On Feb 12, 2024, at 16:10, Mark Millard wrote: >>>>=20 >>>>> On Feb 12, 2024, at 12:00, Mark Millard wrote: >>>>>=20 >>>>>> [Gack: I was looking at the wrong vintage of source code, = predating >>>>>> your changes: wrong system used.] >>>>>>=20 >>>>>>=20 >>>>>> On Feb 12, 2024, at 10:41, Mark Millard = wrote: >>>>>>=20 >>>>>>> On Feb 12, 2024, at 09:32, John Baldwin wrote: >>>>>>>=20 >>>>>>>> On 2/9/24 8:13 PM, Mark Millard wrote: >>>>>>>>> Summary: >>>>>>>>> pcib0: mem = 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: = 0x40000000 >>>>>>>>> . . . >>>>>>>>> rman_manage_region: request: start = 0x600000000, end 0x6000fffff >>>>>>>>> panic: Failed to add resource to rman >>>>>>>>=20 >>>>>>>> Hmmm, I suspect this is due to the way that = bus_translate_resource works which is >>>>>>>> fundamentally broken. It rewrites the start address of a = resource in-situ instead >>>>>>>> of keeping downstream resources separate from the upstream = resources. For example, >>>>>>>> I don't see how you could ever release a resource in this = design without completely >>>>>>>> screwing up your rman. That is, I expect trying to detach a = PCI device behind a >>>>>>>> translating bridge that uses the current approach should = corrupt the allocated >>>>>>>> resource ranges in an rman long before my changes. >>>>>>>>=20 >>>>>>>> That said, that doesn't really explain the panic. Hmm, the = panic might be because >>>>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the = bus_translate_resource >>>>>>>> hack only kicks in the activate_resource method of = pci_host_generic.c. >>>>>>>>=20 >>>>>>>>> Detail: >>>>>>>>> . . . >>>>>>>>> pcib0: mem = 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: = 0x40000000 >>>>>>>>=20 >>>>>>>> This indicates this is a translating bus. >>>>>>>>=20 >>>>>>>>> pcib1: irq 91 at device 0.0 on pci0 >>>>>>>>> rman_manage_region: request: start 0x1, = end 0x1 >>>>>>>>> pcib0: rman_reserve_resource: start=3D0xc0000000, = end=3D0xc00fffff, count=3D0x100000 >>>>>>>>> rman_reserve_resource_bound: request: = [0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1 >>>>>>>>> rman_reserve_resource_bound: trying 0xffffffff = <0xc0000000,0xfffff> >>>>>>>>> considering [0xc0000000, 0xffffffff] >>>>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 = (requested 0x100000) >>>>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000 >>>>>>>>> allocating from the beginning >>>>>>>>> rman_manage_region: request: start = 0x600000000, end 0x6000fffff >>>>>=20 >>>>> What you later typed does not match: >>>>>=20 >>>>> 0x600000000 >>>>> 0x6000fffff >>>>>=20 >>>>> You later typed: >>>>>=20 >>>>> 0x60000000 >>>>> 0x600fffffff >>>>>=20 >>>>> This seems to have lead to some confusion from using the >>>>> wrong figure(s). >>>>>=20 >>>>>>>> The fact that we are trying to reserve the CPU addresses in the = rman is because >>>>>>>> bus_translate_resource rewrote the start address in the = resource after it was allocated. >>>>>>>>=20 >>>>>>>> That said, I can't see why rman_manage_region would actually = fail. At this point the >>>>>>>> rman is empty (this is the first call to rman_manage_region for = "pcib1 memory window"), >>>>>>>> so only the check that should be failing are the checks against = rm_start and >>>>>>>> rm_end. For the memory window, rm_start is always 0, and = rm_end is always >>>>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new = (0x60000000 - 0x600fffffff) >>>>>>>> ranges are within those bounds. >>>>>=20 >>>>> No: >>>>>=20 >>>>> 0xffffffff >>>>>=20 >>>>> .vs (actual): >>>>>=20 >>>>> 0x600000000 >>>>> 0x6000fffff >>=20 >> Ok, then this explains the failure if the "raw" addresses are above = 4G. I have >> access to an emag I'm currently using to test fixes to = pci_host_generic.c to >> avoid corrupting struct resource objects. I'll post the diff once = I've got >> something verified to work. >>=20 >>> It looks to me like in sys/dev/pci/pci_pci.c the: >>> static void >>> pcib_probe_windows(struct pcib_softc *sc) >>> { >>> . . . >>> pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, = 0xffffffff); >>> . . . >>> is just inappropriately restrictive about where in the system >>> address space a PCIe can validly be mapped to on the high end. >>> That, in turn, leads to the rejection on the RPi4B now that >>> the range use is checked. >>=20 >> No, the physical register in PCI-PCI bridges is only 32-bits. Only = the >> prefetchable BAR supports 64-bit addresses. >=20 > Just for my edification . . . >=20 > As I understand, SYS_RES_MEMORY for the BCM2711 > means the 35 bit addressing space in the BCM2711, > not a PCIe device internal address range that > corresponds. Am I wrong about that? >=20 > If I'm wrong, what does identify the 35 bit > addressing space in the BCM2711? >=20 > If I'm correct, then the 0..0xffffffff > seems to be from the wrong address space up > front. Or, may be, the SYS_RES_MEMORY and the > 0xffffffff argments are not related as I > expected and the 0xffffffff is not a > SYS_RES_MEMORY value? >=20 >> This is why the host bridge >> is doing a translation from the CPU side (0x600000000) to the PCI BAR >> addresses (0xc0000000) so that the BAR addresses are down in the = 32-bit >> address range. It's also true that many PCI devices only support = 32-bit >> addresses in memory BARs. 64-bit BARs are an optional extension not >> universally supported. >>=20 >> The translation here is somewhat akin to a type of MMU where the CPU >> addresses are mapped to PCI addresses. The problem here is that the >> PCI BAR resources need to "stay" as PCI addresses since we depend on >> being able to use rman_get_start/end to get the PCI addresses of >> allocated resources, but pci_host_generic.c currently rewrites the >> addresses. >>=20 >> Probably I should remove rman_set_start/end entirely (Warner added = them >> back in 2004) as the methods don't do anything to deal with the = fallout >> that the rman.rm_list linked-list is no longer sorted by address once >> some addresses get rewritten, etc. >>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com