From nobody Wed Feb 14 17:57:33 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZmBQ6swKz518Ll for ; Wed, 14 Feb 2024 17:57:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-24.consmr.mail.gq1.yahoo.com (sonic304-24.consmr.mail.gq1.yahoo.com [98.137.68.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZmBQ4TxTz4Msv for ; Wed, 14 Feb 2024 17:57:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1707933467; bh=RbzmeBWXn/60twKqejHyraerRAaQJoY0gqzcIyAj/Mk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Ny5WtN3ziXLVjX8fnTAiuuFAgfEVyST4d8JN80B9xKuI90J8ghqmzQJWDiHrs+f+ITTzkkBnWfqLNDTGa9zBZGr+5jpGiKwqqmb9UMe5xYIK2Y2ob2x056TCl+yvojts7uzugIoGQO4BLs4+R1jmczHIaXAPfCExlfs2CZ33RsoRAcIl08uqZtn8jmcF6BglsakiQ5rVWgPyACap8jjb+kiVzmfBPXQuKA7+O3SEDOt9yUgA6F6Z+wjsBZsCwCWwpAeK5sREo2wq6ZO7QTjA1mHISO+jwDcqmAdq/rC+opJ9sz801Z2stCQtOarZo9l1V9xGJqXv7Sa0DB2yYTG8KQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1707933467; bh=vtJGLELHLITbF/ZS8UJ5bB2o/yXIkArHJCzIgKYvvbZ=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=nD81xY/2/hdG0oPuB8S8YE6NNnQYVGAmBPEy50Q+gdW+4jCk6vDZ0imJb2t0D8FQSBD3GzFhYivynZ7kW13qnfMN58sNmvGwLA/9Z/PUuKpTdfdKMoX5DICBAmKFql/bZlJzLRq0V4famoCOUqQk2JCgmWsMne/4rTwv2lMB5aHT7ZYMXmWFeUDEPC9VnH1e90yZT+p8hBkZnB6UFLsz/t1W/LQim6XTBg7MibqM5MAzKP/C2SUymvIb1xHI+5CEMUIFL6xAlM3hXubFLcCFZga04IBGXqur+8yHWWtpymQ0tC7+KvC6zhL5SfbKMFFk0UB1Ry6byzDZXEnf4PBdEw== X-YMail-OSG: KO2UigkVM1nKjO9OPgctBmpFB4XReArEo1fpGCjJO8QTnid2LPBPsLYQy_Gf0tx 1QXfFDOH5H779h9aYifRQ28SUe5Gh7Kzvjl0YiKwkmBbghWw8kxhWcTbGH31tJKsjSKZO_moBxhd cRsjWOGUCL0h8tUxyMQvpK_b0JO1tfqpm0AosY6uhikz19iAGsaxsizjfY_hR.oPtMceb5PDQ6mi BtbBwWDtJ3k_WPhsbopmh5bWwLAtbGiMljTvRW6XegVGj4VSoJHi5iVEkkqqQ7w2aUqSIYt0Tiiv 5krGRBEbuq6fATmlMxlaLraUxJDJ1z84c4JY5uJBUAM361DmMaEgTJ..eBuMLG2D8WQTuy5oJLZF qMfr8JXLGgG0B0XfEcPjGaIUxbsZd2z67LWwx8YZwNmkqKjBhS2B1UhCSyjTc6VSWIEIrDI5hig5 I_ZkEc3_ureATzFAadcavcxFQbKV8d6wioRTCgQtVpUCW0v7kwCghuAPRQ3wpdUpvmdBQlE935Hq efLuRnHDXckSisoLiSVpAFEUsXkBlUuYsI3Updq7LzkjTbC38Zh7qDtgc5rJuycv6_1Fq3wPquee D1QuCwuHYbzBBqf7cR49h2ES8KaG__PynC7lVsVEoMfcphw2sYo1lY_dP_PA4oIDVzw6Uy_Th0lC 2dFsK7sqstf2kT9CtJXEetS97SqAsES2lY1Y_gH6tKAfCS_63._lBCtKTSqQQWT_RTHPSlMto5xF 3YqJcBNqg0L96JHp7WzdNe4G4L79dYjJpIuLnZw0RRggoV_ptGgxM3_6jdLUCLx7MVr1D0BoRk5k s0Fxv6Ji1L3YWDbBFSzXEFaTIFQt.hz4L.sOQofhq6iWF183ozG_13N2.nstKTouAYict4hK_aN3 nGEqtB0rENvQ7MBNLcxE2bGGVS4gmBDMa2X3FhP9012sM8GNH2hmGItIeEEDdyOLvxG3CljTQNGA kbbVTKv8b1VT9B.DOmvlNpTdmLMznFAsoiecw1kl3qjOOOCMOkq1QB8p5lFG9K_k2OoRdLtyfEEM xl3T84U8X3o54RMGF_E.iEtQKumgpTuQId3bf5l269l_Zkh0F5Lbqqt.xQkMVWVZt9DOPMPsSNIs Mku1wWMbNEh9EU.cfmSm_A0r1aKU9cXE5vN9kNcGOvH8X3Ym5FS0uamERC9BEwQYhQcaNGUie6Cs PiiBTOq9itF2yqlrLSmMeIDA3EJ5_rxNzTpp965fLz2i.sKtnmAthUhiYaBSxaU5ycCsGpQf3BQM MD2SGZMr4nywFqDj44QCpLqzfMDVlXuZjSPLrt13FWy.g.nrXn7w3AP9ucD4GWGNJLfCkjkFHW9g 0E3iYH5lL8O5DkGTXpIhuAvjIJFlg4L61w_iteakW6s5rz8E41k5FWoP._8Jeq0KAL5sprLec3Nu KGh58FkXJLqImrClNv7Kd1ffhZAdHGxo07loU0Tys1.TL8mkFjUv1pNlBnW55mqmnsuq4iJK5RLu _p_jkZBeqx5Wc6_tLA9FwceQ_ORypt4x4CtW6_uEhZ1YvKSMXGkrssbzQQzVaOn51PwWgkFY_Yuq b7JTb_AaByzmj9NkywTLIBUUeQdPbIlt1OrmOeN5qfZ1OGCYoY.8AqRuPzRm6VDFazWEwwhNVGPG 7kt5RmCNC1rF8suKyb6vU3PVKwNpE7rvNy7.EMjQY1GDEQnMgliW83TAsJwbEXkpSR5erUfYpghv PtP26AGipFnD1rDPkZJZU.4vimyTd7pJ_ejUQXhJdlmY7hiDUdxW2RvS6vdUjKbBVXNHYprgJeC0 bQ.eReO13DNrozSFgK56ZvX4I9tML0QHkaITM1OypcWxqSpqxUL9Ffkm00T2cC5T2KeaWNSbxUpr c6238u191s6Do8BwsjrIMNNPSmaWxBMp4uDdIBlPWc3ZO8iIEq3omvSUcFjHiZ5sq9N7chBxhbTe sTOPEjR5BqhrF.pJVRk_Ve6YbbiJ9VqQ9vOVrltbbpF72bFyQvPrbkHB9hRy3HuAmwrt.0V1n2vt ML2GCrTd_HEUX.5KGsAiqEzzLr65p15wZafp2DFyxvJJz3KLIx1WBbOIMVwkZM4WnsH1PcIOxUJ. dbq2f0uv4nacMLjeyFXH4G5ZNgfJx3notEQZoPDKfRQ6k1ZPpMrHMPREDGWEIhioV_Dia3x.oxuW mtTkIOEWUlB6Q4XIDuzomeZIGK1BDpbhik_wlWqDYFk4_c_m8SiItjk1_fxiiob7AaxtdRNRjF_h ZHcmPj_GxvEeXPzdBG8G8Bl1aT8ocZOKlIxt7ZgviP9N.pfpaTHwQyNa4ZZybKRS0.tfDG8mS18t E X-Sonic-MF: X-Sonic-ID: d036ca9a-cc79-47b5-8043-c2dbc5079fab Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Wed, 14 Feb 2024 17:57:47 +0000 Received: by hermes--production-gq1-5c57879fdf-tnlsv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a26232e89c8c9752c58f7c52f7c49bc1; Wed, 14 Feb 2024 17:57:44 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: request" leads to panic From: Mark Millard In-Reply-To: Date: Wed, 14 Feb 2024 09:57:33 -0800 Cc: FreeBSD ARM List , Current FreeBSD , Warner Losh Content-Transfer-Encoding: quoted-printable Message-Id: <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> References: <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5.ref@yahoo.com> <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5@yahoo.com> <1F704317-FDB8-4BDA-8A67-61CF48794DFE@yahoo.com> <9AFDF067-96E4-4E67-90D2-F40DAF3F138F@yahoo.com> <4C279710-5F88-4295-B1A4-7C395F3587E5@yahoo.com> <3A145420-399D-4EBD-9FF4-18924908AB1D@yahoo.com> <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> To: John Baldwin X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4TZmBQ4TxTz4Msv X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated On Feb 14, 2024, at 08:08, John Baldwin wrote: > On 2/12/24 5:57 PM, Mark Millard wrote: >> On Feb 12, 2024, at 16:36, Mark Millard wrote: >>> On Feb 12, 2024, at 16:10, Mark Millard wrote: >>>=20 >>>> On Feb 12, 2024, at 12:00, Mark Millard wrote: >>>>=20 >>>>> [Gack: I was looking at the wrong vintage of source code, = predating >>>>> your changes: wrong system used.] >>>>>=20 >>>>>=20 >>>>> On Feb 12, 2024, at 10:41, Mark Millard wrote: >>>>>=20 >>>>>> On Feb 12, 2024, at 09:32, John Baldwin wrote: >>>>>>=20 >>>>>>> On 2/9/24 8:13 PM, Mark Millard wrote: >>>>>>>> Summary: >>>>>>>> pcib0: mem = 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: = 0x40000000 >>>>>>>> . . . >>>>>>>> rman_manage_region: request: start = 0x600000000, end 0x6000fffff >>>>>>>> panic: Failed to add resource to rman >>>>>>>=20 >>>>>>> Hmmm, I suspect this is due to the way that = bus_translate_resource works which is >>>>>>> fundamentally broken. It rewrites the start address of a = resource in-situ instead >>>>>>> of keeping downstream resources separate from the upstream = resources. For example, >>>>>>> I don't see how you could ever release a resource in this design = without completely >>>>>>> screwing up your rman. That is, I expect trying to detach a PCI = device behind a >>>>>>> translating bridge that uses the current approach should corrupt = the allocated >>>>>>> resource ranges in an rman long before my changes. >>>>>>>=20 >>>>>>> That said, that doesn't really explain the panic. Hmm, the = panic might be because >>>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the = bus_translate_resource >>>>>>> hack only kicks in the activate_resource method of = pci_host_generic.c. >>>>>>>=20 >>>>>>>> Detail: >>>>>>>> . . . >>>>>>>> pcib0: mem = 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: = 0x40000000 >>>>>>>=20 >>>>>>> This indicates this is a translating bus. >>>>>>>=20 >>>>>>>> pcib1: irq 91 at device 0.0 on pci0 >>>>>>>> rman_manage_region: request: start 0x1, end = 0x1 >>>>>>>> pcib0: rman_reserve_resource: start=3D0xc0000000, = end=3D0xc00fffff, count=3D0x100000 >>>>>>>> rman_reserve_resource_bound: request: = [0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1 >>>>>>>> rman_reserve_resource_bound: trying 0xffffffff = <0xc0000000,0xfffff> >>>>>>>> considering [0xc0000000, 0xffffffff] >>>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 = (requested 0x100000) >>>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000 >>>>>>>> allocating from the beginning >>>>>>>> rman_manage_region: request: start = 0x600000000, end 0x6000fffff >>>>=20 >>>> What you later typed does not match: >>>>=20 >>>> 0x600000000 >>>> 0x6000fffff >>>>=20 >>>> You later typed: >>>>=20 >>>> 0x60000000 >>>> 0x600fffffff >>>>=20 >>>> This seems to have lead to some confusion from using the >>>> wrong figure(s). >>>>=20 >>>>>>> The fact that we are trying to reserve the CPU addresses in the = rman is because >>>>>>> bus_translate_resource rewrote the start address in the resource = after it was allocated. >>>>>>>=20 >>>>>>> That said, I can't see why rman_manage_region would actually = fail. At this point the >>>>>>> rman is empty (this is the first call to rman_manage_region for = "pcib1 memory window"), >>>>>>> so only the check that should be failing are the checks against = rm_start and >>>>>>> rm_end. For the memory window, rm_start is always 0, and rm_end = is always >>>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new = (0x60000000 - 0x600fffffff) >>>>>>> ranges are within those bounds. >>>>=20 >>>> No: >>>>=20 >>>> 0xffffffff >>>>=20 >>>> .vs (actual): >>>>=20 >>>> 0x600000000 >>>> 0x6000fffff >=20 > Ok, then this explains the failure if the "raw" addresses are above = 4G. I have > access to an emag I'm currently using to test fixes to = pci_host_generic.c to > avoid corrupting struct resource objects. I'll post the diff once = I've got > something verified to work. >=20 >> It looks to me like in sys/dev/pci/pci_pci.c the: >> static void >> pcib_probe_windows(struct pcib_softc *sc) >> { >> . . . >> pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, = 0xffffffff); >> . . . >> is just inappropriately restrictive about where in the system >> address space a PCIe can validly be mapped to on the high end. >> That, in turn, leads to the rejection on the RPi4B now that >> the range use is checked. >=20 > No, the physical register in PCI-PCI bridges is only 32-bits. Only = the > prefetchable BAR supports 64-bit addresses. Just for my edification . . . As I understand, SYS_RES_MEMORY for the BCM2711 means the 35 bit addressing space in the BCM2711, not a PCIe device internal address range that corresponds. Am I wrong about that? If I'm wrong, what does identify the 35 bit addressing space in the BCM2711? If I'm correct, then the 0..0xffffffff seems to be from the wrong address space up front. Or, may be, the SYS_RES_MEMORY and the 0xffffffff argments are not related as I expected and the 0xffffffff is not a SYS_RES_MEMORY value? > This is why the host bridge > is doing a translation from the CPU side (0x600000000) to the PCI BAR > addresses (0xc0000000) so that the BAR addresses are down in the = 32-bit > address range. It's also true that many PCI devices only support = 32-bit > addresses in memory BARs. 64-bit BARs are an optional extension not > universally supported. >=20 > The translation here is somewhat akin to a type of MMU where the CPU > addresses are mapped to PCI addresses. The problem here is that the > PCI BAR resources need to "stay" as PCI addresses since we depend on > being able to use rman_get_start/end to get the PCI addresses of > allocated resources, but pci_host_generic.c currently rewrites the > addresses. >=20 > Probably I should remove rman_set_start/end entirely (Warner added = them > back in 2004) as the methods don't do anything to deal with the = fallout > that the rman.rm_list linked-list is no longer sorted by address once > some addresses get rewritten, etc. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com