From nobody Fri Jan 20 07:23:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nyrb83kg4z2skyd for ; Fri, 20 Jan 2023 07:24:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.gq1.yahoo.com (sonic309-22.consmr.mail.gq1.yahoo.com [98.137.65.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nyrb72qlPz4KKv for ; Fri, 20 Jan 2023 07:24:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="uS/W00iJ"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1674199441; bh=q9JEipvvbkkOPMKHY2OaI9gZn9f07ii9YJDXiIep8Gk=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=uS/W00iJZSbXC9+8vhAExu1vTvMfoE6M9MH4pUZyHYkwcAZ8m9EreNIfm8SwesTswYH1Q5JaXnDjI0YJdOUQBBz55hcSowRpDWjTzjmHyCmEAS00HsJKlqe/cneg5Sncesf91OZBQtAqYNyMSL5gAOqFxyucPzPpdBRKIVVcJTQQ/T5/4wCa15U1b6ZXsP1DBq4nh9sgsoCJLYdEIpdquzSVzz637NjCCMBAHL36qpgzv64Wtsb+yxVO2zbCVAe3QfoV2Eqb1dJVXn50WxEqKKRax9lyBVEKXPemT45ox9a6TB5tGskj1pdweKlm5Y/UrLdbnOJUGhsYC49NyzZFOg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1674199441; bh=09PADtWju+EwUAfzJ2QiBztn1CcgnFnkVHGwaWajDO+=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Fgg0vgBKpZQOk+qjonQdgvUbxFzhRHElYx9Y3XGwx5N/piucXhFlIJezsiq2M3+4JZe69fYtx/vds3u8tGhLfqNmUNQCoEA8thUIftlTFl2LWv8Lhw5aOxp+f5Sfp/7gAzYanOGq9EodRT9gEjvXi5sYbrGI0wlyKtpWjmnC0bSaTnXOeYmBUbHbhc5PvVJaljmhnvbY+TT/291HbUQZfeanH37dKKDMPseQ7c35DtFJiy3lfK8hMZMBFuEsYGhLUkBsOn7t52MA5VlFn7x9uszLGvWxPP4PwhCwq7P4SmS/2VSIy7r3sLX1QtTFYPdfQLPsNo0zuxJRjJVmzPAIGw== X-YMail-OSG: 8FVSyqMVM1mhhXpJek9kq8o80McR5JPHS2IRX9MW7.oj6d7NspFPCOjSH7l_hvs S.2mEx31DMyGhQuzDdLM2PryrinhdKo8zycRqtuJohy6MahtVwquLivcSkgHyUGpa141lPFceHKw Rrtd2DdAZFdTiPIld9.F1pZ.ExPNA0RclsEOHiBYm9DfIbPPaQlP5SHe841rILTtB4IHydw6n3bE U9wIUYcZliVK6zCWJne3eyURJivOmVc5wBB_alHDq51mER.80Yglc89UMba2tplDLFz3ae97MLgd HoIV4UEsx7y65AMJPHQm3BeAyV0Whn9MNXjAa3XcO0iDNFiFW85lqjx5j3YIj0LA5alVetJjs978 yr4OOztV.T874CrzZj89Vu.t.Deq0g5XpCbUbdpRq9FXEpXjeNyvyy4dMOB2ZcDN.QeZWP_7iPBy p48mBbhkTAVB0SHgL3MIX6R0N7HuaJs6z7JkFapvu_wc3KH_hIQ1fKEhXQXvrTPUPZ0jV6EQTfwI wuha2CN7MWsTv.jEFCPxbLfyA_FBQMblDaaLyrhTR0IszCdjTJZuJ0NmCOZImokTJV4NNkhZSJPd 0l.nBvuuhlE9rLILABYxieC7pa.76gn_wwO_vOc.AupAINGFb051fFYFJuMV.o4PKkjTuAPZk33V pq7DwdEArYXJKjc2Tz9zjRu9PjRqPFsNxYj6ZsCSHQ.CfmI.r4XICx6OCSYjm7CHCBsakPd.FZt_ KTEpIEnc7hhqTX9GS2rzc8PHhogbvZnbuXxwyXqXY2m2wBPJalxvE0EpA_2uGacN9sQdBoaOTKYO u5TuAPhJdI_kJgYpzjfHb_KPUBz2FaygByHmJ5j838WmNACI4SIN4.kFezc9Ux5Eoq_usm4e4Ukn jdzNOulm._UKVLEt82ijCUYBpr.3j24u0gfRjD7p.V8O4bsG1jUchy_56jiuN_gQiqDxQVDUUEen Hr9I9Cw0Txj4P_Q4vwTU7cnbhLPuNECZ09rN28o522dT1zTRb3wBUMqul6ynRUfCCLHhFzinr2PP rrnORQWodtq7zQSo7g6oijboc1K96RzyJjMYlaXnujBxHF523cblFScsnhx6D5Z_h6AyRL5mijUw GeScwRci3tM9oNNIUQAbpFl4.r2pCvotyzLXdlh_4JlA86glfvfsKXpHrIC2e1Qcdv2Y8GwhbDVJ DFmgmbm_h3rs0LGoLs2hrIf0LcFxU1bmtVtX2F1HdSJ1iT4d6IotjH9n61SyWVyxDmOu0N6VRmDc 1I92dXoRps3rZExMqNBra4j89cw_kQftHp0PcmkXND7CqELsnyRpl6TncWEv9Cg_sA67JzHuvYuB HZWdlazBUGPtzQ.lYEPt9.w2D5rB3sdl_VulIaPhfJ8iRM.r9QBY6DUYOAT4822D2vhLTSW1i4j2 XbiodSz._lzelgeacHPiRqnDUyeNLU6w7bvF.43PNImNACLRGoE8OcfMMijWQzPoy9R7e6RqSJV. fdC04HHhnjB8g13Gn_adWKWVohKJvbG1Axr5Xy6fbpqBNoLe7q1l.ukHYB1e5Dmt447hmfu9qlAu QAOTcTeS1.cmeCQxam2jsy6xzcb3ud8T1SlwsqVktuVwLzMr5HGldezLt8uwIY7fYVfuZUA84jHq fRP94UWNHVuD358hL43EVKnR4muQZlP9E_kiZ8FMkt7eyIcCNu3ivAZQ31uKIRU1IAyJ7PE6Tusp FMEMPFlT_XGJzIT39tgvlW4BlS9Pfjbu2eIgvfZAhSM0qbNWPbswnk5.6_9wnecFYcBwhP6nEiZo eUcRpEXxxdgQep4SQX59RpzfCL4Wls5XY8aFR3qn8scjsWMIecocaQ0hJ42uaEAmspBHKXO4XdnP 7MXrpzJvh9fph7v5XvYp3oD2h19OtzPdMacZtzxZ.hZFjk.gr5Z1ebL_COp5y98hBq7W8qpP.JLf VE1brB3w0wMaOr_QsTunD1FfhGiSHgzDmXxczf_Cia61eL3VOHoOkBhWO1ZsGcpc0IREtVEnRYsx 7giUJW7ySzbl3i8YvShrX.iYo9hxZQkPJgsWMuoavebO6IWBmzhfjZV5ASZP8rbvMKKOYTyG7S6z SByS7Mt5RkbWPTJfbtdfPhLC3wdiCuBGVpmqmZOWE8FyYuqs9_0tqaZ.m6tDre4d52vk2aOoU0pT tojW.b2B6QnOzlKo05DtwQ5JFxEZlqBD2UEMabk6na8uN3.u2BE5m7KweeHseNLxLMgLIAoKeAP. z7zVkcYM.tWF7RItRAukfoNiXu1CTVD34uSm7nQi.wwZ5ZJAqn0.hfWDJpRBqkiORy_qdHbA- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Fri, 20 Jan 2023 07:24:01 +0000 Received: by hermes--production-bf1-6bb65c4965-lwg94 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID ab0aeaf2d470d99a9475b164205a0f85; Fri, 20 Jan 2023 07:24:00 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Any hints about how to get (a RPi4B) PCIe/XHCI combination to deal with a "PCI addr: 0x400000000, CPU addr: 0x0" from dma-ranges? Message-Id: Date: Thu, 19 Jan 2023 23:23:48 -0800 Cc: Hans Petter Selasky To: FreeBSD Hackers X-Mailer: Apple Mail (2.3731.300.101.1.3) References: X-Spamd-Result: default: False [-2.48 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.978]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.148:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.148:from] X-Rspamd-Queue-Id: 4Nyrb72qlPz4KKv X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N For my own edification, I've been looking into having RPi4B's use the dma-ranges device tree information to set up the inbound configuration for its PCIe/XHCI context. This would allow the modern "C0T" parts that have more than 3 GiBytes of RAM to avoid bounce buffering associated with the only-lower-3-GiBytes works PCIe wrapper-logic problem that exists for "B0T" parts. For a "C0T" 8 GiByte RPI4B the live Device Tree sections involved look like: . . . #address-cells =3D <0x00000002>; #size-cells =3D <0x00000001>; . . . scb { compatible =3D "simple-bus"; #address-cells =3D <0x00000002>; #size-cells =3D <0x00000002>; . . . pcie@7d500000 { compatible =3D "brcm,bcm2711-pcie", = "brcm,bcm7445-pcie"; reg =3D <0x00000000 0x7d500000 0x00000000 = 0x00009310>; device_type =3D "pci"; #address-cells =3D <0x00000003>; #interrupt-cells =3D <0x00000001>; #size-cells =3D <0x00000002>; . . . ranges =3D <0x02000000 0x00000000 0xc0000000 = 0x00000006 0x00000000 0x00000000 0x40000000>; dma-ranges =3D <0x02000000 0x00000004 0x00000000 = 0x00000000 0x00000000 0x00000002 0x00000000>; . . . pci@0,0 { device_type =3D "pci"; #address-cells =3D <0x00000003>; #size-cells =3D <0x00000002>; ranges; reg =3D <0x00000000 0x00000000 = 0x00000000 0x00000000 0x00000000>; usb@0,0 { reg =3D <0x00000000 0x00000000 = 0x00000000 0x00000000 0x00000000>; resets =3D <0x0000002e = 0x00000000>; }; }; }; . . . xhci@7e9c0000 { compatible =3D "generic-xhci"; status =3D "disabled"; reg =3D <0x00000000 0x7e9c0000 0x00000000 = 0x00100000>; interrupts =3D <0x00000000 0x000000b0 = 0x00000004>; power-domains =3D <0x00000013 0x00000006>; phandle =3D <0x000000e2>; }; . . . Showing in boot -v output like form, the pcib0 would look like (showing both Inbound and Outbound): pcib0: Inbound pcie dma-range: PCI addr: 0x400000000, CPU addr: 0x0, = Size: 0x200000000 pcib0: parsing FDT for ECAM0: pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 Note the: PCI addr: 0x400000000, . . ., Size: 0x200000000 vs. PCI addr: 0xc0000000, . . ., Size: 0x40000000 which nicely avoid PCI addr overlaps and leave the Inbound context with a size of 8 GiBytes, matching the RAM size. Unfortunately, for as little as I've figured out, I have to override (just) the 0x400000000 and use 0x0 (to numerically match the CPU addr values): pcib0: Inbound pcie dma-range: PCI addr: 0x0, CPU addr: 0x0, Size: = 0x200000000 pcib0: parsing FDT for ECAM0: pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 Note the: PCI addr: 0x00000000, . . ., Size: 0x200000000 vs. PCI addr: 0xc0000000, . . ., Size: 0x40000000 With this they do overlap. While it boots this way, it is only because it has not yet run into the overlap. (I can do USB tests that lead to a crash.) It appears that I need a hint about how to get the XHCI to deal with the "PCI addr: 0x400000000, CPU addr: 0x0" combination, translating addresses between spaces appropriately. I am already setting the PCIe RC Inbound PCIe related registers. But I expect that there is more to it in order for the XHCI activity to use that mapping. The hint might just be pointing me at something to read that I've not managed to find. For reference, the 0x400000000 context starts its boot failure with a "Run timeout": . . . bcm_xhci0: (New XHCI DeviceId=3D0x34831106) bcm_xhci0: Run timeout. bcm_xhci0: XHCI halt/start/probe failed err=3D18 device_attach: bcm_xhci0 attach returned 6 . . . =3D=3D=3D Mark Millard marklmi at yahoo.com