From nobody Tue Jan 24 23:13:22 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P1jT25bLWz30xPX for ; Tue, 24 Jan 2023 23:13:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P1jT13DDpz3NhQ for ; Tue, 24 Jan 2023 23:13:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PHBJblGG; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1674602019; bh=HD+0B+8XZgSgbgL4tyXMO3SHYxyFxIkHJgyQA13kRjU=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=PHBJblGGf+0NQXOOzrwMZgQv+R8ODVVY3tWQcsBoZUq16MQnv7XRaM7XZwij6uOkj3PPyFEKjL7H6qDujQRy1zkZ72Xz1nVuDPQUnzz2ARCWuysrTtnsdE9KAJUgKNsKpIbA9TlkaKxMa/GbtS0Lh3oAwAozREZBTbSQoUSPoiUrEPyeeQVnaYyLIhFMxde71Henvws+NKaerk57HVgZiMceeEXngrrefJ2V5HIs9C9ThILdj4h/9UtPSolCcBzrytoaNGzToICQ+AulZnIJM/c7mdU0rkQLihtrwbSIgV9T0cZ2hjiUudANBNdCdbqTV9/RMRSB9q1Ohoat+KU4Yw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1674602019; bh=CTL6SHTqFM4Q/+s0xVH8KK4oa0IjOWaLIuTtE9bpZRG=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Xbs3RIjjhtJ7piqjgiIwIU1lK65BAgsw4vP2oFhH3vxFvmuvC2TJBo4TvqT4p9c96xmxts+QXH/RWMrbUrOCS6xnyi4FN/10/xoLiDErDr2YxksS8bOqXfTExhS4+NZiSv8Y6lGAga/EvEDqT7FH8a57EEqctdXvEfAqOC9ru7oxX+FdPCC6BitcEfNwxAIwKseA8fChjM5o5X3hAzOq+lQPtHq1Jyl3y4lQq/BXhgv8Ef24YtyX0W2tDB3i4C3KudQtSvSpUAiUSvpneLZeOjv5J/GFs5nxn7EqXu/701SubyrBcdMOVpbud/A81yTF7GmOv+7UwptDkSA0F3QvWA== X-YMail-OSG: M2xr3SMVM1kGWsnovAw2daSNXNPKwwNkzrY9xb3yM5fDyMFi_ZmBrckPQvFB.9p 8a.jwFmoKaPiMeKh1MPi1wQOn0UIdVgUzvG5B.y9O44KgO5jWpq1k6HGILnKhuBQgqmKcjUnaBX5 aYl5rcm3MK5a8som0zll9aqRu03TY_gPg633BpLJeqsoq3bi4RD256Wtc_qb5AIOMllCqyuQuNyD cYxkD0eO2ReoA65ZgHGgtk7khS0552rx7rjNzChiH3F1ERMIt5p.9hlYP0lQrS8C6eg0Blu5qUNQ H188NidYFTk1MubcC8.zIFx2W2zpWgvEtGAELI0g7pUWgApWo4elyty8joRhbVbaWBXCQ73d9Ir. ttrazQPv23bdNDiLQ0Z3IKVDpo0jvN6HOBO9IjGB1tH5w5vGuT1xT3X7h4D1KtE0OMroaejf7z4s CCQOmen9jP_0Q1OZNI7YqgzGJkaz3xl1CYENqEHR5rg1crstU6It0FmIb6nzfPXT93cR3huTSlUF 8ftdpzqNnAK1s_j80sMP1E51PyO..aFS2XrCGFQPB2yUojhWAnrJamLcgsi6xrfLlnnae35uSbqT puSnBzb_Q_XocOm8eGVhu_ldTTjH2Ax2JFyZf7p5eQVkmAmv0XRGHT3.gn2yrbZ6WeEAOb7O6qVg zdCTWbpl3XzSyLchlOK3heQ7dEIcmyohvZKYhGHfRqOHRnF.PVhlxR.Ve8QsLQ9.jirkjZafy2YM HlA3p5ZVRl4mzb0HRagX_GaJS1Orzp9LPW.iHvy_n.EVUyrLVU1OM9vXio3VVsYKGIl.e7dN93nI jOmv8ntQN3l7..1n4aSkOnJay2cyekwOLRaPYTq3MHqtQBdx62CEjDDji6X9Euepoy52P5ZEqE42 9WZ8uvNOEvX8MCNpUmBMIXfdjFkbJHTCOXUE2fmKcwv7l7dDcQQOZXk9D4aqhJXQs1daukbZk6Fp _zfd0wXCdU_cot4E3KdmLtvC5As3uN0AKQlvtv94VTCc_KZ6WPpNGhoshMMBeiwyLkWNfjQN4RIa cgTEp52t5D8dUhKDDBIWMMYISNqnSsURE4Cr4d2Kp6F_vMmvcLvkaRHYOjTO8HbPeWIQoe_9cb00 qUvC351UaFONxsid5BdNK1ZiAhYWwGZ_pw4nd1pn03.gSR5OmNgopg7l59OAA8uyc6R_x36QlDIk ysq4GXyDwzmHCDSgFuD0IbKwouum.BYH43SLjtcyEmftllG8ggSGJIQQPGUAWdywsSfIBmNrW6wz VUAC.7C8hzKoCmR9U0TaT6zWcXZkWfRkH.w0wOF5quYg0Iva8l6W9RGKfFks5eB0Ywi9GqT36pd_ I.0USbgLiy7F6hlTqg3W8e4OdO4TxUvwMeB6VfgcV6hTQv32pXCgaGiO1VSYkPkwVvigJGmTPvyk yFbxCzba2XMbLzoCSGau7njOcbZ9EOPYjq7nBNE.y.EH9GQp7gCsEV8UAejsbdDf9hxMeft.FrbB f529nI.nnkdfI89SnwtzzuFtPXp6Sm1IDLD52qg8D6H9Nw_rrrucK3e0saiyVtlucXFLY.AIb7e0 RUYXrxSGlQBpU.bWQU6QrnqmE15ClNh4E0E7wLAWbtVKQSAxmjghqYmQI._Qf8Ro8K9zpeahDkIM 7QAMieqVw2Nkiqnz.Rx2BO5KWERViglbUQLyaf2JZLUICJCYmNPW4f27KVvI7ugYIWwRVFiDCQjJ IoxK1KYgsnSpcjm_1yNml2vnG.EdQQp.Pz1sg0AwEvrQRbDaUYixw1P.PWPVp17pT8Bp7ce0Uovw BoJQ2P2dAhJRSuPG.8Q55UtekRE28NA1uHD_7XWyFk6ENiHdzisEnQ94B5EHrrXw48IeP6jEi139 AcNV8QR36oAoDBUcLXnMdEbxrUwZSsLPEa._BWH3.7or_BBca3.lOD_.JzILZwNgZTbrdqwZ1TVi c6e_Ike1ZA4PKE4dCSUusTeaD4_hxTZ4erWaDJe81b7ZioN12JDayz4eYIY4Ns40qKrYQ0T3eUko 2fmL2bWzrrxt1hoiiGK0VfsoXN8O19Wjhjd2ujW5g32N2WZGjbQSZmqdvgP0chMz8av.kJuhFCDh AAQhs8ONq.RJ.iHpRQ7hfhyaRJfDKJ3NlrS778B.rww14vEN5gqkKvRPQwdYYrW2JmiagIxPK5mv aeKwnHbpoKO99Iua7W9Rr3pkGrtAevWfljWi1RMrfWN7jkROs3MFVqL9JsUkujZDTMMAuqaVHCYi qKjJLaquXqehg4GRNdesT.VYWD64tDOtjyx6KxOMrL7F.zLaSxdIAqjHzDkxo9Q-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Tue, 24 Jan 2023 23:13:39 +0000 Received: by hermes--production-ne1-644674576b-9thps (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3becd1baaefc510647da10edf2b3c2dc; Tue, 24 Jan 2023 23:13:34 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: FYI: RPi4B "C0T" vs. "B0T" (Inbound from PCIe 3 GiByte limit for XHCI use) in various Operating System/Firmware combinations Message-Id: <69C5120B-B0F3-48A0-875C-1CA8E29F78B0@yahoo.com> Date: Tue, 24 Jan 2023 15:13:22 -0800 To: freebsd-arm X-Mailer: Apple Mail (2.3731.300.101.1.3) References: <69C5120B-B0F3-48A0-875C-1CA8E29F78B0.ref@yahoo.com> X-Spamd-Result: default: False [-3.19 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.69)[-0.693]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4P1jT13DDpz3NhQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N [To a notable extent here, "C0T" vs. "B0T" RPi4B handling is a stand-in example for the more general issue of Device Tree dma-ranges support for address space translations (or analogous in UEFI/ACPI), not limited to RPi*'s at all.] In exploring the status of handling "C0T" RPi4B's (and more) vs. "B0T" RPi4B's for the XHCI related context that involved the 3 GiByte mistake in the PCIe wrapper logic for the "B0T" parts, what I have found is that . . . A) OpenBSD has had full Device Tree handling of the dma-ranges in place for some time. It supports "C0T" XHCI use in a manor that avoids needing the bouncing. For "B0T" it uses the dma-ranges and spans the 3 GiByte range as well, no longer using a hand coded, much smaller value somewhat under 1 GiByte. OpenBSD normally uses U-Boot. Some (possibly outdated?) material indicates that RPi400's require(?) https://github.com/pftf/RPi4/ (EDK2) use instead, warning to probably use a specific known-working release, 1.21 as I remember. So see later EDK2 related notes for if that is the case. (I do not have access to a RPi400.) B) Linux has had Device Tree handling of the dma-ranges in place for some time. It supports "C0T" XHCI use in a manor that avoids needing the bouncing. For "B0T" it uses the dma-ranges and spans the 3 GiByte range as well. Fedora uses U-Boot. RaspiOS64 (my abbreviation) does not (nor does it use EDK2). Both avoid bouncing for "C0T". (I've not looked at others in operation.) C) The EDK2 implementation treats "C0T" parts in same the manor it handles the "B0T" parts. So bounce buffers end up being used for above the lower 3 GiBytes. Thus, even if something using EDK2 could support avoiding that bouncing for "C0T" parts, it would not avoid such: only a smaller-size aspect is published, using an identity for the address space translation. But see below for OpenBSD mixed with EDK2/DeviceTree. D) For EDK2/DeviceTree, OpenBSD actually undoes some of what EDK2 provides. This was for OPenBSD to support more modern RPi* firmware to support more modern devices according to comments in the code. (Some of what EDK2 does was to avoid earlier problems with OpenBSD and Linux when used with EDK2. So much for a clean firmware/OS partitioning.) E) NetBSD is based on UEFI/ACPI and, so, is an example of (C) for https://github.com/pftf/RPi4/ . I'm not sure if NetBSD would "just work" if EDK2 started to allow "C0T" parts to have the PCIe space vs. CPU memory space address translations that are involved in avoiding bouncing. As stands, it might be untested because of EDK2 avoiding involving such space translations: only a smaller-size aspect is actually in use --instead of a address space translation with a full-size. F) FreeBSD Device Tree handling hand codes a "B0T"-like structure and ignores dma-ranges completely. If I have understood the PCIe/XHCI related code correctly, there is no general infrastructure for supporting the generality of the PCIe space vs. CPU memory space addressing translations that dma-ranges support would need to be effective. More than just decoding the dma-ranges in order to initialize an existing infrastructure would be required as far as code changes go from what I can tell. So, if I understand right, any Device Tree with dma-ranges indicating non-identity address space translations currently needs to hand code a no-address-translation alternative unless they go to the effort to add the infrastructure. It is not just an RPi* issue. Also, for an identity address space translation but a smaller-size, this too is ignored. But it appears that the existing infrastructure could be initialized for this case, if I understood correctly. It did not appear to have to be handled in RPi* specific code. Note: FreeBSD's UEFI/ACPI support for the RPI4B used to not handle even just the smaller-size aspect when the translation was just an identity translation: it was set up such that one page past the end of the smaller-size ended up being allowed to be used. That lead to corrupted files and such in deliberate testing. I've not checked the status of this in recent times. === Mark Millard marklmi at yahoo.com