From nobody Sun Dec 18 19:10:45 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZsr91JgczswgM for ; Sun, 18 Dec 2022 19:11:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZsr726Vwz3Hkc for ; Sun, 18 Dec 2022 19:11:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="tJ2/qIfK"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1671390660; bh=WqzZ8P+kElKLruSlxvAsvI8EUJrHIPr7GCUvQcWXitE=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=tJ2/qIfKnbq64Kh7VnE7xP25z1U9uqv2NX606VNg3enJPlrnym/hWPs74ERrtQS5lK4Vy1vV2j8D3zH/95uByyWFwTqfUFvixKAJS35XjLjhmUPQ7JSowdtrCQXOD1umDK3DS8/2Fp/T9MFntfl0+FSJEnIIFNoaN804789yPytvaW5zWnqDQFIgkteYT595lL1Q8W8E9MHWbUOM3BKMCdq2hZO3ry8EE6zpjuW5m2BVVpP7HBd/8TiJSuOhD9TFlwB6l/hpFYHubr6kcD1vtGjNaen3h7DkSS04u7/btEFDeCwkHTzuzo1KF0+C9aHdlsyv5/C9vE6MTYlIK3Y+EQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1671390660; bh=vIJtL/ZTu8js3hKqv51w6GKF3FAJIKPWs776Xu7K/dG=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=dRz1HgVMb8qIMO1GFDqJDEsQ0l/emaxBxSQhTz9SXfysubWNQCUiXaaRwN6iPbOxfP9lBdfgYnacnkMNPvaoIRozc1CHDOIIUZSdn9Nt2xDKC2kdLln+4nQpEXZnvmDhIVpKte9cPoeKf0UdW8MJ+uKhnhWScuekTmiRHHLb10xwkk503zAL/b2hLRFgDjQHU076anqEhdu94geX5WC7xAweMG4/hYB/ehgUejkEfNDp2pF9ZNrSsPvbkAuhhzMoOCEotQqT2EvdBFoGbCFp/8q+KSF7utBBErhE8SfHTzxeQI9llUrYkcW4HMUZrPrLAJZ/P0EF26z9rk1d6arZFw== X-YMail-OSG: WXDI4pEVM1mzNjAeiHITvIJ8dVW4WOikqWGNmPFhGCTFuIqrs8RSJPDC3W2wW_. OLOFT.JcVZslrkUwv_ETjVcoceWD4cgmF6KRZzmOePIg1kf6lpgJWx94lrPAwYLRhhIt2E7gwESe bf1QJ982OasGq._SNEJUj_H9.BUAL0EbXRnr.6osw2h1ngxrgbm4d65GTs0fKgycdaG_jeLdPSoF rk505eXmAv_z7AJMsiu0FUgFnI3RhXjXYPUf6hVranXdM78xEXDScmlYJt4WKBkd4sqgMUafNtce irqQfoYGRlOor4L_kbnAXYNgwxqvUCe5dQyUz.35nH4tGb5C7NJJJuMOcZwuBLL16I9JNhZ4JgKF 3QPMubkg8rTozj5dSMXoPkAmRmWq0jNXA.Z41S7GgLvUpjCgdKVHrm2Ga7WsydlCaGUiaHfjB7eY fshUXR7EN4e6r0D31B9BMw82m6ywzw88ObMOLf.t3Iqe.ySda0R7xG7RejAFStTQ9fFVIJzBmdt1 .HWmLPBZDFmxTfydvtjFZgA7jZcpnK.6vq1riXJhsnWTsQI.JELnkDZkhKzGYl87_c.W4Yn0_9nr VR_0nBT3xc847JlahyaDjqSd.gjaH3HeM6p8TZYuKSQkoufIh_Z5GKkZGxZhBgDXvdRjXmkmqGN. zyyxQwjGCmVG_FaodBgLkXBxvm4cfGEUzrgvUTSK1LOF2vPHzwY401utTfNGdLLZ97OrX9VesVeF KWnMa5an.7hc2b0ofBXBsgLc.dtGoDLWN8Ye_osxqbyIe0tKIyekjuJTJeh8lYr7ypm0C4uI4Qvv E0gAiGXWZtzI0m8753_NiJJujRx6t00xJGKnhO1E2ltVLlwHiLgVhQve3nGDVREdXbDLGimmdt6L cJxqKLQLr9RYPrwp0CcLIRGx.b4IJJpB0Vql11q5DeTM59xN.I9HMRb4CBt3.ty_4tq2TgT_ZDHH Gs88FqN4QRQ5IBhh5xktgWgUqo_iNXn9kqTJTGo3Kc39mjrSLOECwCGr0hmy5SyfhrbOf6NMr31W s0XRT4gVu6FDWAHPd1OR.7ZOQrCw9fNlSh6NgjoKr3kdw6WuYeep4pNB60rck5eTfcEogApRORAX .OhuVq6Ay1eAHM_SZ7LZQCkAkPe2rJviAnfedYC_zmvLoCIbW7EKTg8brhDXGjJsGOo9REfhznwm AXRAd0dO.iU7ktyRqbGNcz4tyiy17DVVCKOwLTWpVB5OOapFfq.ttajrR3Pxgy5MfZBCHspNFqqH RdZ7BtDI5x0HV23u_.gd7D96JhtkKi8skVC_JZAElgmQlNb_PB13XNaf3zM8SN0msjl5Hvr7Uuft k_9df.u.o.Yv.ob88Xq9VRA7.tljWEsorboUudsZJCn6Iee3XFg.x4wudZcPKLDMuOg2VZ3D.BQI PuliA4c1l9_sUQ2SZJMZF9EOpmLcwExKiTW4tD9JjUrvk1dthSr2BkcYDg8ZIoZcgAOnE8tONsw6 dE7DD_rNNXS9cNDmsxyZtNmgl4.k7c5MnwQISdf789GqTttf.aIdUlKxWP1cXQz51TmS_t0nhBdY 4WJxSiE5biQFJkQNskdX7ytt6fWYaTZIjI.Pkla5EopQl4rYrKj8LCSqO7SaWhbsGIdG9uIPGaOI u56vCf7sXHajb3Q7wxeNLiV5jfCYgv1FMH1HFrHm2LK9gkTI1ngPRHHAxiQp0tZKe.qDD6zIhREL 9JU2AP3wvFMi39EeHtYYQePCI4BfZ0pdHUfQdBm1wu4k75BEHV21b3pMEmNu2HWN5REGKuLs0hg6 nKGLDR6jmNqr6KB3uffQvHnkfIznFgJwI6JVygL_Yu0O9cvOSysR9QpQ.WTrA9INVmKXIAaBr1is ajxXxXuJSs5rlTEQYFgTNp.OC0qcjNyRsAr48zgbntMKs31JbcfV2ZQgDwFyJkN9sAsBqGQvtNix TiDyQ4nK2FfIcBVV0eMboKL8IhzL.atbI3Zu.WznH1IryhNFjI31KJA6FAvWskWz7UdQMBuy8xu4 0WDgEFY21bhnau8sRn84WlNbXxJ6pOjp2bwtXyngmwlnp.3d0tYsaw7BekiQFb9LxtsZ1XjDQU57 O1cFIcDl2av.nbVJLsAeqP6SXW1K5Xm9e5imDa14du2cY64xzqJ.07chycDmaPIYz9_NwvmJjD.J Z8ivsBeaRt2fm.1hgQ.feCWtX5p3Z0l6AVe5u9PW9mq9Vff3C0IVxATiIcgqwToaC8cqtIZohRc2 5pTEaNR4vjYxUuh9TSCh7.mHZIXqs4OH9IsIzMiG7JVwWFzKW3Re3czfwaK2dVuM- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sun, 18 Dec 2022 19:11:00 +0000 Received: by hermes--production-ne1-7b69748c4d-7hwnr (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID f5f6c49f7d75e3c4fe4c0ce493dbe8ad; Sun, 18 Dec 2022 19:10:57 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: RPi4B boot -v "pci0: failed to allocate bus number" and "device_attach: pci0 attach returned 6"; more Message-Id: <7DDD8F49-B02C-484B-B92F-D894502B46DA@yahoo.com> Date: Sun, 18 Dec 2022 11:10:45 -0800 To: freebsd-arm X-Mailer: Apple Mail (2.3731.200.110.1.12) References: <7DDD8F49-B02C-484B-B92F-D894502B46DA.ref@yahoo.com> X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.95)[-0.951]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.205:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4NZsr726Vwz3Hkc X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N For reference, the example context used is (long line output split for readability): # uname -apKU FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch64 1301509 1301509 (So: not my build.) =46rom the boot -v capture: pcib0: mem = 0x7d500000-0x7d50930f irq 80,81 on simplebus2 pcib0: parsing FDT for ECAM0: pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0 pcib0: Bus is not cache-coherent pcib0: hardware identifies as revision 0x304. pcib0: note: reported link speed is 5.0 GT/s. pci1: on pcib0 pci1: domain=3D0, physical bus=3D0 found-> vendor=3D0x14e4, dev=3D0x2711, revid=3D0x00 domain=3D0, bus=3D0, slot=3D0, func=3D0 class=3D06-04-00, hdrtype=3D0x01, mfdev=3D0 cmdreg=3D0x0000, statreg=3D0x0010, cachelnsz=3D0 (dwords) lattimer=3D0x00 (0 ns), mingnt=3D0x00 (0 ns), maxlat=3D0x00 (0 = ns) intpin=3Da, irq=3D0 powerspec 3 supports D0 D3 current D0 secbus=3D1, subbus=3D1 pcib1: irq 91 at device 0.0 on pci1 pcib0: rman_reserve_resource: start=3D0xc0000000, end=3D0xc00fffff, = count=3D0x100000 pcib1: domain 0 pcib1: secondary bus 1 pcib1: subordinate bus 1 pcib1: memory decode 0xc0000000-0xc00fffff pci2: on pcib1 pcib1: allocated bus range (1-1) for rid 0 of pci2 pci2: domain=3D0, physical bus=3D1 found-> vendor=3D0x1106, dev=3D0x3483, revid=3D0x01 domain=3D0, bus=3D1, slot=3D0, func=3D0 class=3D0c-03-30, hdrtype=3D0x00, mfdev=3D0 cmdreg=3D0x0000, statreg=3D0x0010, cachelnsz=3D0 (dwords) lattimer=3D0x00 (0 ns), mingnt=3D0x00 (0 ns), maxlat=3D0x00 (0 = ns) intpin=3Da, irq=3D0 powerspec 3 supports D0 D3 current D0 MSI supports 4 messages, 64 bit map[10]: type Memory, range 64, base 0, size 12, memory disabled pcib1: slot 0 INTA is routed to irq 92 bcm_xhci0: irq 92 at = device 0.0 on pci2 bcm_xhci0: note: xhci firmware not found. bcm_xhci0: note: installing xhci firmware. bcm_xhci0: note: xhci firmware detected; firmware is revision 138a1. pcib1: allocated memory range (0xc0000000-0xc0000fff) for rid 10 of = bcm_xhci0 bcm_xhci0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xc0000000 bcm_xhci0: 32 bytes context size, 64-bit DMA bcm_xhci0: attempting to allocate 1 MSI vectors (4 supported) bcm_xhci0: using IRQ 93 for MSI bcm_xhci0: MSI enabled bcm_xhci0: (New XHCI DeviceId=3D0x34831106) usbus0 on bcm_xhci0 bcm_xhci0: usbpf: Attached bcm_xhci0: note: switched to 32-bit DMA. pci0: on pcib0 pci0: failed to allocate bus number device_attach: pci0 attach returned 6 In order to get more directly comparable output, I tried using lspci port and use of lspci on raspiOS64 (my abbreviation) and fedora 37. raspiOS64: # lspci -v 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge = (rev 10) (prog-if 00 [Normal decode]) Device tree node: = /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0 Flags: bus master, fast devsel, latency 0 Bus: primary=3D00, secondary=3D01, subordinate=3D01, = sec-latency=3D0 I/O behind bridge: 00000000-00000fff [size=3D4K] Memory behind bridge: c0000000-c00fffff [size=3D1M] Prefetchable memory behind bridge: [disabled] Capabilities: [48] Power Management version 3 Capabilities: [ac] Express Root Port (Slot-), MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0= Len=3D028 Capabilities: [240] L1 PM Substates 01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host = Controller (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller Device tree node: = /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0 Flags: bus master, fast devsel, latency 0, IRQ 63 Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+ Capabilities: [c4] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd fedora: # lspci -v 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge = (rev 20) (prog-if 00 [Normal decode]) Device tree node: = /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0 Flags: bus master, fast devsel, latency 0, IRQ 36 Bus: primary=3D00, secondary=3D01, subordinate=3D01, = sec-latency=3D0 Memory behind bridge: 00000000-000fffff [size=3D1M] [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Capabilities: [48] Power Management version 3 Capabilities: [ac] Express Root Port (Slot-), MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0= Len=3D028 Capabilities: [240] L1 PM Substates Kernel driver in use: pcieport 01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller Device tree node: = /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0 Flags: bus master, fast devsel, latency 0, IRQ 38 Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+ Capabilities: [c4] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd (00:00.0's "Memory behind bridge" displayed based on different spaces, unfortunately. Not as uniform as I was intending. Still. . .) FreeBSD: # lspci -v 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge = (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 91 Bus: primary=3D00, secondary=3D01, subordinate=3D01, = sec-latency=3D0 I/O behind bridge: 0000-0fff [size=3D4K] [16-bit] Memory behind bridge: c0000000-c00fffff [size=3D1M] [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Capabilities: [48] Power Management version 3 Capabilities: [ac] Express Root Port (Slot-), MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0= Len=3D028 Capabilities: [240] L1 PM Substates 01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 = Controller Flags: bus master, fast devsel, latency 0, IRQ 92 Memory at c0000000 (64-bit, non-prefetchable) Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+ Capabilities: [c4] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Note the 01:00.0's "Memory at": c0000000 vs. 600000000 for the two linux exmaples. (Also: No size shown for FreeBSD.) According to "PhilE Raspberry Pi Engineer & Forum Moderator" at: = https://forums.raspberrypi.com/viewtopic.php?t=3D288902&sid=3D792675eb2c65= 13e936b1cd9fe385a038&start=3D75 QUOTE It's worth stating a few facts: (1) The PCIe controller in BCM2711 has a single outbound window. This = means that all addresses for PCIe devices have to be mapped in a single = block, such that the spacing in PCIe space matches the spacing in host = physical address space. (2) The size of the outbound window must be a power of two, and the base = address of the outbound window must be aligned to that size. END QUOTE According to bcm2711-peripherals.pdf figure 1 the standard placement of PCIe space is 0x600000000..0x7ffffffff (avoiding overlaps with anything else in any space). For reference, the live Device Tree for the FreeBSD context has for pcie@7d500000 : ranges =3D <0x02000000 0x00000000 0xc0000000 = 0x00000006 0x00000000 0x00000000 0x40000000>; The "0x00000006 0x00000000" part agrees with that "figure 1". (More recent vintages agree for that part as well.) I'm far from expert for such things but the 01:00.0 having "Memory at c0000000 (64-bit, non-prefetchable)" looks risky for PCI'e outbound memory mapping due to overlaps having potential consequences. An additional note of something that might be involved: The RPi* firmware version that FreeBSD uses still has: pcie@7d500000 { . . . pci@1,0 { . . . usb@1,0 { . . . but modern firmware (such as in the raspiOS64 and fedora I used) has: pcie@7d500000 { . . . pci@0,0 { . . . usb@0,0 { . . . I do not know if that is somehow related to what I'm reporting or not. =3D=3D=3D Mark Millard marklmi at yahoo.com