From nobody Thu Sep 30 16:39:43 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2749D17D6A66 for ; Thu, 30 Sep 2021 16:39:45 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HKzVT0RSQz4Tgq; Thu, 30 Sep 2021 16:39:45 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Received: from freebsd2.freebsd.lan (mail.northatlanticmusicsupplies.com [212.237.182.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jsm) by smtp.freebsd.org (Postfix) with ESMTPSA id 91036782D; Thu, 30 Sep 2021 16:39:44 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Subject: Re: Pinebook pro IOMMU enabled crashes From: Jesper Schmitz Mouridsen To: Emmanuel Vadot Cc: freebsd-arm@freebsd.org References: <5434fca8-4f88-2e76-6977-2de150536915@FreeBSD.org> <20210923091921.563b4621a1801975ad52adc7@bidouilliste.com> <307185e1-a02b-9a1f-155b-e5e7bcac818c@FreeBSD.org> <6d578a7e-bb0a-ac5c-96ce-160aeeb74cce@FreeBSD.org> Message-ID: <5e537303-0807-7902-c90f-f9fab5120548@FreeBSD.org> Date: Thu, 30 Sep 2021 18:39:43 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 In-Reply-To: <6d578a7e-bb0a-ac5c-96ce-160aeeb74cce@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 30.09.2021 18.28, Jesper Schmitz Mouridsen wrote: > On 23.09.2021 19.58, Jesper Schmitz Mouridsen wrote: >> Hi >> >> I just rebuild a generic arm64 with only this change: >> >> diff --git a/sys/arm64/conf/GENERIC b/sys/arm64/conf/GENERIC >> index c716183aae61..7a609db412ca 100644 >> --- a/sys/arm64/conf/GENERIC >> +++ b/sys/arm64/conf/GENERIC >> @@ -19,7 +19,7 @@ >> >>   cpu            ARM64 >>   ident          GENERIC >> - >> +options                IOMMU >>   include                "std.arm64" >>   include                "std.dev" >> >> FreeBSD 14.0-CURRENT #6 main-n249584-fd69939e79a6-dirty >> >> It does not happen without the nvme attached. >> >> pcib0: mem >> 0xf8000000-0xf9ffffff,0xfd000000-0xfdffffff irq 6,7,8 on ofwbus0 >> pci0: on pcib0 >> pcib1: at device 0.0 on pci0 >> pcib0: failed to reserve resource for pcib1 >> pcib1: failed to allocate initial memory window: 0-0xfffff >> pci1: on pcib1 >> nvme0: at device 0.0 on pci1 >> Fatal data abort: >>    x0:                0 >>    x1:             1000 >>    x2:            10040 >>    x3:             2000 >>    x4:                1 >>    x5: ffff00009a7e0168 >>    x6: 1400000000000000 >>    x7:   10000000000000 >>    x8:             1168 >>    x9:                1 >>   x10:                0 >>   x11: ffff000000e8c8c0 >>   x12: ffff000000e8c840 >>   x13:                1 >>   x14:            10000 >>   x15:                1 >>   x16:            10000 >>   x17: ffff000000e8c85c >>   x18: ffff000001064180 >>   x19: ffff000001064248 >>   x20:                0 >>   x21: ffff00009a7df000 >>   x22: ffffa0000102ea00 >>   x23: ffffa00000bb6b80 >>   x24: ffffa00001086200 >>   x25: ffff000000aa8478 >>   x26: ffffa00001086300 >>   x27: ffff000000dda000 >>   x28:                7 >>   x29: ffff000001064190 >>    sp: ffff000001064180 >>    lr: ffff00000075f20c >>   elr: ffff00000078a654 >> spsr:         200000c5 >>   far:                0 >>   esr:         96000004 >> panic: vm_fault failed: ffff00000078a654 error 1 >> cpuid = 0 >> time = 1 >> KDB: stack backtrace: >> db_trace_self() at db_trace_self >> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >> vpanic() at vpanic+0x184 >> panic() at panic+0x44 >> data_abort() at data_abort+0x23c >> handle_el1h_sync() at handle_el1h_sync+0x78 >> --- exception, esr 0x96000004 >> iommu_map_msi() at iommu_map_msi+0x20 >> gicv3_iommu_init() at gicv3_iommu_init+0x4c >> intr_alloc_msix() at intr_alloc_msix+0x13c >> rk_pcie_alloc_msix() at rk_pcie_alloc_msix+0xfc >> pci_alloc_msix_method() at pci_alloc_msix_method+0x1a8 >> nvme_pci_attach() at nvme_pci_attach+0x378 >> device_attach() at device_attach+0x400 >> device_probe_and_attach() at device_probe_and_attach+0x7c >> bus_generic_attach() at bus_generic_attach+0x18 >> pci_attach() at pci_attach+0xe8 >> device_attach() at device_attach+0x400 >> device_probe_and_attach() at device_probe_and_attach+0x7c >> bus_generic_attach() at bus_generic_attach+0x18 >> device_attach() at device_attach+0x400 >> device_probe_and_attach() at device_probe_and_attach+0x7c >> bus_generic_attach() at bus_generic_attach+0x18 >> pci_attach() at pci_attach+0xe8 >> device_attach() at device_attach+0x400 >> device_probe_and_attach() at device_probe_and_attach+0x7c >> bus_generic_attach() at bus_generic_attach+0x18 >> rk_pcie_attach() at rk_pcie_attach+0x14cc >> device_attach() at device_attach+0x400 >> device_probe_and_attach() at device_probe_and_attach+0x7c >> bus_generic_new_pass() at bus_generic_new_pass+0xf8 >> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >> bus_set_pass() at bus_set_pass+0x4c >> mi_startup() at mi_startup+0x12c >> virtdone() at virtdone+0x6c >> >> /jsm >> >> >> On 23.09.2021 09.19, Emmanuel Vadot wrote: >>> On Sat, 18 Sep 2021 13:15:45 +0200 >>> Jesper Schmitz Mouridsen wrote: >>> >>>> Hi >>>> >>>> Perhaps this one >>>> https://www.mail-archive.com/svn-src-head@freebsd.org/msg126068.html is >>>> giving troubles? >>>> >>>> main-n249225-f673cc5edac3-dirty >>>> nvme0: at device 0.0 on pci1 >>>> Fatal data abort: >>>>     x0:                0 >>>>     x1:             1000 >>>>     x2:            10040 >>>>     x3:             2000 >>>>     x4:                1 >>>>     x5: ffff00009a7a0168 >>>>     x6: 1d00000000000000 >>>>     x7:   10000000000000 >>>>     x8:             1168 >>>>     x9:                1 >>>>    x10:                0 >>>>    x11: ffff000000f35140 >>>>    x12: ffff000000f350c0 >>>>    x13:                1 >>>>    x14:            10000 >>>>    x15:                1 >>>>    x16:            10000 >>>>    x17: ffff000000f350dc >>>>    x18: ffff00000110d180 >>>>    x19: ffff00000110d248 >>>>    x20:                0 >>>>    x21: ffff00009a79f000 >>>>    x22: ffffa000010b0a00 >>>>    x23: ffffa000010a2880 >>>>    x24: ffffa0000116da00 >>>>    x25: ffff000000b4fd78 >>>>    x26: ffffa0000116db00 >>>>    x27: ffff000000e83000 >>>>    x28:                7 >>>>    x29: ffff00000110d190 >>>>     sp: ffff00000110d180 >>>>     lr: ffff00000077520c >>>>    elr: ffff0000007a03ac >>>> spsr:         200000c5 >>>>    far:                0 >>>>    esr:         96000004 >>>> panic: vm_fault failed: ffff0000007a03ac error 1 >>>> cpuid = 0 >>>> time = 1 >>>> KDB: stack backtrace: >>>> db_trace_self() at db_trace_self >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >>>> vpanic() at vpanic+0x184 >>>> panic() at panic+0x44 >>>> data_abort() at data_abort+0x23c >>>> handle_el1h_sync() at handle_el1h_sync+0x78 >>>> --- exception, esr 0x96000004 >>>> iommu_map_msi() at iommu_map_msi+0x20 >>>> gicv3_iommu_init() at gicv3_iommu_init+0x4c >>>> intr_alloc_msix() at intr_alloc_msix+0x13c >>>> rk_pcie_alloc_msix() at rk_pcie_alloc_msix+0xfc >>>> pci_alloc_msix_method() at pci_alloc_msix_method+0x1a8 >>>> nvme_pci_attach() at nvme_pci_attach+0x378 >>>> device_attach() at device_attach+0x400 >>>> device_probe_and_attach() at device_probe_and_attach+0x7c >>>> bus_generic_attach() at bus_generic_attach+0x18 >>>> pci_attach() at pci_attach+0xe8 >>>> device_attach() at device_attach+0x400 >>>> device_probe_and_attach() at device_probe_and_attach+0x7c >>>> bus_generic_attach() at bus_generic_attach+0x18 >>>> device_attach() at device_attach+0x400 >>>> device_probe_and_attach() at device_probe_and_attach+0x7c >>>> bus_generic_attach() at bus_generic_attach+0x18 >>>> pci_attach() at pci_attach+0xe8 >>>> device_attach() at device_attach+0x400 >>>> device_probe_and_attach() at device_probe_and_attach+0x7c >>>> bus_generic_attach() at bus_generic_attach+0x18 >>>> rk_pcie_attach() at rk_pcie_attach+0x14cc >>>> device_attach() at device_attach+0x400 >>>> device_probe_and_attach() at device_probe_and_attach+0x7c >>>> bus_generic_new_pass() at bus_generic_new_pass+0xf8 >>>> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >>>> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >>>> bus_set_pass() at bus_set_pass+0x4c >>>> mi_startup() at mi_startup+0x12c >>>> virtdone() at virtdone+0x6c >>>> >>>   That's an old commit. Did you see this panic only recently or ? >>> >> > > > Even on stable/13-n247374-9faebc1e664d-dirty > > I get the same backtrace when IOMMU is enabled and the nvme is attached. > > pcib1: at device 0.0 on pci0 > pcib0: failed to reserve resource for pcib1 > pcib1: failed to allocate initial memory window: 0-0xfffff > pci1: on pcib1 > nvme0: at device 0.0 on pci1 > Fatal data abort: >   x0:                0 >   x1:             1000 >   x2:            10040 >   x3:             2000 >   x4:                1 >   x5: ffff00009a99e160 >   x6: 1400000000000000 >   x7:   10000000000000 >   x8:             1160 >   x9: ffff000000cd7cc0 >  x10:                0 >  x11: ffff000000d89540 >  x12: ffff000000d894c0 >  x13:                1 >  x14:            10000 >  x15:                1 >  x16:            10000 >  x17:                0 >  x18: ffff000000f5c250 >  x19: ffff000000f5c318 >  x20:                0 >  x21: ffff00009a99d000 >  x22: ffffa00000f06200 >  x23: ffffa00000f49700 >  x24: ffffa00000f8f500 >  x25: ffff0000009b85f8 >  x26: ffffa00000f8f600 >  x27: ffff000000cd7000 >  x28:                7 >  x29: ffff000000f5c260 >   sp: ffff000000f5c250 >   lr: ffff0000006bf3dc >  elr: ffff0000006e15d0 > spsr:         600001c5 >  far:                0 >  esr:         96000004 > panic: vm_fault failed: ffff0000006e15d0 > cpuid = 0 > time = 1 > KDB: stack backtrace: > #0 0xffff00000047c304 at kdb_backtrace+0x60 > #1 0xffff000000437fd8 at vpanic+0x184 > #2 0xffff000000437e50 at panic+0x44 > #3 0xffff0000006d692c at data_abort+0x204 > #4 0xffff0000006bb874 at handle_el1h_sync+0x74 > #5 0xffff0000006bf3d8 at gicv3_iommu_init+0x4c > #6 0xffff0000006bf3d8 at gicv3_iommu_init+0x4c > #7 0xffff0000006b1940 at intr_alloc_msix+0x110 > #8 0xffff0000007860c0 at rk_pcie_alloc_msix+0xfc > #9 0xffff000000219bbc at pci_alloc_msix_method+0x1a8 > #10 0xffff00000020ba64 at nvme_pci_attach+0x378 > #11 0xffff00000046bd80 at device_attach+0x400 > #12 0xffff00000046d14c at bus_generic_attach+0x4c > #13 0xffff000000221f30 at pci_attach+0xe0 > #14 0xffff00000046bd80 at device_attach+0x400 > #15 0xffff00000046d14c at bus_generic_attach+0x4c > #16 0xffff00000046bd80 at device_attach+0x400 > #17 0xffff00000046d14c at bus_generic_attach+0x4c > Uptime: 1s > > git checkout 50cedfede3d21824ec6023324b3ad41a435e1815 sys/arm64/arm64/gicv3_its.c and the problem goes away. The commit is one before Add IOMMU support to GICv3 Interrupt Translation Service (ITS) driver. (ba196aec7dad1b73a9a3b86a06259d5e81f16fad)