From nobody Thu Sep 30 16:28:09 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E4D0217D5899 for ; Thu, 30 Sep 2021 16:28:34 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HKzFZ61sLz4SVv; Thu, 30 Sep 2021 16:28:34 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Received: from freebsd2.freebsd.lan (mail.northatlanticmusicsupplies.com [212.237.182.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jsm) by smtp.freebsd.org (Postfix) with ESMTPSA id 6171C7827; Thu, 30 Sep 2021 16:28:34 +0000 (UTC) (envelope-from jsm@FreeBSD.org) Subject: Re: Pinebook pro IOMMU enabled crashes From: Jesper Schmitz Mouridsen To: Emmanuel Vadot Cc: freebsd-arm@freebsd.org References: <5434fca8-4f88-2e76-6977-2de150536915@FreeBSD.org> <20210923091921.563b4621a1801975ad52adc7@bidouilliste.com> <307185e1-a02b-9a1f-155b-e5e7bcac818c@FreeBSD.org> Message-ID: <6d578a7e-bb0a-ac5c-96ce-160aeeb74cce@FreeBSD.org> Date: Thu, 30 Sep 2021 18:28:09 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 In-Reply-To: <307185e1-a02b-9a1f-155b-e5e7bcac818c@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 23.09.2021 19.58, Jesper Schmitz Mouridsen wrote: > Hi > > I just rebuild a generic arm64 with only this change: > > diff --git a/sys/arm64/conf/GENERIC b/sys/arm64/conf/GENERIC > index c716183aae61..7a609db412ca 100644 > --- a/sys/arm64/conf/GENERIC > +++ b/sys/arm64/conf/GENERIC > @@ -19,7 +19,7 @@ > >  cpu            ARM64 >  ident          GENERIC > - > +options                IOMMU >  include                "std.arm64" >  include                "std.dev" > > FreeBSD 14.0-CURRENT #6 main-n249584-fd69939e79a6-dirty > > It does not happen without the nvme attached. > > pcib0: mem > 0xf8000000-0xf9ffffff,0xfd000000-0xfdffffff irq 6,7,8 on ofwbus0 > pci0: on pcib0 > pcib1: at device 0.0 on pci0 > pcib0: failed to reserve resource for pcib1 > pcib1: failed to allocate initial memory window: 0-0xfffff > pci1: on pcib1 > nvme0: at device 0.0 on pci1 > Fatal data abort: >   x0:                0 >   x1:             1000 >   x2:            10040 >   x3:             2000 >   x4:                1 >   x5: ffff00009a7e0168 >   x6: 1400000000000000 >   x7:   10000000000000 >   x8:             1168 >   x9:                1 >  x10:                0 >  x11: ffff000000e8c8c0 >  x12: ffff000000e8c840 >  x13:                1 >  x14:            10000 >  x15:                1 >  x16:            10000 >  x17: ffff000000e8c85c >  x18: ffff000001064180 >  x19: ffff000001064248 >  x20:                0 >  x21: ffff00009a7df000 >  x22: ffffa0000102ea00 >  x23: ffffa00000bb6b80 >  x24: ffffa00001086200 >  x25: ffff000000aa8478 >  x26: ffffa00001086300 >  x27: ffff000000dda000 >  x28:                7 >  x29: ffff000001064190 >   sp: ffff000001064180 >   lr: ffff00000075f20c >  elr: ffff00000078a654 > spsr:         200000c5 >  far:                0 >  esr:         96000004 > panic: vm_fault failed: ffff00000078a654 error 1 > cpuid = 0 > time = 1 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x184 > panic() at panic+0x44 > data_abort() at data_abort+0x23c > handle_el1h_sync() at handle_el1h_sync+0x78 > --- exception, esr 0x96000004 > iommu_map_msi() at iommu_map_msi+0x20 > gicv3_iommu_init() at gicv3_iommu_init+0x4c > intr_alloc_msix() at intr_alloc_msix+0x13c > rk_pcie_alloc_msix() at rk_pcie_alloc_msix+0xfc > pci_alloc_msix_method() at pci_alloc_msix_method+0x1a8 > nvme_pci_attach() at nvme_pci_attach+0x378 > device_attach() at device_attach+0x400 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_attach() at bus_generic_attach+0x18 > pci_attach() at pci_attach+0xe8 > device_attach() at device_attach+0x400 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_attach() at bus_generic_attach+0x18 > device_attach() at device_attach+0x400 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_attach() at bus_generic_attach+0x18 > pci_attach() at pci_attach+0xe8 > device_attach() at device_attach+0x400 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_attach() at bus_generic_attach+0x18 > rk_pcie_attach() at rk_pcie_attach+0x14cc > device_attach() at device_attach+0x400 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_new_pass() at bus_generic_new_pass+0xf8 > bus_generic_new_pass() at bus_generic_new_pass+0xa8 > bus_generic_new_pass() at bus_generic_new_pass+0xa8 > bus_set_pass() at bus_set_pass+0x4c > mi_startup() at mi_startup+0x12c > virtdone() at virtdone+0x6c > > /jsm > > > On 23.09.2021 09.19, Emmanuel Vadot wrote: >> On Sat, 18 Sep 2021 13:15:45 +0200 >> Jesper Schmitz Mouridsen wrote: >> >>> Hi >>> >>> Perhaps this one >>> https://www.mail-archive.com/svn-src-head@freebsd.org/msg126068.html is >>> giving troubles? >>> >>> main-n249225-f673cc5edac3-dirty >>> nvme0: at device 0.0 on pci1 >>> Fatal data abort: >>>     x0:                0 >>>     x1:             1000 >>>     x2:            10040 >>>     x3:             2000 >>>     x4:                1 >>>     x5: ffff00009a7a0168 >>>     x6: 1d00000000000000 >>>     x7:   10000000000000 >>>     x8:             1168 >>>     x9:                1 >>>    x10:                0 >>>    x11: ffff000000f35140 >>>    x12: ffff000000f350c0 >>>    x13:                1 >>>    x14:            10000 >>>    x15:                1 >>>    x16:            10000 >>>    x17: ffff000000f350dc >>>    x18: ffff00000110d180 >>>    x19: ffff00000110d248 >>>    x20:                0 >>>    x21: ffff00009a79f000 >>>    x22: ffffa000010b0a00 >>>    x23: ffffa000010a2880 >>>    x24: ffffa0000116da00 >>>    x25: ffff000000b4fd78 >>>    x26: ffffa0000116db00 >>>    x27: ffff000000e83000 >>>    x28:                7 >>>    x29: ffff00000110d190 >>>     sp: ffff00000110d180 >>>     lr: ffff00000077520c >>>    elr: ffff0000007a03ac >>> spsr:         200000c5 >>>    far:                0 >>>    esr:         96000004 >>> panic: vm_fault failed: ffff0000007a03ac error 1 >>> cpuid = 0 >>> time = 1 >>> KDB: stack backtrace: >>> db_trace_self() at db_trace_self >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >>> vpanic() at vpanic+0x184 >>> panic() at panic+0x44 >>> data_abort() at data_abort+0x23c >>> handle_el1h_sync() at handle_el1h_sync+0x78 >>> --- exception, esr 0x96000004 >>> iommu_map_msi() at iommu_map_msi+0x20 >>> gicv3_iommu_init() at gicv3_iommu_init+0x4c >>> intr_alloc_msix() at intr_alloc_msix+0x13c >>> rk_pcie_alloc_msix() at rk_pcie_alloc_msix+0xfc >>> pci_alloc_msix_method() at pci_alloc_msix_method+0x1a8 >>> nvme_pci_attach() at nvme_pci_attach+0x378 >>> device_attach() at device_attach+0x400 >>> device_probe_and_attach() at device_probe_and_attach+0x7c >>> bus_generic_attach() at bus_generic_attach+0x18 >>> pci_attach() at pci_attach+0xe8 >>> device_attach() at device_attach+0x400 >>> device_probe_and_attach() at device_probe_and_attach+0x7c >>> bus_generic_attach() at bus_generic_attach+0x18 >>> device_attach() at device_attach+0x400 >>> device_probe_and_attach() at device_probe_and_attach+0x7c >>> bus_generic_attach() at bus_generic_attach+0x18 >>> pci_attach() at pci_attach+0xe8 >>> device_attach() at device_attach+0x400 >>> device_probe_and_attach() at device_probe_and_attach+0x7c >>> bus_generic_attach() at bus_generic_attach+0x18 >>> rk_pcie_attach() at rk_pcie_attach+0x14cc >>> device_attach() at device_attach+0x400 >>> device_probe_and_attach() at device_probe_and_attach+0x7c >>> bus_generic_new_pass() at bus_generic_new_pass+0xf8 >>> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >>> bus_generic_new_pass() at bus_generic_new_pass+0xa8 >>> bus_set_pass() at bus_set_pass+0x4c >>> mi_startup() at mi_startup+0x12c >>> virtdone() at virtdone+0x6c >>> >>   That's an old commit. Did you see this panic only recently or ? >> > Even on stable/13-n247374-9faebc1e664d-dirty I get the same backtrace when IOMMU is enabled and the nvme is attached. pcib1: at device 0.0 on pci0 pcib0: failed to reserve resource for pcib1 pcib1: failed to allocate initial memory window: 0-0xfffff pci1: on pcib1 nvme0: at device 0.0 on pci1 Fatal data abort: x0: 0 x1: 1000 x2: 10040 x3: 2000 x4: 1 x5: ffff00009a99e160 x6: 1400000000000000 x7: 10000000000000 x8: 1160 x9: ffff000000cd7cc0 x10: 0 x11: ffff000000d89540 x12: ffff000000d894c0 x13: 1 x14: 10000 x15: 1 x16: 10000 x17: 0 x18: ffff000000f5c250 x19: ffff000000f5c318 x20: 0 x21: ffff00009a99d000 x22: ffffa00000f06200 x23: ffffa00000f49700 x24: ffffa00000f8f500 x25: ffff0000009b85f8 x26: ffffa00000f8f600 x27: ffff000000cd7000 x28: 7 x29: ffff000000f5c260 sp: ffff000000f5c250 lr: ffff0000006bf3dc elr: ffff0000006e15d0 spsr: 600001c5 far: 0 esr: 96000004 panic: vm_fault failed: ffff0000006e15d0 cpuid = 0 time = 1 KDB: stack backtrace: #0 0xffff00000047c304 at kdb_backtrace+0x60 #1 0xffff000000437fd8 at vpanic+0x184 #2 0xffff000000437e50 at panic+0x44 #3 0xffff0000006d692c at data_abort+0x204 #4 0xffff0000006bb874 at handle_el1h_sync+0x74 #5 0xffff0000006bf3d8 at gicv3_iommu_init+0x4c #6 0xffff0000006bf3d8 at gicv3_iommu_init+0x4c #7 0xffff0000006b1940 at intr_alloc_msix+0x110 #8 0xffff0000007860c0 at rk_pcie_alloc_msix+0xfc #9 0xffff000000219bbc at pci_alloc_msix_method+0x1a8 #10 0xffff00000020ba64 at nvme_pci_attach+0x378 #11 0xffff00000046bd80 at device_attach+0x400 #12 0xffff00000046d14c at bus_generic_attach+0x4c #13 0xffff000000221f30 at pci_attach+0xe0 #14 0xffff00000046bd80 at device_attach+0x400 #15 0xffff00000046d14c at bus_generic_attach+0x4c #16 0xffff00000046bd80 at device_attach+0x400 #17 0xffff00000046d14c at bus_generic_attach+0x4c Uptime: 1s