From nobody Sun Nov 21 20:09:35 2021 X-Original-To: arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D1AE7188D808 for ; Sun, 21 Nov 2021 20:10:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hy1k01DDVz4sNh for ; Sun, 21 Nov 2021 20:10:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637525440; bh=PkWcy2dtb5VpjuCKQ6TprLl2ogwQvyH59OHLljTezco=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=pWdn2kvRh5r0VYf5DKSXDTTDvl2WCdR7JNNVFPPOyqT5+s/stN91L8tmNyKj6HT8+tTMItBoTmNZMF0iRvkfpFP3wkZynxJMR8IX0i7QSgRHLDtjRhpaElZ0E1m4ZKa/gWCMe2AJaXAAGA8sGhd0R4xR9UC+r7+ey1beh4IjIL7MkMSn36U2qb7qgYXNgaDQoplSlhJc0EqIeBSs5NaLkrNXa2VX2AGpMZzn2GXa37yPVYct6gSeY5d4Con2ffEDrh6MzLYKuvq6aeDLud4IlUHd/kP6oC8/F925xD3urDgzO9GF3d7+gj/IU//tBmFer1ky0TvAnamBY/BEkfKSUg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637525440; bh=kYLQ7mRrGnHtzZpQ6z9VnO7Sc3cAtixFp8+KWz25hvl=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kbdCKgbBWyvWgJ7mswCTufeCZxTII7cGE4N2xKeUuoN0R26oO8FgSYxBzzZof8bTO3vKE0Sri6vbC7QFyuuyLnRFtZCFXe/TdVK3q4VriK15xihGX09EHW14Akld3vTk1LR7p/Dk21ft89jAJ50IkYjaqdrZa2tGNZTFFwLXrMUb31wNFaMeQo3h9GzAKimHV1WQ1Kx7bvN/6kQ6PhrjCqC9k1y30jRhPRZnyPdoBAXdGQvFtYgu5RFgu3QkEJVv+m3nzdeb1M2amPhloa3oO1Y+QNq6+0dY4gXg5kHuSBJs4W2DJSxT7tZprU46QkzqzJFbkXc4i0od18Fs//U85A== X-YMail-OSG: VRDWmcwVM1mHMTgaPGaKlbZ4dJPtaF3HNwSduGP8QOBCRzpmG0twKjSWywWPFSY xjDeTrTIs1mstwI04ZhtCbWKRsElH.QqRaeRBKsm6z2K6.JSxcSCOvSWITsvsBEJXnIUsvpwDpA. Ph1NA2i22Nh4FDe9t4NpdN9P5jCoExjEpNLnx0WbeFhfRYtVI13B5tfJiD4xFk0FAQDbaJInP1E1 tARj2WzBEMpQz3490iEzyLlY7S39tolGEr6Znn_fofEYOD9U7CDtUlxD7BHAAQbNwPi70FUkIjz. 9Uir5FCS02DNSMCwShcgV7U3IzoOiW6oRQOAEYOjYhMuP9ex_iwQmTY.onXGcHGXRX0SuqbD5ASD mWYggkd34Ixwuy.D.V9c.6vpSN021EoTYeLxunMwNhmYDDhIBuOFYQnQJyzW9L7DXwPdcgDVbvGP rGqQxnf6jEGQ6UfpzoWqxxenz1DubaXi4BtDMu.QDLlr0uNSwf_MpAB0ofdaRp2Wd3cNf.gRthe8 jzGF5ypxP0mgKOYxuTIuptC.BuRZ8Rf6pt5MKhYNVgvZs12u1gEMUFOFSNSsLOCaig7qJpzU0O0d FWr2.hwkOzeLdAPlIOUD.U2lfrhLgIIxwHFQxIxvFKGBYwsGL4UOLtrraNmQWwLiM.zOx9yvQCz_ BF5EU9w6ZfW.D3r1dq0D2KNGYfY82TE.3lV1UroY5RYYw5D1iLeC.sQSA6yQiy14zAs5sQC9ePEU ScZHAdS_VRW89RzxBQu14mwDkaVE2pE5h_dDYNNH8EMx_2OR25FQ4K70ZqJvM7pEr.rTIgDdmLSL brAm5xhWZMYDjxZXEB_5RmW4JlfUW_OY_rm_7ry3nR6sZhWkDfyUsma7C6qN5bmNWiR.Cz2kE2Cj 5jH6wnqXpKY9imTCIJue5yXP9G4PVfJf6ftgwmKJvUzW70sGBAl2gfn0.7047ZqINyy_dS2EB96O TlKOr6YT4k9vg_TcHPvgdwLael6jqSq.BkXuprAgfoJ8Zvqc6Ml3aRLTSS8739BYctBsUwbSUpoc YHPrcM91_cLfAgDxLn.V..bgJ32Skg8w8.jfvb9C0uKuslZs0GXwOHHcAviUmzwdWrsPAXZN1YzK eDNO0YDAAoGCdjW5C0vPsfqxtxikTLl4s_O02bw9EayhWMpkh9LIT6l6fZ9aK5GUkPNGWW3gMKHM gV0c0ZQQhp9RR.Zy9hWP6z0pToo.aQe4bRb_DqEGMLyofNMhW7aM6anwjmmKTgeo1ZGWN5zUDWDF 5DIM58VJq9kvqkh4.2OkZ.cnCPaCUWSJjzxZqsH7vpVXKEp32ETF1TBwbGZ57wZQgSiAjYmuiPsL LV_k9hojiK44CM0mLkfFXSRr6J_5FEr4wAXTC73B0GgmrfcJepMhvEjyr68A1h1P4cpesF6N6kKH RDEiMuB4Nnhej3e.0NLLd3EOqb.efinh78R91yP0TkE12RyYnLur8v_w_6wjqNOJFaZNDbdZqU_k 2EWTbY7EDPPN_ye.UzZsVUy0BQjZ9U9tjw3XKxUWRyMWGDUNUmurBTIWwpBHpPryzEw3uNJbMKNU .GrEQytFgwu6PkApzhGewJ0UrOcfz5Ulj8XNbenD0SXxRicubsrI5NFQtk8LcFx89eTdw0vJyUpk Y6k6SSJ8oa0UO8dOW59Wpj.EXnfgVcsLYY4GyLtZPSrw3e7tjgm.BPcPryGkFPSimw5dEp60RT8W jvqwxW4v5Ij7ggYVW0PWyHTXQk_OY0_K4SG09ZE1TaPDlLqoQ9ygH0TuXTRBmhjDQzfl5dCJzmQo iud7WU.1MISY0VUTI8AUptiXqzlR22_0trZtaZ3HjcPYHBDk45S5FoGzXAYZh7SpTiJCUHjB2V3G zqlrk4Xur3dNl7DXcBOUvzjtINd86tHaWS50SDOFIUMsnPNaBbJe7DzkhzGohDsLz6jmWU262zRH jnrpqUR3KhVwyejA2XNq7fE157IzpmqS08v8OFoxKWHsK_1iJpd4xI8p663VT_AOgfHC6jEANX0I 4WRLCg36I8SdzT4WF70zs2b6QRhprOygKdfRFLq_dSdhBq8.kQODWufkuheLaw4GrPQjUkGQlUlH ybuaAs0MoZXCxczMMhliH3tWwhtVdDQj3CQZ9Pfz1LUsuHTazReos5OnFWUoaiLJhQxygjy5RZJH 5LWuIe6Frlt4MCwjgGvCsgTwo3lEUmOcFwZJrh3Pzt_UCCVfi4j8hfVFbroOsQ_OdI0b1ivAR0fw JPhmaF9ChN1hx4VO5h4P02RcVOgChg5cp0Gh1dRKZTBILag-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 Nov 2021 20:10:40 +0000 Received: by kubenode545.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 9488c875f50d8690f753f7afb465ae80; Sun, 21 Nov 2021 20:09:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: FYI: aarch64 boot (HoneyComb): example crash during system checks (power-off/power-on form of reboot still fails) Date: Sun, 21 Nov 2021 12:09:35 -0800 References: To: "freebsd-arm@freebsd.org" , FreeBSD-STABLE Mailing List In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Hy1k01DDVz4sNh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=pWdn2kvR; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.83:from]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.83:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] Reply-To: marklmi@yahoo.com From: Mark Millard via arm X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Nov-21, at 11:36, Mark Millard wrote: > On 2021-Nov-21, at 11:26, Mark Millard wrote: >=20 >> Starting file system checks: >> /dev/gpt/CA72opt0EFI: 41 files, 242 MiB free (15469 clusters) >> FIXED >> /d x0: ffff000000e43ec8 (blocked_lock + 0) >> x1: ffff00013efa9f50 >> x2: ffff00000090e39a (cam_status_table + 1d132) >> x3: deadc0d8 >> x4: 0 >> x5: ffff00000082a138 (data_abort + 0) >> x6: 5 >> x7: 601 >> x8: ffff000000e43ec8 (blocked_lock + 0) >> x9: deadc0de >> x10: 0 >> x11: 3938700 >> x12: 0 >> x13: 8000 >> x14: 1de >> x15: 81ce >> x16: 425b9080 >> x17: 8000 >> x18: ffff00013efa9f40 >> x19: ffff000000e43ec8 (blocked_lock + 0) >> x20: ffffa0001a826000 >> x21: 0 >> x22: ffffa0001a826000 >> x23: 0 >> x24: ffff000000bed000 (queue_ops + 0) >> x25: 98967f >> x26: ffff000000e43ee0 (blocked_lock + 18) >> x27: 0 >> x28: 114 >> x29: ffff00013efa9f40 >> sp: ffff00013efa9f40 >> lr: ffff0000004b9028 (thread_lock_flags_ + c0) >> elr: ffff0000004b9028 (thread_lock_flags_ + c0) >> spsr: 2c5 >> far: deadc178 >> esr: 96000004 >> timeout stopping cpus >> panic: data abort in critical section or under mutex >> cpuid =3D 5 >> time =3D 1637492224 >> KDB: stack backtrace: >> db_trace_self() at db_trace_self_wrapper+0x30 >> pc =3D 0xffff000000807770 lr =3D 0xffff00000011d9ec >> sp =3D 0xffff00013efa9990 fp =3D 0xffff00013efa9b90 >>=20 >> db_trace_self_wrapper() at vpanic+0x188 >> pc =3D 0xffff00000011d9ec lr =3D 0xffff0000004e1d10 >> sp =3D 0xffff00013efa9ba0 fp =3D 0xffff00013efa9c00 >>=20 >> vpanic() at panic+0x44 >> pc =3D 0xffff0000004e1d10 lr =3D 0xffff0000004e1b84 >> sp =3D 0xffff00013efa9c10 fp =3D 0xffff00013efa9cc0 >>=20 >> panic() at data_abort+0x290 >> pc =3D 0xffff0000004e1b84 lr =3D 0xffff00000082a3c8 >> sp =3D 0xffff00013efa9cd0 fp =3D 0xffff00013efa9d50 >>=20 >> data_abort() at handle_el1h_sync+0x78 >> pc =3D 0xffff00000082a3c8 lr =3D 0xffff00000080a078 >> sp =3D 0xffff00013efa9d60 fp =3D 0xffff00013efa9eb0 >>=20 >> handle_el1h_sync() at thread_lock_flags_+0xbc >> pc =3D 0xffff00000080a078 lr =3D 0xffff0000004b9024 >> sp =3D 0xffff00013efa9ec0 fp =3D 0xffff00013efa9f40 >>=20 >> thread_lock_flags_() at thread_lock_flags_+0xbc >> pc =3D 0xffff0000004b9024 lr =3D 0xffff0000004b9024 >> sp =3D 0xffff00013efa9f50 fp =3D 0xffff00013efa9f60 >>=20 >> thread_lock_flags_() at sleepq_timeout+0x10 >> pc =3D 0xffff0000004b9024 lr =3D 0xffff00000054b2a8 >> sp =3D 0xffff00013efa9f70 fp =3D 0xffff00013efa9fb0 >>=20 >> sleepq_timeout() at softclock_call_cc+0x14c >> pc =3D 0xffff00000054b2a8 lr =3D 0xffff000000503134 >> sp =3D 0xffff00013efa9fc0 fp =3D 0xffff00013efaa020 >>=20 >> softclock_call_cc() at callout_process+0x17c >> pc =3D 0xffff000000503134 lr =3D 0xffff000000502df0 >> sp =3D 0xffff00013efaa030 fp =3D 0xffff00013efaa0a0 >>=20 >> callout_process() at handleevents+0x188 >> pc =3D 0xffff000000502df0 lr =3D 0xffff00000045b42c >> sp =3D 0xffff00013efaa0b0 fp =3D 0xffff00013efaa100 >>=20 >> handleevents() at timercb+0x304 >> pc =3D 0xffff00000045b42c lr =3D 0xffff00000045be7c >> sp =3D 0xffff00013efaa110 fp =3D 0xffff00013efaa170 >>=20 >> timercb() at arm_tmr_intr+0x5c >> pc =3D 0xffff00000045be7c lr =3D 0xffff0000007ff850 >> sp =3D 0xffff00013efaa180 fp =3D 0xffff00013efaa1d0 >>=20 >> arm_tmr_intr() at intr_event_handle+0xac >> pc =3D 0xffff0000007ff850 lr =3D 0xffff000000493c54 >> sp =3D 0xffff00013efaa1e0 fp =3D 0xffff00013efaa1e0 >>=20 >> intr_event_handle() at intr_isrc_dispatch+0x70 >> pc =3D 0xffff000000493c54 lr =3D 0xffff0000007fb238 >> sp =3D 0xffff00013efaa1f0 fp =3D 0xffff00013efaa230 >>=20 >> intr_isrc_dispatch() at arm_gic_v3_intr+0x11c >> pc =3D 0xffff0000007fb238 lr =3D 0xffff00000080ff34 >> sp =3D 0xffff00013efaa240 fp =3D 0xffff00013efaa250 >>=20 >> arm_gic_v3_intr() at intr_irq_handler+0x7c >> pc =3D 0xffff00000080ff34 lr =3D 0xffff0000007faff0 >> sp =3D 0xffff00013efaa260 fp =3D 0xffff00013efaa2b0 >>=20 >> intr_irq_handler() at handle_el1h_irq+0x74 >> pc =3D 0xffff0000007faff0 lr =3D 0xffff00000080a140 >> sp =3D 0xffff00013efaa2c0 fp =3D 0xffff00013efaa3f0 >>=20 >> handle_el1h_irq() at handle_el1h_sync+0x78 >> pc =3D 0xffff00000080a140 lr =3D 0xffff00000080a078 >> sp =3D 0xffff00013efaa400 fp =3D 0xffff00013efaa500 >>=20 >> handle_el1h_sync() at handle_el1h_sync+0x78 >> pc =3D 0xffff00000080a078 lr =3D 0xffff00000080a078 >> sp =3D 0xffff00013efaa510 fp =3D 0xffff00013efaa660 >>=20 >> handle_el1h_sync() at sched_switch+0x6a8 >> pc =3D 0xffff00000080a078 lr =3D 0xffff0000005197fc >> sp =3D 0xffff00013efaa670 fp =3D 0xffff00013efaa6f0 >>=20 >> sched_switch() at sched_switch+0x6a8 >> pc =3D 0xffff0000005197fc lr =3D 0xffff0000005197fc >> sp =3D 0xffff00013efaa700 fp =3D 0xffff00013efaa790 >>=20 >> sched_switch() at mi_switch+0xf4 >> pc =3D 0xffff0000005197fc lr =3D 0xffff0000004f03a0 >> sp =3D 0xffff00013efaa7a0 fp =3D 0xffff00013efaa7f0 >>=20 >> mi_switch() at sleepq_timedwait+0x28 >> pc =3D 0xffff0000004f03a0 lr =3D 0xffff00000054bd0c >> sp =3D 0xffff00013efaa800 fp =3D 0xffff00013efaa830 >>=20 >> sleepq_timedwait() at _cv_timedwait_sbt+0x110 >> pc =3D 0xffff00000054bd0c lr =3D 0xffff00000045e7b0 >> sp =3D 0xffff00013efaa840 fp =3D 0xffff00013efaa850 >>=20 >> _cv_timedwait_sbt() at dbuf_evict_thread+0x410 >> pc =3D 0xffff00000045e7b0 lr =3D 0xffff0000013ca59c >> sp =3D 0xffff00013efaa860 fp =3D 0xffff00013efaa8f0 >>=20 >> dbuf_evict_thread() at fork_exit+0x94 >> pc =3D 0xffff0000013ca59c lr =3D 0xffff00000048fbf0 >> sp =3D 0xffff00013efaa900 fp =3D 0xffff00013efaa950 >>=20 >> fork_exit() at fork_trampoline+0x10 >> pc =3D 0xffff00000048fbf0 lr =3D 0xffff000000828ed8 >> sp =3D 0xffff00013efaa960 fp =3D 0x0000000000000000 >>=20 >> KDB: enter: panic >> [ thread pid 26 tid 100194 ] >> Stopped at kdb_enter+0x48: undefined f906411f >>=20 >> For reference: >>=20 >> # uname -apKU >> FreeBSD CA72_16Gp_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #13 = stable/13-n248062-109330155000-dirty: Sat Nov 13 23:55:14 PST 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/13S-CA72-nodbg-clang/usr/13S-src/arm64.= aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300520 1300520 >>=20 >> It is a root-on-ZFS context on Optane media in the PCie slot. >>=20 >> I've no clue if this will repeat. I've never gotten >> this before. >=20 >=20 > The reboot attempt got the following, involving > zthr_procedure instead of dbuf_evict_thread . >=20 > Starting file system checks: > /dev/gpt/CA72opt0EFI: FILESYSTEM CLEAN; SKIPPING CHECKS > x0: ffff000000e43ec8 (blocked_lock + 0) > x1: ffff00013ef9ff90 > x2: ffff00000090e39a (cam_status_table + 1d132) > x3: deadc0d8 > x4: 0 > x5: ffff00000082a138 (data_abort + 0) > x6: 9 > x7: 601 > x8: ffff000000e43ec8 (blocked_lock + 0) > x9: deadc0de > x10: 0 > x11: 3938700 > x12: 1 > x13: 8000 > x14: 1ee > x15: 81cd > x16: 425b9080 > x17: 8000 > x18: ffff00013ef9ff80 > x19: ffff000000e43ec8 (blocked_lock + 0) > x20: ffffa0000c011000 > x21: 0 > x22: ffffa0000c011000 > x23: 0 > x24: ffff000000bed000 (queue_ops + 0) > x25: 98967f > x26: ffff000000e43ee0 (blocked_lock + 18) > x27: 0 > x28: 114 > x29: ffff00013ef9ff80 > sp: ffff00013ef9ff80 > lr: ffff0000004b9028 (thread_lock_flags_ + c0) > elr: ffff0000004b9028 (thread_lock_flags_ + c0) > spsr: 2c5 > far: deadc178 > esr: 96000004 > timeout stopping cpus > panic: data abort in critical section or under mutex > cpuid =3D 9 > time =3D 1637492224 > KDB: stack backtrace: > db_trace_self() at db_trace_self_wrapper+0x30 > pc =3D 0xffff000000807770 lr =3D 0xffff00000011d9ec > sp =3D 0xffff00013ef9f9d0 fp =3D 0xffff00013ef9fbd0 >=20 > db_trace_self_wrapper() at vpanic+0x188 > pc =3D 0xffff00000011d9ec lr =3D 0xffff0000004e1d10 > sp =3D 0xffff00013ef9fbe0 fp =3D 0xffff00013ef9fc40 >=20 > vpanic() at panic+0x44 > pc =3D 0xffff0000004e1d10 lr =3D 0xffff0000004e1b84 > sp =3D 0xffff00013ef9fc50 fp =3D 0xffff00013ef9fd00 >=20 > panic() at data_abort+0x290 > pc =3D 0xffff0000004e1b84 lr =3D 0xffff00000082a3c8 > sp =3D 0xffff00013ef9fd10 fp =3D 0xffff00013ef9fd90 >=20 > data_abort() at handle_el1h_sync+0x78 > pc =3D 0xffff00000082a3c8 lr =3D 0xffff00000080a078 > sp =3D 0xffff00013ef9fda0 fp =3D 0xffff00013ef9fef0 >=20 > handle_el1h_sync() at thread_lock_flags_+0xbc > pc =3D 0xffff00000080a078 lr =3D 0xffff0000004b9024 > sp =3D 0xffff00013ef9ff00 fp =3D 0xffff00013ef9ff80 >=20 > thread_lock_flags_() at thread_lock_flags_+0xbc > pc =3D 0xffff0000004b9024 lr =3D 0xffff0000004b9024 > sp =3D 0xffff00013ef9ff90 fp =3D 0xffff00013ef9ffa0 >=20 > thread_lock_flags_() at sleepq_timeout+0x10 > pc =3D 0xffff0000004b9024 lr =3D 0xffff00000054b2a8 > sp =3D 0xffff00013ef9ffb0 fp =3D 0xffff00013ef9fff0 >=20 > sleepq_timeout() at softclock_call_cc+0x14c > pc =3D 0xffff00000054b2a8 lr =3D 0xffff000000503134 > sp =3D 0xffff00013efa0000 fp =3D 0xffff00013efa0060 >=20 > softclock_call_cc() at callout_process+0x17c > pc =3D 0xffff000000503134 lr =3D 0xffff000000502df0 > sp =3D 0xffff00013efa0070 fp =3D 0xffff00013efa00e0 >=20 > callout_process() at handleevents+0x188 > pc =3D 0xffff000000502df0 lr =3D 0xffff00000045b42c > sp =3D 0xffff00013efa00f0 fp =3D 0xffff00013efa0140 >=20 > handleevents() at timercb+0x304 > pc =3D 0xffff00000045b42c lr =3D 0xffff00000045be7c > sp =3D 0xffff00013efa0150 fp =3D 0xffff00013efa01b0 >=20 > timercb() at arm_tmr_intr+0x5c > pc =3D 0xffff00000045be7c lr =3D 0xffff0000007ff850 > sp =3D 0xffff00013efa01c0 fp =3D 0xffff00013efa0210 >=20 > arm_tmr_intr() at intr_event_handle+0xac > pc =3D 0xffff0000007ff850 lr =3D 0xffff000000493c54 > sp =3D 0xffff00013efa0220 fp =3D 0xffff00013efa0220 >=20 > intr_event_handle() at intr_isrc_dispatch+0x70 > pc =3D 0xffff000000493c54 lr =3D 0xffff0000007fb238 > sp =3D 0xffff00013efa0230 fp =3D 0xffff00013efa0270 >=20 > intr_isrc_dispatch() at arm_gic_v3_intr+0x11c > pc =3D 0xffff0000007fb238 lr =3D 0xffff00000080ff34 > sp =3D 0xffff00013efa0280 fp =3D 0xffff00013efa0290 >=20 > arm_gic_v3_intr() at intr_irq_handler+0x7c > pc =3D 0xffff00000080ff34 lr =3D 0xffff0000007faff0 > sp =3D 0xffff00013efa02a0 fp =3D 0xffff00013efa02f0 >=20 > intr_irq_handler() at handle_el1h_irq+0x74 > pc =3D 0xffff0000007faff0 lr =3D 0xffff00000080a140 > sp =3D 0xffff00013efa0300 fp =3D 0xffff00013efa0430 >=20 > handle_el1h_irq() at handle_el1h_sync+0x78 > pc =3D 0xffff00000080a140 lr =3D 0xffff00000080a078 > sp =3D 0xffff00013efa0440 fp =3D 0xffff00013efa0540 >=20 > handle_el1h_sync() at handle_el1h_sync+0x78 > pc =3D 0xffff00000080a078 lr =3D 0xffff00000080a078 > sp =3D 0xffff00013efa0550 fp =3D 0xffff00013efa06a0 >=20 > handle_el1h_sync() at sched_switch+0x6a8 > pc =3D 0xffff00000080a078 lr =3D 0xffff0000005197fc > sp =3D 0xffff00013efa06b0 fp =3D 0xffff00013efa0730 >=20 > sched_switch() at sched_switch+0x6a8 > pc =3D 0xffff0000005197fc lr =3D 0xffff0000005197fc > sp =3D 0xffff00013efa0740 fp =3D 0xffff00013efa07d0 >=20 > sched_switch() at mi_switch+0xf4 > pc =3D 0xffff0000005197fc lr =3D 0xffff0000004f03a0 > sp =3D 0xffff00013efa07e0 fp =3D 0xffff00013efa0830 >=20 > mi_switch() at sleepq_timedwait+0x28 > pc =3D 0xffff0000004f03a0 lr =3D 0xffff00000054bd0c > sp =3D 0xffff00013efa0840 fp =3D 0xffff00013efa0870 >=20 > sleepq_timedwait() at _cv_timedwait_sbt+0x110 > pc =3D 0xffff00000054bd0c lr =3D 0xffff00000045e7b0 > sp =3D 0xffff00013efa0880 fp =3D 0xffff00013efa0890 >=20 > _cv_timedwait_sbt() at zthr_procedure+0x20c > pc =3D 0xffff00000045e7b0 lr =3D 0xffff0000014e2fb0 > sp =3D 0xffff00013efa08a0 fp =3D 0xffff00013efa08f0 >=20 > zthr_procedure() at fork_exit+0x94 > pc =3D 0xffff0000014e2fb0 lr =3D 0xffff00000048fbf0 > sp =3D 0xffff00013efa0900 fp =3D 0xffff00013efa0950 >=20 > fork_exit() at fork_trampoline+0x10 > pc =3D 0xffff00000048fbf0 lr =3D 0xffff000000828ed8 > sp =3D 0xffff00013efa0960 fp =3D 0x0000000000000000 >=20 > KDB: enter: panic > [ thread pid 26 tid 100192 ] > Stopped at kdb_enter+0x48: undefined f906411f >=20 >=20 > Note: >=20 > All this started after a stress based I/O hangup test > that required a forced reboot. It is a bectl environment for that Optane: # bectl list BE Active Mountpoint Space Created 13S-CA72-nodbg R - 6.38G 2021-09-29 00:57 13_0R-CA72-nodbg N / 3.57G 2021-09-29 00:45 main-CA72-nodbg - - 7.99G 2021-09-29 00:42 13_0R-CA72-nodbg and main-CA72-nodbg boot fine, just 13S-CA72-nodbg has the problem. For reference: =3D> 40 1875384928 nda1 GPT (894G) 40 532480 nda1p1 CA72opt0EFI (260M) 532520 2008 - free - (1.0M) 534528 515899392 nda1p2 CA72opt0SWP (246G) 516433920 20971520 - free - (10G) 537405440 1337979528 nda1p3 CA72opt0ZFS (638G) Another attempt at booting have gotten reports of . . . lots of waiting notices . . . Root mount waiting for: CAM usbus1 Root mount waiting for: CAM usbus1 Root mount waiting for: CAM usbus1 Root mount waiting for: CAM usbus1 Root mount waiting for: CAM usbus1 x0: ffff000000e43ec8 (blocked_lock + 0) x1: ffff00013efa9f60 x2: ffff00000090e39a (cam_status_table + 1d132) x3: deadc0d8 x4: 0 x5: ffff00000082a138 (data_abort + 0) x6: 4 x7: 601 x8: ffff000000e43ec8 (blocked_lock + 0) x9: deadc0de x10: 0 x11: 3938700 x12: 0 x13: 8000 x14: 1dd x15: 81cd x16: 0 x17: 1 x18: ffff00013efa9f50 x19: ffff000000e43ec8 (blocked_lock + 0) x20: ffffa0000c806000 x21: 0 x22: ffffa0000c806000 x23: 0 x24: ffff000000bed000 (queue_ops + 0) x25: 98967f x26: ffff000000e43ee0 (blocked_lock + 18) x27: 0 x28: 114 x29: ffff00013efa9f50 sp: ffff00013efa9f50 lr: ffff0000004b9028 (thread_lock_flags_ + c0) elr: ffff0000004b9028 (thread_lock_flags_ + c0) spin lock 0xffff00004317b100 (sched lock 4) held by 0xffffa0000c806000 = (tid 100194) too long spin lock 0xffff00004317b100 (sched lock 4) held by 0xffffa0000c806000 = (tid 100194) too long spin lock 0xffff00004317b100 (sched lock 4) held by 0xffffa0000c806000 = (tid 100194) too long spsr: 2c5 far: deadc178 esr: 96000004 timeout stopping cpus panic: spin lock held too long cpuid =3D 9 time =3D 56 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x30 pc =3D 0xffff000000807770 lr =3D 0xffff00000011d9ec sp =3D 0xffff000156eea430 fp =3D 0xffff000156eea630 db_trace_self_wrapper() at vpanic+0x188 pc =3D 0xffff00000011d9ec lr =3D 0xffff0000004e1d10 sp =3D 0xffff000156eea640 fp =3D 0xffff000156eea6a0 vpanic() at panic+0x44 pc =3D 0xffff0000004e1d10 lr =3D 0xffff0000004e1b84 sp =3D 0xffff000156eea6b0 fp =3D 0xffff000156eea760 panic() at _mtx_lock_indefinite_check+0x8c pc =3D 0xffff0000004e1b84 lr =3D 0xffff0000004b8ea4 sp =3D 0xffff000156eea770 fp =3D 0xffff000156eea770 _mtx_lock_indefinite_check() at _mtx_lock_spin_cookie+0xb8 pc =3D 0xffff0000004b8ea4 lr =3D 0xffff0000004b8a0c sp =3D 0xffff000156eea780 fp =3D 0xffff000156eea790 _mtx_lock_spin_cookie() at sched_add+0x400 pc =3D 0xffff0000004b8a0c lr =3D 0xffff00000051ac94 sp =3D 0xffff000156eea7a0 fp =3D 0xffff000156eea7f0 sched_add() at kthread_add+0x238 pc =3D 0xffff00000051ac94 lr =3D 0xffff0000004a01e0 sp =3D 0xffff000156eea800 fp =3D 0xffff000156eea880 kthread_add() at vm_pageout+0x1b8 pc =3D 0xffff0000004a01e0 lr =3D 0xffff0000007e5a18 sp =3D 0xffff000156eea890 fp =3D 0xffff000156eea8f0 . . . Garbled . . . sp =3D 0xffff00013efa9bb0 fp =3D 0xffff00013efa9c10 vpanic() at panic+0x44 pc =3D 0xffff0000004e1d10 lr =3D 0xffff0000004e1b84 sp =3D 0xffff00013efa9c20 fp =3D 0xffff00013efa9cd0 panic() at data_abort+0x290 pc =3D 0xffff0000004e1b84 lr =3D 0xffff00000082a3c8 sp =3D 0xffff00013efa9ce0 fp =3D 0xffff00013efa9d60 data_abort() at handle_el1h_sync+0x78 pc =3D 0xffff00000082a3c8 lr =3D 0xffff00000080a078 sp =3D 0xffff00013efa9d70 fp =3D 0xffff00013efa9ec0 handle_el1h_sync() at thread_lock_flags_+0xbc pc =3D 0xffff00000080a078 lr =3D 0xffff0000004b9024 sp =3D 0xffff00013efa9ed0 fp =3D 0xffff00013efa9f50 thread_lock_flags_() at thread_lock_flags_+0xbc pc =3D 0xffff0000004b9024 lr =3D 0xffff0000004b9024 . . . Garbled . . . But after that the next attempt booted just fine. zpool scrub got: # zpool status pool: zopt0 state: ONLINE scan: scrub repaired 0B in 00:02:05 with 0 errors on Sun Nov 21 = 12:00:09 2021 config: NAME STATE READ WRITE CKSUM zopt0 ONLINE 0 0 0 nda1p3 ONLINE 0 0 0 errors: No known data errors Further reboots into 13S-CA72-nodbg have worked fine. For reference: The HoneyComb is a 16 Cortex-A72 system. I've no clue if releng/13 (-p5) or main could be induced to get similar behavior to what 13S-CA72-nodbg showed for the Optane media. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)