From nobody Thu Aug 25 03:57:00 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MCq0h0BlVz4ZhTw for ; Thu, 25 Aug 2022 03:57:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4MCq0f59J2z3DTv for ; Thu, 25 Aug 2022 03:57:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1661399824; bh=HmmdYBXOOwD9R0RRhn3uQiVIg39Link6xC7CCDFSUGM=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=LJXjcww5tIQ4qGBozkK2Rw+OREtBoaXKOd0nXguqvTKWxynt2utsct1evk8msQxlejqndbOF6wbw0GRtnAmqQLnXNEXbU5Xkmxza+ZZdVIFuj25NJSlrhz1vnwJAQGdpXzxV45ClaA5F717PLIi4Nl4RP4wKfZi94ef5Rw2CoAPmqRCbYUbtXtD+cz8LHQQp8QW28gbr4BHIbq7VGugVVzRBx2+QM+vi5fikrhZEda97dNS4Zd3P3Lo1u0Wa8EYH0SzpXk96vV+l7qKqOJH6caMhOG9n7BYqNu6rN7At5FO0zInZvaRGMwY3AD0HBz5WpM7K1xsbUvamsvjUjSFXDQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1661399824; bh=lh0MQajiawQ+vIgX9jL2z2u+HttL+dQBifm0LjtUT3a=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Edx0pdHqJ0QPYRdRSwmVX7Ep5MKsRbNRsPoX+I3SUnw/hgbpaezPE9OMXMVDBeSLmsYLK10e0qNnGCYwmn4o99b+Na4vmuxtUSdRa2IMxKsrkx9Gutf4NuA+fs38UMIhjDOujX9InyFp9uhInCD9xAHuNYxGw5pzUbLElp7igGKg72SRH8846n0PFur4sIgWzUAgJavFexoDUYuBtI9aQniiefM4XZGzAZJqZwf9GPp0Igx1Fn7n5Diytr9ioQYFbT5DXbNhdoegDJpoys1kzh8duD0/gWaLmBYnOy/tB2549X1b0PHMsxvPFuj1wgWhSwwVzEy6uTSpduvzphF5AA== X-YMail-OSG: selVKtIVM1n6Dj3EdxOxTMJE9h2J4iyZ.sxoMZK3PVZ.3Pwp3Aqp5rIourMU2Sf H1ahVJO45VUl7HjMCFs13SEjXr03KOeA368njZH8BCqHGFe.iUS4Zcev2lV4ODS6Bdq2sSqi8s7R ZTt7uDg7WsxK4cDNsqFnJaXTzbTsFhgQc8LuGsffS4JHctXvRy6zv.wkw83imm_tpEvfBUGuHPuR SrsAVloOTG.7c3HK1izt4qZFfN0vqyhUwdLi1ASEfmzrc2PGSDIIBwbWJJ5J2enioW0q5GPTLeIF LxP7W3w6u39YXPQ2wiee5hATXXsbATh4J08JosvgzpM7zFrMBL5WPBvMdSaJlmOMX0NVhgYxuNXM .Au_m5DABt3G2UDf4lXpfAqVFhz5k7c7KhY51RzPvl7I80NbiZa3A6Eoys0IvwjceNFk4ov40Ot9 sg.k2ecbNQik6X0ycnrxFLZqjabZXidW3LTVyspmlDcWyqnKHko5gWLy098XT5tepL_3YdEsbYQ1 WQtZD1TuPnUzzbSSpq6Wefv3hEnp8fCDCE7XhydCVqK090ALNS33XxResY5_oEoxGjOat291MGHP KA2Zf0iE2f_.nXkMSbu4_E_sqBQKQ_YA7oRLP5XlLA5jnCTXafG1Hh8f0qdPuTMLR5jNLEUh07z3 VG2TvXet1UZ4w87ggSF.D9jl6siZcHMWoqDafbzFTqFwsHXAGMtkMYhfhK8ygMubngJ2Yblkk_PY 4MW6.a5wC_U8qM71H0A2juO3b6vV7J264qTZS4No.VnPQhg4O12D6P5zwCCgoTEirvc_YQEX9bSe 2G.bnhqGCo35C12ZmI4ni7sWatkypdz4fcupbg4Iv2VZ0O65xHC90mmYIm2x5smBTMGrXuKHV5ny 83RIe1Df8lbZxvfSmgE8E7G8jp_zTbfPi6gy38l6YxDO223EXG9sVI0WHMG06hFUje40xO8832BC 0k167SJ69tEbWjHj.cyoGsZwjo.iFyPHwSltadINr0Rg87QKj1U7GD4JN.87SE5wYuDuhinhSSYm YJZHeW6bTDZFKRmwshrYv85afPNB36oEV2ilgI2u_RpI6SBwnDfkBTupoB9Sn9C._TQDcLQ0j7SZ KyJ4n6Px5LCYHy8a93OFc.hzxNw.pcjfOhvN0L_defRARRxOILbJkfHgj3Is95GWjwmhS32ohETL 0yh9yImcb7J76Dyr6q47MT1hjvm3d_3MTDOFy_xQnTUx9Jw82myFuHSDFRepf8ztYpKnDRwIK5wV KNLfrhvjsIeZMmNqRzDB0nmwYEnfk2Uuqgn3EhXd7EO7QFzLNO_nMLT76iGxtW5q0I5n7ax_h0sv 6yZ.qewx.dJ2xbp86A9MPa55YcTc8HIwkMiWtJSqggpasxMpvipEPtYn1Dc2s1kCrkJRo.FRm5dr a1EtJecknY_UZYbux33mBAcpnS_Ubs79rsc_4lW2OS9KNgDRO1QsYFuglI.Yd2CKeXZ0fHiSbBFP KUbx.o1OPU0RJ030XOboxS7vjBi57OQ2w3ViDe0thS7PUhNoC9ppST7OXWbit4czhQYCFUc6HR5K FBJIHAUUIYXuJln1sCkJa_g5OA3oIKyxL51mmEnWd5.R34lddyYSTi6HL3IndkFlfD2w_n8IFrpJ 7nI16COIaluurY2vGwSVIqyzVWtDQLvZotPi0o7YQb3sVQRh4LQW5jBRYtKu3PXiVs_99Uy1XLOj mYdqOhf_wwAuNh_A08XWBNCE.sfKbHaK5Tx7j_ZuYrPfQX6SHGkq0olZBp2sbebwpvLokZgXQ8wB NKfOZiyP2.kt7Xyy2l2W9xHtz1YdmrnM082P68ZU9YtAkxqa6yzVJ.g9JTlQaZBgLfcEFRb.C075 wPj4NZfqrYIl.DRwBkaFhRCq0DFKUQaGilcGG_XZJNTYD4Zcp5Jp8_J3aECSqFecM_IayIqZm_TY WfYWzBU2yFBBMRan7l1HHy7OM17sKwH6Lu8z.tcDwCqEYhXH46rbhWNX_M_piEWMW.d5TxtVl024 12xJqDxlSZlC2YCYSetQQx.wSOPTaouD5gdUCJd3p3Nydc.LiURPepXhef0xpSRZe5F7OHPydtlT nIoMFHZFowyxOFwe4OKZCcAOWsz.3kjAFFrpwuSH0MKNLADGhUIupuK1CG13RVxdbDc3d6A386Vf FVfInN7H4FXth3mt6dudTAaVXrvAySfCK5ddzVNMb_q6Ko1tXEVpLwmFWk806AAbemzFNthlIqmK Gmf9exKqsdlyHZBHBLEtUDEdMuJf03yqPadptioVLVeC8GD3mB35S6qM25MpS X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Thu, 25 Aug 2022 03:57:04 +0000 Received: by hermes--production-gq1-686964ccb6-z5t9k (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2bc86e8e7f7c8aaeb13575f9069afd46; Thu, 25 Aug 2022 03:57:00 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: How to get past "internal error: cannot import 'zroot': Integrity check failed" (no ability to import the pool)? Message-Id: <52858985-533A-4D68-A2E8-54251DFB0215@yahoo.com> Date: Wed, 24 Aug 2022 20:57:00 -0700 To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3654.120.0.1.13) References: <52858985-533A-4D68-A2E8-54251DFB0215.ref@yahoo.com> X-Rspamd-Queue-Id: 4MCq0f59J2z3DTv X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=LJXjcww5; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-2.49 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N I seem to have gotten into a state where no zpool related command that required identification of a pool (such as by name) can work because import can not make the zpool available. (I give more context later.) How do I re-establish the freebsd-zfs partition into a form that I can repopulate it when its failed pool can not be imported? I'm appearently limited to zpool commands that reference the device instead of the pool (name) because the likes of "zpool import -f -FX . . ." leads to a panic. Note that this was for media that used zfs just to use bectl, not for other typical zfs reasons. For example, redundancy was-not/is-not a goal. For reference: =3D> 40 3907029088 da0 GPT (1.8T) 40 32728 - free - (16M) 32768 524288 1 efi (256M) 557056 7340032 2 freebsd-swap (3.5G) 7897088 26214400 - free - (13G) 34111488 20971520 3 freebsd-swap (10G) 55083008 12582912 - free - (6.0G) 67665920 29360128 4 freebsd-swap (14G) 97026048 4194304 - free - (2.0G) 101220352 33554432 5 freebsd-swap (16G) 134774784 67108864 6 freebsd-swap (32G) 201883648 364904448 7 freebsd-swap (174G) 566788096 2795503616 8 freebsd-zfs (1.3T) 3362291712 544737416 - free - (260G) At this point no attempt to preserve the content of the freebsd-zfs partition seems a likely way of going. But I'm unclear on how to even start over, given no ability to make the pool accessible by name. The sequence leading to how things are went like . . . # git -C /usr/ports fetch error: error reading from .git/objects/pack/pack-8e819c78469vm_fault: = pager read error, pid 1370 (git) c212148fe5d3922cc807e6858768e.pack: Input/output error vm_fault: pager read error, pid 1370 (git) . . . # bectl activate main-CA72 panic: VERIFY3(0 =3D=3D bpobj_open(&bpo, dl->dl_os, dlce->dlce_bpobj)) = failed (0 =3D=3D 97) cpuid =3D 0 time =3D 1661389515 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x30 pc =3D 0xffff0000007fcfd0 lr =3D 0xffff000000101b80 sp =3D 0xffff0000b49f4ee0 fp =3D 0xffff0000b49f50e0 db_trace_self_wrapper() at vpanic+0x178 pc =3D 0xffff000000101b80 lr =3D 0xffff0000004cef08 sp =3D 0xffff0000b49f50f0 fp =3D 0xffff0000b49f5150 vpanic() at spl_panic+0x40 pc =3D 0xffff0000004cef08 lr =3D 0xffff00000129f360 sp =3D 0xffff0000b49f5160 fp =3D 0xffff0000b49f51f0 spl_panic() at dsl_deadlist_space_range+0x264 pc =3D 0xffff00000129f360 lr =3D 0xffff00000133d4f4 sp =3D 0xffff0000b49f5200 fp =3D 0xffff0000b49f53c0 dsl_deadlist_space_range() at snaplist_space+0x4c pc =3D 0xffff00000133d4f4 lr =3D 0xffff00000133899c sp =3D 0xffff0000b49f53d0 fp =3D 0xffff0000b49f5440 snaplist_space() at dsl_dataset_promote_check+0x648 pc =3D 0xffff00000133899c lr =3D 0xffff0000013385e8 sp =3D 0xffff0000b49f5450 fp =3D 0xffff0000b49f5530 dsl_dataset_promote_check() at dsl_sync_task_sync+0xcc pc =3D 0xffff0000013385e8 lr =3D 0xffff00000135fb3c sp =3D 0xffff0000b49f5540 fp =3D 0xffff0000b49f5590 dsl_sync_task_sync() at dsl_pool_sync+0x3cc pc =3D 0xffff00000135fb3c lr =3D 0xffff00000135251c sp =3D 0xffff0000b49f55a0 fp =3D 0xffff0000b49f55e0 dsl_pool_sync() at spa_sync+0x8f8 pc =3D 0xffff00000135251c lr =3D 0xffff00000138be28 sp =3D 0xffff0000b49f55f0 fp =3D 0xffff0000b49f57f0 spa_sync() at txg_sync_thread+0x1d8 pc =3D 0xffff00000138be28 lr =3D 0xffff0000013a2bf8 sp =3D 0xffff0000b49f5800 fp =3D 0xffff0000b49f58f0 txg_sync_thread() at fork_exit+0x88 pc =3D 0xffff0000013a2bf8 lr =3D 0xffff00000047c568 sp =3D 0xffff0000b49f5900 fp =3D 0xffff0000b49f5950 fork_exit() at fork_trampoline+0x14 pc =3D 0xffff00000047c568 lr =3D 0xffff00000081edd4 sp =3D 0xffff0000b49f5960 fp =3D 0x0000000000000000 KDB: enter: panic [ thread pid 4 tid 100198 ] Stopped at kdb_enter+0x48: undefined f907011f I was unable to boot from the media after this. Plugged the media into another machine . . . # zpool import -F -n zroot cannot import 'zroot': pool was previously in use from another system. Last accessed by (hostid=3D0) at Wed Aug 24 18:05:15 2022 The pool can be imported, use 'zpool import -f' to import the pool. # zpool import -f zroot Aug 24 18:13:47 CA72_16Gp_ZFS ZFS[1588]: failed to load zpool zroot Aug 24 18:13:47 CA72_16Gp_ZFS ZFS[1612]: failed to load zpool zroot Aug 24 18:13:47 CA72_16Gp_ZFS ZFS[1616]: failed to load zpool zroot internal error: cannot import 'zroot': Integrity check failed Abort trap (core dumped) # gdb zpool zpool.core . . . Core was generated by `zpool import -f zroot'. Program terminated with signal SIGABRT, Aborted. Sent by thr_kill() from pid 1716 and user 0. #0 thr_kill () at thr_kill.S:4 4 RSYSCALL(thr_kill) (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x00002b0c6f3794f0 in __raise (s=3Ds@entry=3D6) at = /usr/main-src/lib/libc/gen/raise.c:52 #2 0x00002b0c6f420494 in abort () at = /usr/main-src/lib/libc/stdlib/abort.c:67 #3 0x00002b0c69415744 in zfs_verror (hdl=3D0x2b0c76263000, error=3D2092, = fmt=3Dfmt@entry=3D0x2b0c693d3135 "%s", ap=3D...) at = /usr/main-src/sys/contrib/openzfs/lib/libzfs/libzfs_util.c:344 #4 0x00002b0c69416324 in zpool_standard_error_fmt = (hdl=3Dhdl@entry=3D0x2b0c76263000, error=3Derror@entry=3D97, = fmt=3D0x2b0c693d3135 "%s") at = /usr/main-src/sys/contrib/openzfs/lib/libzfs/libzfs_util.c:729 #5 0x00002b0c69415ec8 in zpool_standard_error (hdl=3D0x0, = hdl@entry=3D0x2b0c76263000, error=3D0, error@entry=3D97, = msg=3D0x2b0c6ea23350 <__thr_sigprocmask> "\377\203",=20 msg@entry=3D0x2b0c665668e8 "cannot import 'zroot'") at = /usr/main-src/sys/contrib/openzfs/lib/libzfs/libzfs_util.c:619 #6 0x00002b0c6940687c in zpool_import_props (hdl=3D0x2b0c76263000, = config=3Dconfig@entry=3D0x2b0c95939080, newname=3Dnewname@entry=3D0x0, = props=3Dprops@entry=3D0x0, flags=3Dflags@entry=3D2) at /usr/main-src/sys/contrib/openzfs/lib/libzfs/libzfs_pool.c:2193 #7 0x00002b0be60f3344 in do_import (config=3D0x2b0c95939080, = newname=3D0x0, mntopts=3D0x0, props=3Dprops@entry=3D0x0, = flags=3Dflags@entry=3D2) at = /usr/main-src/sys/contrib/openzfs/cmd/zpool/zpool_main.c:3190 #8 0x00002b0be60f3108 in import_pools = (pools=3Dpools@entry=3D0x2b0c762780e0, props=3D, = mntopts=3Dmntopts@entry=3D0x0, flags=3Dflags@entry=3D2, = orig_name=3D0x2b0c7622d028 "zroot", new_name=3D0x0,=20 do_destroyed=3Ddo_destroyed@entry=3DB_FALSE, = pool_specified=3Dpool_specified@entry=3DB_TRUE, do_all=3DB_FALSE, = import=3D0x2b0c665684a0) at = /usr/main-src/sys/contrib/openzfs/cmd/zpool/zpool_main.c:3318 #9 0x00002b0be60e9074 in zpool_do_import (argc=3D1, argv=3D) at /usr/main-src/sys/contrib/openzfs/cmd/zpool/zpool_main.c:3804 #10 0x00002b0be60e3ce8 in main (argc=3D4, argv=3D) at = /usr/main-src/sys/contrib/openzfs/cmd/zpool/zpool_main.c:10918 (gdb) quit # zpool import -f -FX -N -R /zroot-mnt -t zroot zprpi . . . evantually . . . panic: Solaris(panic): zfs: adding existent segment to range tree = (offset=3D7a001ba000 size=3D40000) cpuid =3D 8 time =3D 1661395806 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 vcmn_err() at vcmn_err+0x10c zfs_panic_recover() at zfs_panic_recover+0x64 range_tree_add_impl() at range_tree_add_impl+0x184 range_tree_walk() at range_tree_walk+0xa4 metaslab_load() at metaslab_load+0x6a4 metaslab_preload() at metaslab_preload+0x8c taskq_run() at taskq_run+0x1c taskqueue_run_locked() at taskqueue_run_locked+0x190 taskqueue_thread_loop() at taskqueue_thread_loop+0x130 fork_exit() at fork_exit+0x88 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 6 tid 108968 ] Stopped at kdb_enter+0x44: undefined f907c27f db>=20 I had to unplug the disk to avoid reboots simply retrying the import and crashing the same way again. SIDE NOTE: Then, on reboot, I saw the following: . . . Setting hostid: 0x6522bfc4. cannot import 'zroot': no such pool or dataset Destroy and re-create the pool from a backup pid 49 (zpool) is attempting to use unsafe AIO requests = - not logging anymore pid 49 (zpool), jid 0, uid 0: exited on signal 6 source. cachefile import failed, retrying nvpair_value_nvlist(nvp, &rv) =3D=3D 0 (0x16 =3D=3D 0) ASSERT at = /usr/main-src/sys/contrib/openzfs/module/nvpair/fnvpair.c:592:fnvpair_valu= e_nvlist()Abort trap Import of zpool cache /etc/zfs/zpool.cache failed, will retry after root = mount hold release cannot import 'zroot': no such pool or dataset Destroy and re-create the pool from a backup source. cachefile imporpid 55 (zpool), jid 0, uid 0: exited on signal 6 t failed, retrying nvpair_value_nvlist(nvp, &rv) =3D=3D 0 (0x16 =3D=3D 0) ASSERT at = /usr/main-src/sys/contrib/openzfs/module/nvpair/fnvpair.c:592:fnvpair_valu= e_nvlist()Abort trap Starting file system checks: /dev/gpt/CA72opt0EFI: 281 files, 231 MiB free (14770 clusters) FIXED . . . Removing /etc/zfs/zpool.cache allowed reboots to avoid such. END SIDE NOTE. For reference, for the machine where I can plug in the media: # uname -apKU # line split for better readability FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #59 main-n256584-5bc926af9fd1-dirty: Wed Jul 6 18:10:52 PDT 2022 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400063 1400063 =3D=3D=3D Mark Millard marklmi at yahoo.com