From nobody Thu Oct 13 10:17:11 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mp56c3zbhz4fd6j for ; Thu, 13 Oct 2022 10:17:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mp56c1KpMz42Wl for ; Thu, 13 Oct 2022 10:17:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Mp56c0G07zNpl for ; Thu, 13 Oct 2022 10:17:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29DAHBa7023411 for ; Thu, 13 Oct 2022 10:17:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29DAHB3V023410 for bugs@FreeBSD.org; Thu, 13 Oct 2022 10:17:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 267009] OpenZFS: panic: VERIFY3(0 == nvlist_lookup_uint64(nvl, name, &rv)) failed (0 == 22) Date: Thu, 13 Oct 2022 10:17:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: Trond.Endrestol@ximalas.info X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1665656232; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=h+HZEcrynnI9Pp3BVY5v1mJGBEZXx8JNmPPaD6UUA7w=; b=qUJbkFWseHKArGaa2CAHAXS3VdgqJ4CSXq8QyZizM82O+QDzRbOIxaIlBq3dispIkPIAOd hm8F84PPgFwN5fc2urqO5LKY0CKXe9T7bTwXH2j36tYRfJ0shugkuYMAzAaYzWQ8mQefBS WoeMAZouxYL//gpBW6f639XTQtqVUt/Yl7QPnWuYluZC32upPRE1G+rfX2IEJUIByM1MHY yRJEi4i7Pwt8hX0Xl2d7sPowfGFEBVcc6ZDlZh2qogIgy+lBmueiqeGIwRVsNmJkyqvvOR 11T7FLzdFdBA08mCmn8J0Sphry9MNAdrX01w3F/OlM0wsjwuxjhGWH8k+hBK0Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1665656232; a=rsa-sha256; cv=none; b=tjS46k4b3htcnX9efWqioJnVe1xMOnFDIf9ifINLQQP6C2qhWQ//GiJ2D85ZLrDfAGaY8O dnKc4rZWfrlm3QDg4KhoN9p8sQIoY2jfGUnUCm9lwwshKAUNAkHAzk1i6/Vzh+kqFOPdRS xJW/VHhA74xdHveLYFqDUJpZ0QRa/16aaqXmHp9xH9RyJPs+AAfvD0UNpyNQJoVDo9Ie5a ZyaX3xFZy9ZdcZybmJpW5SxjQbaYJLOj4Dj5RTil/pWHRw10FQdUU+a0fCV8uAgIMEDCN4 e+XpidMDmYPsvMDvMB+62xTqMtWaDHRH+8HvsRqfTME7k5wLHS/5rcuIkUDAmA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267009 Bug ID: 267009 Summary: OpenZFS: panic: VERIFY3(0 =3D=3D nvlist_lookup_uint64(n= vl, name, &rv)) failed (0 =3D=3D 22) Product: Base System Version: 13.1-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: Trond.Endrestol@ximalas.info My zpools looked very much like this to begin with: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRA= G=20=20=20 CAP DEDUP HEALTH ALTROOT builder01_zroot 164G 117G 47.2G - - 14= %=20=20=20 71% 1.00x ONLINE - raidz1-0 164G 117G 47.2G - - 14= %=20 71.2% - ONLINE gpt/builder01_zroot0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot2 - - - - - = -=20=20=20 - - ONLINE logs - - - - - = -=20=20=20 - - - mirror-1 16.5G 204K 16.5G - - 0= %=20 0.00% - ONLINE gpt/builder01_zroot_zlog0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot_zlog1 - - - - - = -=20=20=20 - - ONLINE builder01_zwork 374G 237G 137G - - 38= %=20=20=20 63% 1.00x ONLINE - raidz1-0 374G 237G 137G - - 38= %=20 63.3% - ONLINE gpt/builder01_zwork0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork2 - - - - - = -=20=20=20 - - ONLINE logs - - - - - = -=20=20=20 - - - mirror-1 16.5G 0K 16.5G - - 0= %=20 0.00% - ONLINE gpt/builder01_zwork_zlog0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork_zlog1 - - - - - = -=20=20=20 - - ONLINE I wanted to remove the mirrored slogs, resize and re-add them. They were originally almost 17 GiB by an oversight, and I wanted them to be 16.0 GiB. I don't know if I should have run zpool labelclear on the slog partitions before resizing, I didn't. zpool remove builder01_zroot mirror-1 zpool remove builder01_zwork mirror-1 gpart resize -i 1 -s 16G xbd4 gpart resize -i 1 -s 16G xbd5 gpart resize -i 1 -s 16G xbd9 gpart resize -i 1 -s 16G xbd10 zpool add builder01_zroot log mirror gpt/builder01_zroot_zlog0 gpt/builder01_zroot_zlog1 zpool add builder01_zwork log mirror gpt/builder01_zwork_zlog0 gpt/builder01_zwork_zlog1 The listing looked very much like this: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRA= G=20=20=20 CAP DEDUP HEALTH ALTROOT builder01_zroot 164G 117G 47.2G - - 14= %=20=20=20 71% 1.00x ONLINE - raidz1-0 164G 117G 47.2G - - 14= %=20 71.2% - ONLINE gpt/builder01_zroot0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot2 - - - - - = -=20=20=20 - - ONLINE logs - - - - - = -=20=20=20 - - - mirror-2 15.5G 68K 15.5G - - 0= %=20 0.00% - ONLINE gpt/builder01_zroot_zlog0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot_zlog1 - - - - - = -=20=20=20 - - ONLINE builder01_zwork 374G 237G 137G - - 38= %=20=20=20 63% 1.00x ONLINE - raidz1-0 374G 237G 137G - - 38= %=20 63.3% - ONLINE gpt/builder01_zwork0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork2 - - - - - = -=20=20=20 - - ONLINE logs - - - - - = -=20=20=20 - - - mirror-2 15.5G 0K 15.5G - - 0= %=20 0.00% - ONLINE gpt/builder01_zwork_zlog0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork_zlog1 - - - - - = -=20=20=20 - - ONLINE I noticed mirror-1 became mirror-2 for both pools. I expected the pairs wou= ld be named mirror-1. Upon reboot, I got the panic below. I booted from a DVD, removed the mirrored slogs from both pools, and the sy= stem could again boot from the root pool. I re-added the mirrored slogs to the work pool while the system was running. The listing now looks like this: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRA= G=20=20=20 CAP DEDUP HEALTH ALTROOT builder01_zroot 164G 117G 47.2G - - 14= %=20=20=20 71% 1.00x ONLINE - raidz1-0 164G 117G 47.2G - - 14= %=20 71.2% - ONLINE gpt/builder01_zroot0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zroot2 - - - - - = -=20=20=20 - - ONLINE builder01_zwork 374G 237G 137G - - 38= %=20=20=20 63% 1.00x ONLINE - raidz1-0 374G 237G 137G - - 38= %=20 63.3% - ONLINE gpt/builder01_zwork0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork1 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork2 - - - - - = -=20=20=20 - - ONLINE logs - - - - - = -=20=20=20 - - - mirror-3 15.5G 0K 15.5G - - 0= %=20 0.00% - ONLINE gpt/builder01_zwork_zlog0 - - - - - = -=20=20=20 - - ONLINE gpt/builder01_zwork_zlog1 - - - - - = -=20=20=20 - - ONLINE Maybe the recent OpenZFS commits fixes this issue. If not, maybe the test s= uite should be extended to cover the kernel's ability to mount a root pool where= the vdevs are numbered non-contiguously, if this is what triggers the panic. Note, local branch commit 7806d3b0243f... as indicated in the BE's name, corresponds to src/stable/13 commit 3ea8c7ad90f75129c52a2b64213c5578af23dc8= d, dated Tue Aug 9 15:47:40 2022 -0400. Here's the panic message, screenshotted, OCR-ed, and edited by hand: Trying to mount root from zfs:builder01_zroot/ROOT/20220810-190437-stable-13-local-n252030-7806d3b024= 3f [] ... cd0 at ata1 bus 0 scbus1 target 1 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number QM00004 cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present panic: VERIFY3(0 =3D=3D nvlist_lookup_uint64(nvl, name, &rv)) failed (0 =3D= =3D 22) cpuid =3D 0 time =3D 1 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff805b804b =3D db_trace_self_urapper+0x2b/frame 0xfffffe009cdec580 vpanic() at 0xffffffff80806fb1 =3D vpanic+0x151/frame 0xfffffe009cdecSd8 spl_panic() at 0xffffffff8036391a =3D spl_panic+0x3a/frame 0xfffffe009cdec6= 30 fnvlist_lookup_uint64() at 0xffffffff80385ef3 =3D fnvlist_lookup_uint64+0x43/frame 0xfffffe009cdec650 spa_import_rootpool() at 0xffffffff8038d10e =3D spa_import_rootpool+0x5e/fr= ame 0xfffffe009cdec6c0 zfs_mount() at 0xffffffff8039aaaf =3D zfs_mount+0x41f/frame 0xfffffe009cdec= 850 vfs_domount_first() at 0xffffffff808e1f03 =3D vfs_domount_first+0x213/frame 0xfffffe009cdec980 vfs_domount() at 0xffffffff808de855 =3D vfs_domount+0x2b5/frame 0xfffffe009cdecab0 vfs_donmount() at 0xffffffff808ddd85 =3D vfs_donmount+0x8d5/frame 0xfffffe009cdecb50 kernel_mount() at 0xffffffff808e100d =3D kernel_mount+0x3d/frame 0xfffffe009cdecba0 parse_mount() at 0xffffffff808e5364 =3D parse_mount+0x4d4/frame 0xfffffe009cdecce0 vfs_mountroot() at 0xffffffff808e37b3 =3D vfs_mountroot+0x763/frame 0xfffffe009cdece50 start_init() at 0xffffffff807932c3 =3D start_init+0x23/frame 0xfffffe009cde= cef0 fork_exit() at 0xffffffff807c2a9e =3D fork_exit+0x7e/trame 0xfffffe009cdecf= 30 fork_trampoline() at 0xffffffff80baf89e =3D fork_trampoline+0xe/frame 0xfffffe009cdecf30 --- trap 0x9ce4aa98, rip=3D 0xffffffff8079288f, rsp =3D 0, rbp =3D 0x20014 = --- mi_startup() at 0xffffffff8079200f =3D mi_startup+0xdf/frame 0x20014 Uptime: ls Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. --=20 You are receiving this mail because: You are the assignee for the bug.=