From nobody Thu Jan 18 14:56:49 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TG5S15Mb4z58SjT for ; Thu, 18 Jan 2024 14:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TG5S14B2Hz4gkL for ; Thu, 18 Jan 2024 14:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705589809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BNRrlz2nfVsdHMnEWn7vtSopi3ZHAwwlCEatJFfWE+k=; b=PgHNlkD6aYuSDqapC8ifiD+xys58S/9KlSPJukVxmgmyRwKBGmtUTKdtN2vJ2TBrrCc3N3 KUU7nE0R5nVidS0o4STTEQbXI5TUc8eeuNLD7U9VYYVxKi+NyoISinia16Rg8Nbv1eHYTp 8T/qYYOUif0FCTJT7FU779fDwhNP+uv0pJN2npt14AoGMQByXl2TVVs2Ac4IMoTvKq7duO zXtSvBofR75T1PXuGWPo7nsDcYHb9AShDiOZ6aIqnZlUiUaactTOjhSbaiOnud8RWhj4Nr xxW+o65wJO25ic+nM5l/resy8N4GXzbeokPB8YLKLfZXPC3bMADubn1EzQj+Mw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705589809; a=rsa-sha256; cv=none; b=M7U9DtgSigcsyZ4CYzfovXGe5/WhWUXg+RfMDlTgyzmLjgvsMnYWau3qN7j2L7jyNUk5FF piaR5+LDiQu6K7/v3DP5auM1JTiq2yoZ1RSrq6qjd9M3+vfSeoW4xf8qD60Z92I0FfPAZk g9mDyhkfgqE+dhM4z8PQIDtXKgwUc8q8NxJY0nov9TuzQWN0rqVKJlLWycN6q7VQkVT3HG XDM7iqUJUQIjeIsr+70LLm4DRsgkca942c2FTuOJNRlbTyoNBTD5TKIjnabRpqRViNgOIA dGmTiKkftSTbPmuNE6ygjYEy9VQ4LYrsl3TeaBUR4WHtn/SacdRKUgYf+P/5/w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TG5S135Nnz19ys for ; Thu, 18 Jan 2024 14:56:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40IEunIm046636 for ; Thu, 18 Jan 2024 14:56:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40IEunmn046635 for fs@FreeBSD.org; Thu, 18 Jan 2024 14:56:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 276420] zfs panic: VERIFY(zio->io_stage == ZIO_STAGE_VDEV_IO_START) failed Date: Thu, 18 Jan 2024 14:56:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276420 --- Comment #2 from John F. Carr --- I have also reproduced the original panic with a crash dump. panic: VERIFY(zio->io_stage =3D=3D ZIO_STAGE_VDEV_IO_START) failed It is from zio_vdev_io_bypass. #5 0xffffffff82c2209e in zio_vdev_io_bypass ( zio=3Dzio@entry=3D0xfffff80767ed29a0) at /usr/home/jfc/freebsd/src/sys/contrib/openzfs/module/zfs/zio.c:4093 No locals. #6 0xffffffff82b70926 in vdev_queue_io_done ( zio=3Dzio@entry=3D0xfffff80767ed29a0) at /usr/home/jfc/freebsd/src/sys/contrib/openzfs/module/zfs/vdev_queue.c:951 zl =3D 0xfffff80763b9d600 vq =3D 0xfffffe03a677b1c0 now =3D nio =3D 0xfffff807698819a0 dio =3D #7 0xffffffff82c27d1f in zio_vdev_io_done (zio=3D0xfffff80767ed29a0) at /usr/home/jfc/freebsd/src/sys/contrib/openzfs/module/zfs/zio.c:3908 vd =3D 0xfffffe03a6778000 ops =3D 0xffffffff82d0e530 unexpected_error =3D 0 #8 0xffffffff82c1ff5b in __zio_execute (zio=3D0xfffff80767ed29a0) at /usr/home/jfc/freebsd/src/sys/contrib/openzfs/module/zfs/zio.c:2219 stage =3D ZIO_STAGE_VDEV_IO_DONE pipeline =3D #9 zio_execute (zio=3D) at /usr/home/jfc/freebsd/src/sys/contrib/openzfs/module/zfs/zio.c:2130 cookie =3D 0 The zio object is {io_bookmark =3D {zb_objset =3D 2437, zb_object =3D 1, zb_level =3D 0, zb_blkid =3D 776249}, io_prop =3D {zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_complevel =3D 0 '\000', zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\000', zp_copies =3D 0= '\000', zp_dedup =3D 0, zp_dedup_verify =3D 0, zp_nopwrite =3D 0, zp_en= crypt =3D 0, zp_byteorder =3D 0, zp_salt =3D "\000\000\000\000\000\000\000", zp_iv =3D '\000' < repeats 11 times >, zp_mac =3D '\000' < repeats 15 times >, zp_zpl_smallblk =3D 0}, io_type =3D ZIO_TYPE_WRITE, io_child_type =3D ZIO_CHILD_VDEV, io_trim_flags =3D 0, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_ASYNC_WRITE, io_reexecute =3D 0 '\000', io_state =3D "\001", io_txg =3D 23709913, io_spa =3D 0xfffffe03a66d8000, io_bp =3D 0x0, io_bp_override =3D 0x0, io_bp_copy =3D {blk_dva =3D {{dva_word =3D {0, 0}}, {dva_word =3D {0, 0}}, {dva_word = =3D {0, 0}}}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth = =3D 0, blk_fill =3D 0, blk_cksum =3D {zc_word =3D {0, 0, 0, 0}}}, io_parent_list =3D {list_size =3D 48, list_offset =3D 16, list_head =3D {list_next =3D 0xfffff80768647490, list_prev =3D 0xfffff80768647490}}, io_child_list =3D {list_size =3D 48, list_offset =3D 32, list_head =3D {list_next =3D 0xfffff80763b9d620, list_prev =3D 0xfffff80763b9d620}}, io_logical =3D 0xfffff8075b3f94d0, io_transform_stack =3D 0x0, io_ready =3D 0x0, io_children_ready =3D 0x0, io_physdone =3D 0xffffffff82a7cd30 < arc_write_physdone >, io_done =3D 0xffffffff82b71880 < vdev_raidz_child_done >, io_private =3D 0xfffff8047802ea70, io_prev_space_delta =3D 0, io_bp_orig =3D {blk_dva =3D {{dva_word =3D {0, 0}}, {dva_word =3D {0, 0}}, {dva_word = =3D {0, 0}}}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth = =3D 0, blk_fill =3D 0, blk_cksum =3D {zc_word =3D {0, 0, 0, 0}}}, io_lsize =3D 8192, io_abd =3D 0xfffff8047802ea88, io_orig_abd =3D 0xfffff8047802ea88, io_size =3D 8192, io_orig_size =3D 8192, io_vd =3D 0xfffffe03a6778000, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_metaslab_class =3D 0xfffff8010a80a000, io_offset =3D 899567525888, io_timestamp =3D 2052263446783, io_queued_timestamp =3D 2052263446703, io_target_timestamp =3D 0, io_delta =3D 4538041, io_delay =3D 0, io_queue_node =3D {avl_child =3D {0x0, 0x0}, avl_pcb =3D 18446735309376501= 621}, io_offset_node =3D {avl_child =3D {0x0, 0x0}, avl_pcb =3D 1844673530946682= 2541}, io_alloc_node =3D {avl_child =3D {0x0, 0x0}, avl_pcb =3D 0}, io_alloc_list =3D {zal_list =3D {list_size =3D 72, list_offset =3D 0, list_head =3D {list_next =3D 0xfffff80767ed2c= 98, list_prev =3D 0xfffff80767ed2c98= }}, zal_size =3D 0}, io_flags =3D 1575040, io_stage =3D ZIO_STAGE_VDEV_IO_DONE, io_pipeline =3D (ZIO_STAGE_VDEV_IO_START | ZIO_STAGE_VDEV_IO_DONE | ZIO_STAGE_VDEV_IO_ASSESS | ZIO_STAGE_DONE), io_orig_flags =3D 1048704, io_orig_stage =3D ZIO_STAGE_READY, io_orig_pipeline =3D (ZIO_STAGE_VDEV_IO_START | ZIO_STAGE_VDEV_IO_DONE | ZIO_STAGE_VDEV_IO_ASSESS | ZIO_STAGE_DONE), io_pipeline_trace =3D (ZIO_STAGE_OPEN | ZIO_STAGE_VDEV_IO_START | ZIO_STAGE_VDEV_IO_DONE), io_error =3D 0, io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 1}, {0, 0}, {0, 0}, {0, 0}}, io_child_count =3D 1, io_phys_children =3D 0, io_parent_count =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff801b82bc000, io_waiter =3D 0x0, io_bio =3D 0x0, io_lock =3D {lock_object =3D {lo_name =3D 0xffffffff82ca3c8d <.L.str .112 = + 1 > "zio->io_lock", lo_flags =3D 577830912, lo_data =3D 0, lo_witnes= s =3D 0x0}, sx_lock =3D 1}, io_cv =3D {cv_description =3D 0xffffffff82caea62 <.L.str .113 + 1 > "zio->= io_cv", cv_waiters =3D 0}, io_allocator =3D 0, io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D {tqent_task =3D {ta_link =3D {stqe_next =3D 0x0}, ta_pending = =3D 0, ta_priority =3D 0 '\000', ta_flags =3D 0 '\000', ta_func =3D 0xffffffff82a2bd90 < taskq_run_ent >, ta_context =3D 0xfffff80767ed2db0}, tqent_timeout_task =3D {q =3D 0x0, t =3D {ta_link =3D {stqe_next =3D 0x0}, ta_pending =3D 0, ta_priority =3D 0 '\000', ta_flags =3D 0 '\000', ta_func =3D 0x0, ta_context =3D 0x0}, c =3D {c_links =3D {le =3D {le_next =3D 0x0, le_prev =3D = 0x0}, sle =3D {sle_next =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_prev =3D 0= x0}}, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0, c_fu= nc =3D 0x0, c_lock =3D 0x0, c_flags =3D 0, c_iflags =3D 0, c_cpu= =3D 0}, f =3D 0}, tqent_func =3D 0xffffffff82c1fee0 < zio_execute >, tqent_arg =3D 0xfffff80767ed29a0, tqent_id =3D 0, tqent_hash =3D {cle_next =3D 0x0, cle_prev =3D 0x0}, tqent_type =3D 0 '\000', tqent_registered =3D 0 '\000', tqent_cancelled =3D 0 '\000', tqent_rc =3D 0}}; Here are the volume properties: NAME PROPERTY VALUE SOURCE fat32 type volume - fat32 creation Thu Jan 18 5:36 2024 - fat32 used 100G - fat32 available 734G - fat32 referenced 74.2G - fat32 compressratio 1.02x - fat32 reservation none default fat32 volsize 279G local fat32 volblocksize 16K - fat32 checksum on default fat32 compression on inherited from private fat32 readonly off default fat32 createtxg 23707850 - fat32 copies 1 default fat32 refreservation 100G local fat32 guid 3356515442365534536 - fat32 primarycache all default fat32 secondarycache all default fat32 usedbysnapshots 0B - fat32 usedbydataset 74.2G - fat32 usedbychildren 0B - fat32 usedbyrefreservation 25.8G - fat32 logbias latency default fat32 objsetid 2437 - fat32 dedup off default fat32 mlslabel none default fat32 sync standard default fat32 refcompressratio 1.02x - fat32 written 74.2G - fat32 logicalused 57.0G - fat32 logicalreferenced 57.0G - fat32 volmode default default fat32 snapshot_limit none default fat32 snapshot_count none default fat32 snapdev hidden default fat32 context none default fat32 fscontext none default fat32 defcontext none default fat32 rootcontext none default fat32 redundant_metadata all default fat32 encryption off default fat32 keylocation none default fat32 keyformat none default fat32 pbkdf2iters 0 default Here are the pool features: private feature@async_destroy enabled local private feature@empty_bpobj active local private feature@lz4_compress active local private feature@multi_vdev_crash_dump enabled local private feature@spacemap_histogram active local private feature@enabled_txg active local private feature@hole_birth active local private feature@extensible_dataset active local private feature@embedded_data active local private feature@bookmarks enabled local private feature@filesystem_limits enabled local private feature@large_blocks active local private feature@large_dnode enabled local private feature@sha512 enabled local private feature@skein enabled local private feature@userobj_accounting active local private feature@encryption enabled local private feature@project_quota active local private feature@device_removal enabled local private feature@obsolete_counts enabled local private feature@zpool_checkpoint enabled local private feature@spacemap_v2 active local private feature@allocation_classes enabled local private feature@resilver_defer enabled local private feature@bookmark_v2 enabled local private feature@redaction_bookmarks enabled local private feature@redacted_datasets enabled local private feature@bookmark_written enabled local private feature@log_spacemap active local private feature@livelist enabled local private feature@device_rebuild enabled local private feature@zstd_compress enabled local private feature@draid enabled local --=20 You are receiving this mail because: You are the assignee for the bug.=