From nobody Tue Apr 11 14:28:31 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pwp9c5hQ9z44xY5 for ; Tue, 11 Apr 2023 14:28:36 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pwp9b0m2hz3N4C; Tue, 11 Apr 2023 14:28:35 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.33) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4004a.ext.cloudfilter.net ([10.228.9.227]) by cmsmtp with ESMTP id luF4p4tRhjvm1mEz4p3Kh7; Tue, 11 Apr 2023 14:28:34 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id mEz2pKvoi3fOSmEz3pZZMp; Tue, 11 Apr 2023 14:28:34 +0000 X-Authority-Analysis: v=2.4 cv=J8G5USrS c=1 sm=1 tr=0 ts=64356e92 a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=dKHAf1wccvYA:10 a=13WrDtVnAAAA:8 a=YxBL1-UpAAAA:8 a=pGLkceISAAAA:8 a=NEAV23lmAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=LXhWB4x9cL4UNslYIKQA:9 a=CjuIK1q_8ugA:10 a=qcMfyop8IQhGkljw9-nY:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 02CFF217C; Tue, 11 Apr 2023 07:28:32 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id DB8245FA; Tue, 11 Apr 2023 07:28:31 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= cc: Cy Schubert , FreeBSD User , Mateusz Guzik , Pawel Jakub Dawidek , FreeBSD CURRENT Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) In-reply-to: <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> Comments: In-reply-to =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= message dated "Tue, 11 Apr 2023 18:04:10 +0900." List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 11 Apr 2023 07:28:31 -0700 Message-Id: <20230411142831.DB8245FA@slippy.cwsent.com> X-CMAE-Envelope: MS4xfH1AfEeyYGCk9zM3ol1N4eoabHaj0sIAhvf/PewxWBKwPBjisyK8a4j3lFA/58qh6dZ4PfBH4EuVrIPJkjRXzzjWm/KSv/nYdmIWwL69fhztLDpHDHAQ oLlEBfQ5bmO41E+Mz4iaSy9ScvhouXvPcE1hbWkYAtJ5+NyjRbbpF/UOvnlDmXwsk7uXWOmfbK2Q25pN1jKbrKeOWIhzCcEZflqNiyec8GPxQ+87sZHxbI+z we47d3shK+hUdFIuvoExcoOhZq/7qPM9c3ZH9BLbIn58Lu0j6bWxIM7dM6pNXaNlA3kSPFYw38tIywXVLsZH5K6viLgw6zHNkOLQoRYGoEE= X-Spamd-Result: default: False [2.36 / 15.00]; R_BAD_CTE_7BIT(3.50)[7bit]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MV_CASE(0.50)[]; NEURAL_HAM_SHORT(-0.34)[-0.339]; RCVD_IN_DNSWL_MED(-0.20)[3.97.99.33:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; R_SPF_NA(0.00)[no SPF record]; FROM_HAS_DN(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[70.66.148.124:server fail]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_CC(0.00)[cschubert.com,walstatt-de.de,gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; GREYLIST(0.00)[pass,meta] X-Rspamd-Queue-Id: 4Pwp9b0m2hz3N4C X-Spamd-Bar: ++ X-ThisMailContainsUnwantedMimeParts: N In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, =?utf-8?Q?Pawe=C 5=82_Jakub_Dawidek?= writes: > > > > On Apr 11, 2023, at 11:31, Cy Schubert wrote: > >=20 > > =EF=BB=BFIn message <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.d= > e>,=20 > > FreeBSD Us > > er writes: > >> Am Sun, 9 Apr 2023 14:37:03 +0200 > >> Mateusz Guzik schrieb: > >>=20 > >>>> On 4/9/23, FreeBSD User wrote: > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e= > 301 > >>> 2b: > >>>>> Sun Apr 9 > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > >>>>>=20 > >>>>> zpool upgrade POOLNAME > >>>>>=20 > >>>>> some boxes keep crashing when starting compiler runs (the trigger is > >>>>> different on boxes). > >>>>>=20 > >>>>> ZFS module is statically compiled into the kernel (if this is of > >>>>> importance) > >>>>>=20 > >>>>> Last known good was: > >>>>>=20 > >>>>> [...] > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > >>>>> main-n262051-75379ea2e461: Sun Apr > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 07:10:04 < > = > 0. > >>> 2> > >>>>> thor kernel: > >>>>> FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git= > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor kernel: > >>>>> VT(efifb): resolution > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl already > >>>>> present! > >>>>> [...] > >>>>>=20 > >>>>> The file /var/crash/info.X > >>>>>=20 > >>>>> contains: > >>>>>=20 > >>>>> [...] > >>>>>=20 > >>>>> root@thor:/var/crash # more info.2 > >>>>> Dump header from device: /dev/gpt/swap > >>>>> Architecture: amd64 > >>>>> Architecture Version: 2 > >>>>> Dump Length: 1095192576 > >>>>> Blocksize: 512 > >>>>> Compression: none > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 > >>>>> Hostname: thor.local > >>>>> Magic: FreeBSD Kernel Dump > >>>>> Version String: FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e3012b: Su= > n=20 > >>> Apr > >>>>> 9 12:01:02 CEST > >>>>> 2023 > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > >>>>>=20 > >>>>> Dump Parity: 2961465682 > >>>>> Bounds: 2 > >>>>> Dump Status: good > >>>>>=20 > >>>>> Until reconfigured for more debug stuff I do not have more to present.= > > >>>>>=20 > >>>>> I rememeber now really scraed that there was a HEADSUP in the list reg= > ard > >>> ing > >>>>> some serious ZFS > >>>>> problems - I didn't find it right now. > >>>>>=20 > >>>>> Thanks in advance, > >>>>>=20 > >>>=20 > >>> That's fallout from the new block cloning feature, adding the author > >>>=20 > >>=20 > >> Thanks. > >>=20 > >> As of this moment, all systems with the newest kernel and the new ZFS opt= > ion=20 > >> enabled, crash - > >> the reason is mostly in different ZFS datasets. I guess there is no way b > = > ack > >> once this faulty > >> option is enabled? > >=20 > > I've run a test on a scratch pool here, first without block_cloning=20 > > enabled, then with. There was no corruption when block_cloning was=20 > > disabled. There was corruption when block_cloning was enabled. > >=20 > > I don't know of any way to revert back nor is there any way to fix or=20 > > recover the corrupted blocks. > > Is the corruption still present after EXDEV fixes? Yes and no. Yes, there is corruption when block_cloning is enabled. There is no corruption when block_cloning is disabled. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0