From nobody Tue Apr 11 14:47:13 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pwpb92ggdz44yT4 for ; Tue, 11 Apr 2023 14:47:17 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pwpb82BpRz4DJM; Tue, 11 Apr 2023 14:47:16 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.32) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4004a.ext.cloudfilter.net ([10.228.9.227]) by cmsmtp with ESMTP id mEGfp0dQruZMSmFH9pHdSv; Tue, 11 Apr 2023 14:47:15 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id mFH8pL1Xm3fOSmFH8pZc38; Tue, 11 Apr 2023 14:47:15 +0000 X-Authority-Analysis: v=2.4 cv=J8G5USrS c=1 sm=1 tr=0 ts=643572f3 a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=dKHAf1wccvYA:10 a=VxmjJ2MpAAAA:8 a=13WrDtVnAAAA:8 a=YxBL1-UpAAAA:8 a=pGLkceISAAAA:8 a=NEAV23lmAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=CK1mnscjJpQ-eMPXql0A:9 a=CjuIK1q_8ugA:10 a=u_YB59FJ7C0A:10 a=7gXAzLPJhVmCkEl4_tsf:22 a=qcMfyop8IQhGkljw9-nY:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id BCE602221; Tue, 11 Apr 2023 07:47:13 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id A94EA5FE; Tue, 11 Apr 2023 07:47:13 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= cc: FreeBSD User , Mateusz Guzik , Pawel Jakub Dawidek , FreeBSD CURRENT Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) In-reply-to: <20230411142831.DB8245FA@slippy.cwsent.com> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> Comments: In-reply-to Cy Schubert message dated "Tue, 11 Apr 2023 07:28:31 -0700." List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 11 Apr 2023 07:47:13 -0700 Message-Id: <20230411144713.A94EA5FE@slippy.cwsent.com> X-CMAE-Envelope: MS4xfDR6UyYzPqzCrOv6cRcUfOXbjwEX3WB3EEkpQyvyBbcBpB6TANgLkB+eXS2DGP14yQik0+vYnrTlKhAdT9gaaTkf7uIpW47105AsSDc68ilp2uNR/y2I T45Pb1L6/jbHIaDexkQxLPJOPi35C0tbePrH1uguePMc7VBG8BSYDro5bMXvWD6iQO0Gd5jlZShgBWMC7RA38pJFcnexzlB5z+unARnWEZ5UVuxPZD9PwSTM V0e00sUQsNWIK3wAqJRXjZiYUMF63BGCpjWrxP0m42FZxn99Y6WbE/dOxHCz6TM2bsHyeX1B9j7UGZWvyvG5DcIOS6qagtWWFhlz2VfE0+A= X-Spamd-Result: default: False [2.11 / 15.00]; R_BAD_CTE_7BIT(3.50)[7bit]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.69)[-0.689]; MV_CASE(0.50)[]; RCVD_IN_DNSWL_LOW(-0.10)[3.97.99.32:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[3.97.99.32:from]; REPLYTO_EQ_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_CC(0.00)[walstatt-de.de,gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; GREYLIST(0.00)[pass,meta] X-Rspamd-Queue-Id: 4Pwpb82BpRz4DJM X-Spamd-Bar: ++ X-ThisMailContainsUnwantedMimeParts: N In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert writes: > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, > =?utf-8?Q?Pawe=C > 5=82_Jakub_Dawidek?= writes: > > > > > > > On Apr 11, 2023, at 11:31, Cy Schubert wrote: > > >=20 > > > =EF=BB=BFIn message <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. > d= > > e>,=20 > > > FreeBSD Us > > > er writes: > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 > > >> Mateusz Guzik schrieb: > > >>=20 > > >>>> On 4/9/23, FreeBSD User wrote: > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 main-n262052-0d4038 > e= > > 301 > > >>> 2b: > > >>>>> Sun Apr 9 > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > > >>>>>=20 > > >>>>> zpool upgrade POOLNAME > > >>>>>=20 > > >>>>> some boxes keep crashing when starting compiler runs (the trigger is > > >>>>> different on boxes). > > >>>>>=20 > > >>>>> ZFS module is statically compiled into the kernel (if this is of > > >>>>> importance) > > >>>>>=20 > > >>>>> Last known good was: > > >>>>>=20 > > >>>>> [...] > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > > >>>>> main-n262051-75379ea2e461: Sun Apr > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 07:10:04 > < > > = > > 0. > > >>> 2> > > >>>>> thor kernel: > > >>>>> FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.gi > t= > > > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor kernel: > > >>>>> VT(efifb): resolution > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl already > > >>>>> present! > > >>>>> [...] > > >>>>>=20 > > >>>>> The file /var/crash/info.X > > >>>>>=20 > > >>>>> contains: > > >>>>>=20 > > >>>>> [...] > > >>>>>=20 > > >>>>> root@thor:/var/crash # more info.2 > > >>>>> Dump header from device: /dev/gpt/swap > > >>>>> Architecture: amd64 > > >>>>> Architecture Version: 2 > > >>>>> Dump Length: 1095192576 > > >>>>> Blocksize: 512 > > >>>>> Compression: none > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 > > >>>>> Hostname: thor.local > > >>>>> Magic: FreeBSD Kernel Dump > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e3012b: S > u= > > n=20 > > >>> Apr > > >>>>> 9 12:01:02 CEST > > >>>>> 2023 > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > >>>>>=20 > > >>>>> Dump Parity: 2961465682 > > >>>>> Bounds: 2 > > >>>>> Dump Status: good > > >>>>>=20 > > >>>>> Until reconfigured for more debug stuff I do not have more to present > .= > > > > >>>>>=20 > > >>>>> I rememeber now really scraed that there was a HEADSUP in the list re > g= > > ard > > >>> ing > > >>>>> some serious ZFS > > >>>>> problems - I didn't find it right now. > > >>>>>=20 > > >>>>> Thanks in advance, > > >>>>>=20 > > >>>=20 > > >>> That's fallout from the new block cloning feature, adding the author > > >>>=20 > > >>=20 > > >> Thanks. > > >>=20 > > >> As of this moment, all systems with the newest kernel and the new ZFS op > t= > > ion=20 > > >> enabled, crash - > > >> the reason is mostly in different ZFS datasets. I guess there is no way > b > > = > > ack > > >> once this faulty > > >> option is enabled? > > >=20 > > > I've run a test on a scratch pool here, first without block_cloning=20 > > > enabled, then with. There was no corruption when block_cloning was=20 > > > disabled. There was corruption when block_cloning was enabled. > > >=20 > > > I don't know of any way to revert back nor is there any way to fix or=20 > > > recover the corrupted blocks. > > > > Is the corruption still present after EXDEV fixes? > > Yes and no. > > Yes, there is corruption when block_cloning is enabled. > > There is no corruption when block_cloning is disabled. I should add some detail to this. The corruption experienced when block cloning is disabled was fixed by: - eb1feadc201a - e2d997d1cbb9 - d012836fb616 (specifically this commit) - 20be1b4fc4b7 When block_cloning is enabled, the pool is corrupted. This has not been fixed. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0