From nobody Tue Apr 11 02:19:18 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PwVGT3zS7z457xn for ; Tue, 11 Apr 2023 02:31:45 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PwVGS2gCnz3qf0; Tue, 11 Apr 2023 02:31:44 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.33) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4003a.ext.cloudfilter.net ([10.228.9.183]) by cmsmtp with ESMTP id lrzJp4h0Tjvm1m3bNp2KiG; Tue, 11 Apr 2023 02:19:21 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id m3bLpq9iqcyvum3bMpzUSz; Tue, 11 Apr 2023 02:19:21 +0000 X-Authority-Analysis: v=2.4 cv=VbHkgXl9 c=1 sm=1 tr=0 ts=6434c3a9 a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=dKHAf1wccvYA:10 a=pGLkceISAAAA:8 a=NEAV23lmAAAA:8 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=BFMMW6rN7m6Y0xqU6JoA:9 a=CjuIK1q_8ugA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 1A9B0E20; Mon, 10 Apr 2023 19:19:19 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 0718F306; Mon, 10 Apr 2023 19:19:18 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: FreeBSD User cc: Mateusz Guzik , Pawel Jakub Dawidek , FreeBSD CURRENT Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) In-reply-to: <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.de> References: <20230409140023.47aab826@thor.intern.walstatt.dynvpn.de> <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.de> Comments: In-reply-to FreeBSD User message dated "Sun, 09 Apr 2023 16:14:09 +0200." List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 10 Apr 2023 19:19:18 -0700 Message-Id: <20230411021919.0718F306@slippy.cwsent.com> X-CMAE-Envelope: MS4xfK+9V2L7/gwseeFLWiJNNCvEOtYVhrFqMD7AHhIXaIT0U4kQeVyiOh0OzYVGMWrmz6GWxbtBOhMuW8BPbESwE+W5TRlN+N+w1pdFEoQ3PqpDLyCf8HBe PLiXZxQ3aJySFbMa9feExDTOSifVcvOCB+3EiFgwPVY0t3iW+l3RUMCRSm9ZxqW7sX+BOBq7M6C0y1zS1BqFVvCqPukhxu0JEcecrkNz70btP3V1K9O3GMfJ n1vV7Ho7k/0y/3qnuY8aA2anKac836dB5sBA8WTiSVpH7G/gBvzrZ1varkuOnPl8 X-Spamd-Result: default: False [1.84 / 15.00]; R_BAD_CTE_7BIT(3.50)[7bit]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.97)[-0.965]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[3.97.99.33:from]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_NA(0.00)[no SPF record]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; REPLYTO_EQ_FROM(0.00)[]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; SUBJECT_HAS_EXCLAIM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4PwVGS2gCnz3qf0 X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N In message <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.de>, FreeBSD Us er writes: > Am Sun, 9 Apr 2023 14:37:03 +0200 > Mateusz Guzik schrieb: > > > On 4/9/23, FreeBSD User wrote: > > > Today, after upgrading to FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e301 > 2b: > > > Sun Apr 9 > > > 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > > > > > > zpool upgrade POOLNAME > > > > > > some boxes keep crashing when starting compiler runs (the trigger is > > > different on boxes). > > > > > > ZFS module is statically compiled into the kernel (if this is of > > > importance) > > > > > > Last known good was: > > > > > > [...] > > > Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > > > main-n262051-75379ea2e461: Sun Apr > > > 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > > > root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 07:10:04 <0. > 2> > > > thor kernel: > > > FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git > > > llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor kernel: > > > VT(efifb): resolution > > > 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl already > > > present! > > > [...] > > > > > > The file /var/crash/info.X > > > > > > contains: > > > > > > [...] > > > > > > root@thor:/var/crash # more info.2 > > > Dump header from device: /dev/gpt/swap > > > Architecture: amd64 > > > Architecture Version: 2 > > > Dump Length: 1095192576 > > > Blocksize: 512 > > > Compression: none > > > Dumptime: 2023-04-09 11:43:41 +0000 > > > Hostname: thor.local > > > Magic: FreeBSD Kernel Dump > > > Version String: FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e3012b: Sun > Apr > > > 9 12:01:02 CEST > > > 2023 > > > root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > > > Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > > > > > Dump Parity: 2961465682 > > > Bounds: 2 > > > Dump Status: good > > > > > > Until reconfigured for more debug stuff I do not have more to present. > > > > > > I rememeber now really scraed that there was a HEADSUP in the list regard > ing > > > some serious ZFS > > > problems - I didn't find it right now. > > > > > > Thanks in advance, > > > > > > > That's fallout from the new block cloning feature, adding the author > > > > Thanks. > > As of this moment, all systems with the newest kernel and the new ZFS option > enabled, crash - > the reason is mostly in different ZFS datasets. I guess there is no way back > once this faulty > option is enabled? I've run a test on a scratch pool here, first without block_cloning enabled, then with. There was no corruption when block_cloning was disabled. There was corruption when block_cloning was enabled. I don't know of any way to revert back nor is there any way to fix or recover the corrupted blocks. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0