From nobody Wed Apr 12 12:57:02 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PxN5l3K2vz44k5v for ; Wed, 12 Apr 2023 12:57:15 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PxN5k72dqz3vfd; Wed, 12 Apr 2023 12:57:14 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681304235; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=fQRpNhVEKa5FXReYdtD1lpsBCTNLagLpqMHCrmJKMhs=; b=hbxCzT3u8seEboHvcwHsDpKMJyZ/CUbRhelstFC8OCHw+lGArYdjAiClHQF23IBuFiIVjp bTMi4PuK7yfviJekqxrktdO4qdE4pp8GQ+xmIJTfmdlcLp/rqaCTMX3bSWbxGoJaKr5fSR 9pLl1f4/mi7GAIX8vdCfPjwvEKLzlPPqmK2GFiDvTsA4psQliV2opRaPhxBldIg6EoJd1F aMC3K3JRTnV+Mcpgvw8/7QJIgiHMSyzUHtXHzsIbuJ6pPgL19HTAWpPELoAVoPPThEpS5f rCjX+ze0h/lTVGscyUn+D0+P/WEHMdBxFfHNh8U6WTMr3trOuscWv2HN0Mezuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681304235; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=fQRpNhVEKa5FXReYdtD1lpsBCTNLagLpqMHCrmJKMhs=; b=qW//CqG4U/lCyMsChYOwJ+X7e19ldrE5orraBysWZVfY3kTUtwLclUrA+LlEHPFPOYyDta kS93kYwP3MttBN03o4Mig6+rwFEWY0hbo88C7nfh5NfYxRyTXyzMjCatzxXwA2NB9C0PT/ 0BtrFTa02e6GIq2EhF2OeGUlC5VXmKKaPfOm6hyYo0+x9G40IoC7cq3QdLPvJYjUzR+B/P lAXG9N+ohKhiNVqeg3xw+vSWhv4+AT07fw5RT7dELS8UBmvTioAu0euYY6dpDW4t2HDqF7 mmVngu0m8r6nfGuF+qvXSeE6B4AYskUtGPe0wzStZANrP38L2ntkWAE/AMjCFg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1681304235; a=rsa-sha256; cv=none; b=BUD9iwXw5v2ma+hq/zU3cs+w+kijIgXF7tvQ7avQs+nBB6RcrA5r/s9z4rXrSIu0KGlzOJ cQ+n6oMrs0URZFAeTHWj+2XOm57ozrCLVr6+gzLEDgC9NpdXmHmboiDGiAE09H7lsRpDCZ +Bh6Vl/wNVK8oO68dalaOKC5zRYGVn2WPahSlvcISZXv7QBcuPKE+QemiHsrL7OF8nOA4b 6GNDI4tO3KaGUIEAZ5OEUNxkiRCM/kghqrTG9GMMUQYsgsWvgV5ehiObWPC1XwK3TtXvPV zfc1WDyNEpV7/cLDDcZl8Uswws5loM69fla56si08SSquyFGLWrgEpg3bVMWTA== Received: from mail-ua1-f50.google.com (mail-ua1-f50.google.com [209.85.222.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PxN5k65BCz1R9d; Wed, 12 Apr 2023 12:57:14 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-ua1-f50.google.com with SMTP id x26so772708uav.3; Wed, 12 Apr 2023 05:57:14 -0700 (PDT) X-Gm-Message-State: AAQBX9cGT4Fxzt5nQ1VZNoinoKR1P/pNUJlPirclTG76HgiZxUjopkCr rpqcaSho9mL06GmUIyGQOYgf9PSbDOUH9TZFbgk= X-Google-Smtp-Source: AKy350YTIXNzTN+x/JVmEk7oNqKzhy8Uids/EHl3NLHTjseMsvZTISrhtEMtxkCakBfc8SpqeGxAp7yZjwY6Oc7gJLw= X-Received: by 2002:ab0:5549:0:b0:68a:7224:2034 with SMTP id u9-20020ab05549000000b0068a72242034mr10478942uaa.0.1681304234348; Wed, 12 Apr 2023 05:57:14 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com> In-Reply-To: <20230411144713.A94EA5FE@slippy.cwsent.com> From: Nuno Teixeira Date: Wed, 12 Apr 2023 13:57:02 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) To: Cy Schubert Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= , FreeBSD User , Mateusz Guzik , Pawel Jakub Dawidek , FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000a5d6f005f92325c6" X-ThisMailContainsUnwantedMimeParts: N --000000000000a5d6f005f92325c6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello all, at current 3fdb40d1befe after `zfs upgrade XXX`: same problem when running compiler: - poudriere: crash without dump - make buildworld (/usr/src): shutdown -p (I will try to get a photo) Is there a way to disable block clone? Cy Schubert escreveu no dia ter=C3=A7a, 11/04/2= 023 =C3=A0(s) 15:47: > In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert > writes: > > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, > > =3D?utf-8?Q?Pawe=3DC > > 5=3D82_Jakub_Dawidek?=3D writes: > > > > > > > > > > On Apr 11, 2023, at 11:31, Cy Schubert > wrote: > > > >=3D20 > > > > =3DEF=3DBB=3DBFIn message > <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. > > d=3D > > > e>,=3D20 > > > > FreeBSD Us > > > > er writes: > > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 > > > >> Mateusz Guzik schrieb: > > > >>=3D20 > > > >>>> On 4/9/23, FreeBSD User wrote: > > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038 > > e=3D > > > 301 > > > >>> 2b: > > > >>>>> Sun Apr 9 > > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > > > >>>>>=3D20 > > > >>>>> zpool upgrade POOLNAME > > > >>>>>=3D20 > > > >>>>> some boxes keep crashing when starting compiler runs (the > trigger is > > > >>>>> different on boxes). > > > >>>>>=3D20 > > > >>>>> ZFS module is statically compiled into the kernel (if this is o= f > > > >>>>> importance) > > > >>>>>=3D20 > > > >>>>> Last known good was: > > > >>>>>=3D20 > > > >>>>> [...] > > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > > > >>>>> main-n262051-75379ea2e461: Sun Apr > > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 > 07:10:04 > > < > > > =3D > > > 0. > > > >>> 2> > > > >>>>> thor kernel: > > > >>>>> FreeBSD clang version 15.0.7 ( > https://github.com/llvm/llvm-project.gi > > t=3D > > > > > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor > kernel: > > > >>>>> VT(efifb): resolution > > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl > already > > > >>>>> present! > > > >>>>> [...] > > > >>>>>=3D20 > > > >>>>> The file /var/crash/info.X > > > >>>>>=3D20 > > > >>>>> contains: > > > >>>>>=3D20 > > > >>>>> [...] > > > >>>>>=3D20 > > > >>>>> root@thor:/var/crash # more info.2 > > > >>>>> Dump header from device: /dev/gpt/swap > > > >>>>> Architecture: amd64 > > > >>>>> Architecture Version: 2 > > > >>>>> Dump Length: 1095192576 > > > >>>>> Blocksize: 512 > > > >>>>> Compression: none > > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 > > > >>>>> Hostname: thor.local > > > >>>>> Magic: FreeBSD Kernel Dump > > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038e3012b: S > > u=3D > > > n=3D20 > > > >>> Apr > > > >>>>> 9 12:01:02 CEST > > > >>>>> 2023 > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > > >>>>>=3D20 > > > >>>>> Dump Parity: 2961465682 > > > >>>>> Bounds: 2 > > > >>>>> Dump Status: good > > > >>>>>=3D20 > > > >>>>> Until reconfigured for more debug stuff I do not have more to > present > > .=3D > > > > > > >>>>>=3D20 > > > >>>>> I rememeber now really scraed that there was a HEADSUP in the > list re > > g=3D > > > ard > > > >>> ing > > > >>>>> some serious ZFS > > > >>>>> problems - I didn't find it right now. > > > >>>>>=3D20 > > > >>>>> Thanks in advance, > > > >>>>>=3D20 > > > >>>=3D20 > > > >>> That's fallout from the new block cloning feature, adding the > author > > > >>>=3D20 > > > >>=3D20 > > > >> Thanks. > > > >>=3D20 > > > >> As of this moment, all systems with the newest kernel and the new > ZFS op > > t=3D > > > ion=3D20 > > > >> enabled, crash - > > > >> the reason is mostly in different ZFS datasets. I guess there is > no way > > b > > > =3D > > > ack > > > >> once this faulty > > > >> option is enabled? > > > >=3D20 > > > > I've run a test on a scratch pool here, first without > block_cloning=3D20 > > > > enabled, then with. There was no corruption when block_cloning was= =3D20 > > > > disabled. There was corruption when block_cloning was enabled. > > > >=3D20 > > > > I don't know of any way to revert back nor is there any way to fix > or=3D20 > > > > recover the corrupted blocks. > > > > > > Is the corruption still present after EXDEV fixes? > > > > Yes and no. > > > > Yes, there is corruption when block_cloning is enabled. > > > > There is no corruption when block_cloning is disabled. > > I should add some detail to this. > > The corruption experienced when block cloning is disabled was fixed by: > > - eb1feadc201a > - e2d997d1cbb9 > - d012836fb616 (specifically this commit) > - 20be1b4fc4b7 > > When block_cloning is enabled, the pool is corrupted. This has not been > fixed. > > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: https://FreeBSD.org > NTP: Web: https://nwtime.org > > e^(i*pi)+1=3D0 > > > > --=20 Nuno Teixeira FreeBSD Committer (ports) --000000000000a5d6f005f92325c6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello all,

at current 3fdb40= d1befe after `zfs upgrade XXX`:

same problem w= hen running compiler:

- poudriere: crash without d= ump
- make buildworld (/usr/src): shutdown -p (I will try to get = a photo)

Is there a way to disable block clone?

Cy Schubert <Cy.Schu= bert@cschubert.com> escreveu no dia ter=C3=A7a, 11/04/2023 =C3=A0(s)= 15:47:
In messa= ge <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert = writes:
> In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek= .net>,
> =3D?utf-8?Q?Pawe=3DC
> 5=3D82_Jakub_Dawidek?=3D writes:
> >
> >
> > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com= > wrote:
> > >=3D20
> > > =3DEF=3DBB=3DBFIn message <20230409161436.5412fa6e@thor.i= ntern.walstatt.dynvpn.
> d=3D
> > e>,=3D20
> > > FreeBSD Us
> > > er writes:
> > >> Am Sun, 9 Apr 2023 14:37:03 +0200
> > >> Mateusz Guzik <mjguzik@gmail.com> schrieb:
> > >>=3D20
> > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrot= e:
> > >>>>> Today, after upgrading to FreeBSD 14.0-CURRE= NT #8 main-n262052-0d4038
> e=3D
> > 301
> > >>> 2b:
> > >>>>> Sun Apr=C2=A0 9
> > >>>>> 12:01:02 CEST 2023=C2=A0 amd64, AND upgradin= g ZPOOLs via
> > >>>>>=3D20
> > >>>>> zpool upgrade POOLNAME
> > >>>>>=3D20
> > >>>>> some boxes keep crashing when starting compi= ler runs (the trigger is
> > >>>>> different on boxes).
> > >>>>>=3D20
> > >>>>> ZFS module is statically compiled into the k= ernel (if this is of
> > >>>>> importance)
> > >>>>>=3D20
> > >>>>> Last known good was:
> > >>>>>=3D20
> > >>>>> [...]
> > >>>>> Apr=C2=A0 9 07:10:04 <0.2> thor kernel= : FreeBSD 14.0-CURRENT #7
> > >>>>> main-n262051-75379ea2e461: Sun Apr
> > >>>>> 9 00:12:57 CEST 2023 Apr=C2=A0 9 07:10:04 &l= t;0.2> thor kernel:
> > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/T= HOR amd64 Apr=C2=A0 9 07:10:04
>=C2=A0 <
> > =3D
> > 0.
> > >>> 2>
> > >>>>> thor kernel:
> > >>>>> FreeBSD clang version 15.0.7 (= https://github.com/llvm/llvm-project.gi
> t=3D
> >
> > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr=C2=A0 9 = 07:10:04 <0.2> thor kernel:
> > >>>>> VT(efifb): resolution
> > >>>>> 2560x1440 Apr=C2=A0 9 07:10:04 <0.2> t= hor kernel: module zfsctrl already
> > >>>>> present!
> > >>>>> [...]
> > >>>>>=3D20
> > >>>>> The file /var/crash/info.X
> > >>>>>=3D20
> > >>>>> contains:
> > >>>>>=3D20
> > >>>>> [...]
> > >>>>>=3D20
> > >>>>> root@thor:/var/crash # more info.2
> > >>>>> Dump header from device: /dev/gpt/swap
> > >>>>>=C2=A0 Architecture: amd64
> > >>>>>=C2=A0 Architecture Version: 2
> > >>>>>=C2=A0 Dump Length: 1095192576
> > >>>>>=C2=A0 Blocksize: 512
> > >>>>>=C2=A0 Compression: none
> > >>>>>=C2=A0 Dumptime: 2023-04-09 11:43:41 +0000 > > >>>>>=C2=A0 Hostname: thor.local
> > >>>>>=C2=A0 Magic: FreeBSD Kernel Dump
> > >>>>>=C2=A0 Version String: FreeBSD 14.0-CURRENT #= 8 main-n262052-0d4038e3012b: S
> u=3D
> > n=3D20
> > >>> Apr
> > >>>>> 9 12:01:02 CEST
> > >>>>> 2023
> > >>>>>=C2=A0 =C2=A0 root@thor:/usr/obj/usr/src/amd6= 4.amd64/sys/THOR
> > >>>>>=C2=A0 Panic String: VERIFY(!zil_replaying(zi= log, tx)) failed
> > >>>>>=3D20
> > >>>>>=C2=A0 Dump Parity: 2961465682
> > >>>>>=C2=A0 Bounds: 2
> > >>>>>=C2=A0 Dump Status: good
> > >>>>>=3D20
> > >>>>> Until reconfigured for more debug stuff I do= not have more to present
> .=3D
> >
> > >>>>>=3D20
> > >>>>> I rememeber now really scraed that there was= a HEADSUP in the list re
> g=3D
> > ard
> > >>> ing
> > >>>>> some serious ZFS
> > >>>>> problems - I didn't find it right now. > > >>>>>=3D20
> > >>>>> Thanks in advance,
> > >>>>>=3D20
> > >>>=3D20
> > >>> That's fallout from the new block cloning featur= e, adding the author
> > >>>=3D20
> > >>=3D20
> > >> Thanks.
> > >>=3D20
> > >> As of this moment, all systems with the newest kernel an= d the new ZFS op
> t=3D
> > ion=3D20
> > >> enabled, crash -
> > >> the reason is mostly in=C2=A0 different ZFS datasets. I = guess there is no way
>=C2=A0 b
> > =3D
> > ack
> > >> once this faulty
> > >> option is enabled?
> > >=3D20
> > > I've run a test on a scratch pool here, first without bl= ock_cloning=3D20
> > > enabled, then with. There was no corruption when block_cloni= ng was=3D20
> > > disabled. There was corruption when block_cloning was enable= d.
> > >=3D20
> > > I don't know of any way to revert back nor is there any = way to fix or=3D20
> > > recover the corrupted blocks.
> >
> > Is the corruption still present after EXDEV fixes?
>
> Yes and no.
>
> Yes, there is corruption when block_cloning is enabled.
>
> There is no corruption when block_cloning is disabled.

I should add some detail to this.

The corruption experienced when block cloning is disabled was fixed by:

- eb1feadc201a
- e2d997d1cbb9
- d012836fb616 (specifically this commit)
- 20be1b4fc4b7

When block_cloning is enabled, the pool is corrupted. This has not been fixed.


--
Cheers,
Cy Schubert <Cy.Schubert@cschubert.com>
FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://FreeB= SD.org
NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<cy@nwtime.org>=C2=A0 =C2=A0 Web:=C2=A0 https://nwt= ime.org

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0





--
Nuno Teixeira
FreeBSD Committ= er (ports)
--000000000000a5d6f005f92325c6--