From nobody Tue Apr 18 19:37:30 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q1Dj864zqz45ZBD for ; Tue, 18 Apr 2023 19:37:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q1Dj71kRQz3Jg0 for ; Tue, 18 Apr 2023 19:37:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=NH4v4ZWf; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1681846665; bh=HJ7yIqeyX5QICTOi3zRCcXtQr2ktDEiMkP7LgjKK1Bo=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=NH4v4ZWfD3W0wCDiUwIow6vy8X4XZWpeUfC06c/Lq8x2+gnlfkQq17qleN/zTffhjJKRPB+TyPXjaTNrT4C/4T9v160DX/EGO9UuzBQN/IPdhhYVkeewNILBsD19Yj4+iY1vFhYUPwp1HMXY2CDKjHmktITdESRPDHrItPUIlYaBACtIuIrNYePeKxLQMaZ2QRJ0g5kIWA3XXL8t2A1YiR+gWIFK5FGk2cBuIlOQiDobRXRmlVna9zUjfG5lqhPznSkCzpet2fJfxuL53q3d0jEm1lM+3hbr/15HhG/RW114K5ZQHPKS0kn3HAa9tgjDAZ2RiI+NuJfu/0Nh/p0wkg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1681846665; bh=fwjDajeFbXwFi6nvBDj8RcY7c2DHi0hTdAbRa4RI5IH=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=L3o7HtmNd6oZHjrrVNNqkApC3Kr1GI2Y/wcDO6YCkns/yeP+IyxAPln+w0qLxca0wyYxfJhMplxWJffEVf0lUwxE+VI8O2IheTcLhBZzKgQJyrjvaVBWX48fFeFLJ7FM6pN9e+FRzGP3LoJRSU5C3ZhzfghAUdEkT0bm5XWim+O/vqgpEYnR0attTQylIwzxaSRyoGd39XRvw3NCtRyykQe/DURvVUZ9fArY2031cdV71VbJfuMIMf/FOx1rddsLkIUhMFkvMErkFiC1OqBDIC4/a/E/qqcCLD7gU6utT+jW/ICpbqiGvjJkqfdue0wC/cPVLTFpIEhkXxC00gmv9Q== X-YMail-OSG: Gw2U1bIVM1lZQ3A_cQFQQhVsgzH54d2WNdDTAs.UXE8EYGY9H1vvYeHWsvMUUHT 08Lgy9SQXKzlj7VOxfePHkC6pcluVXIEjvB8d6xSrk8.YC_7.Md3AtxaL3tjjcQPQjkHlsrUWG2D o3m0vk94Y7rq4FEuSn.hkd0EB04PDpS2U7c4RCgY1_FepezLKAyjwefBIUIPV3l1SvrhDDD7Dopp hQXUu_Cets._aFDrv5gDm6kFcxiRrBI94Hydm5dXzIUtbw2ld8IcjGt6CVqyo0jbqW2cVwPwXuAo cqoy8H182OvQqVnJF8_PAtl28gLauwyEatG4VOrDSMfFi5P0Ah2p7gBrqPqrfLe8Agw_x.NxxXxj agcg0Sq8JKRWbs_jVoJ6FC84jV6gO9ygkij7T5N_vYeoWkurDVFbE_6UXpbfw6KBRllXzF2yWiix oygaYoBD.sk_rEXnUWAJFhXBOqCGMxW9CiyqyC0LNH71ijuCVfUkj0XNYUBhWAx7xGunAuAUlQ6b K.qgxric4m1dziiTEK6LLn4uakAu_ATM8LjMlj_wXPvXrni6ZSBuMnA118Lw57G6p3D7lOExQy8m WjgV.eC5thDkBzrgGuK78mOCt1qdxYINcHxI7RVKoy7W_IaE0sxAhLN6OnpygfF96II0diDO3zPp 0X_49ihr5OLBRN0VEMjoUxr7vW1kSkxCDxvWweN2c_AxhxrB74YLgfLb_n_KbSr27w30JY_YTReC eXihKc3zsQAO9lsxWNJ2furkvCDoNxpZ6fCEW_FmO9p_dGBy7P37FOmdNumJt86CUY61jtNaNsgZ WbOVMYsXN8.85PRnzNoTfYWnzCx6cq2n.3ehYYvP7a9_M7p1GnsP0E926KjWz_H2i5AhEIqfMCUl 4n.Qwam6Lal98djxijS9If9Ww0VpHS6Y6uZFEhKHu7ZCH_Pd071P1XK8d4vXZOm8l.dWSxqIGeBj .xIZMOlipH3pqLWEwyf65minf17JyTsqh5ufGc.tFkojHxrvlWDrpGjfz2mrh_ktZqayw8LXAaDS paCF3nv9nuYN6oXSpOs895bmhEgvrMw44NT7W0GCRYEWMB6i8EtMJSzMJhy92Ei3ihc2Wt9rASNk tu3r5Qs6gyqAxXMxz7BNzSovoKdd9tPgr_Rem3ccGa0FMMzS._EqLJJp2W_Rp9scRGNIwgT7adkL B.NUbFJ84n2m1TjDoUibMMuitEBpSYL_.GK0DKtzJ748vq_A8ChCgCYRUiL9R2EoXJM2m4bbCFXy .XMJwyM2mTpUV7Y.tS5M2kR_6nnvXc5P_49kUpDTMS4PB0dN30aLaClykGlVvdFyMm0WvfUne0PG sQ4flWre5Morg64p_Qp5BxdWmGlahl_4166ngqatTIAyifVakXbbEjtCmHyQto.OQyYGLO9vxeO8 kUQBVhpc8DGqKxgAcSWnfDIIjX.LqkD.PK0qr.JIC.Yl1b5SZs920IeDIHtAVZF_Eib8xmP.boJS ofiMlHorImfI1ouOk0Tfhoa.y24xzNf3cYvoEJ9ZkB0AGEqE2UXTGmZLd6tSSdklXzSI174RnzeW fJDbaUWtJr7LaU4DtrMHuAou26AyMqhwvUvBbVEpIf16.21PiGK2zksy8HUqK1DFzHR4.VtJ3bNM FbEZwIScjCoNBZbKKoCX3kFmzUbA6kIrSenbHBSyYxm9PJlkMLfv1u3amxhW5mQRqHOFRFaCOsqs 7iBYeLaXNt0D0wOL9g5LrdJ0aT6CeTQIa3tE2jndOFHsMN7Q2EmyAZYAsPyE3nlRPKPJkK9xjo6b k_bb82Vd9mcbWLJZYxoM0LwrwNfbpTgeQ6PSTET2BykJJhZZ.ezfsL0iFfngLv_.3maZcIcOcoNL W.TVL0OcRzCoQOzxisQclhYEGOrPljwXcwuoq03z4mYRLmFEdxCwN.pGkKI06UVewHUsO6wMIt4_ amzfkSmnBjuYpV_DZdkjinNmEKPpnVX9axh1um3gIf764QWwfBMQyUXySpywddQu2pd7vKUfyt3P RyAg7QGy_cjNHP3eUTPeHjlSfQSq7QqfqOqqkz065fy44cSsTglKFvKLyHwolXr15HTLs6hCJTiW O.qSq1JNZY7HShDo1UnjRlbX7hSpqwa1mkWV6AWlF3doCIYaW3OzbumOg69Wtk8eMAkyKx64_qO1 XtwQ5j8nziCKb.QL7YmBkB0lZDJr0fgkfV5uScpP4mzLdQmuMYRLIuIlWzC_nHCIqaaZM9J9Wd5H e_v7guyGuBPDyheebRanTXqfJI1p.B.qnSLf3O8UueAND3mOJUSHkUHHLPwZWfOsiTaSwSdxglBk - X-Sonic-MF: X-Sonic-ID: 84b69ef5-9fc8-491a-a4f0-c5f86270fe0c Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Tue, 18 Apr 2023 19:37:45 +0000 Received: by hermes--production-gq1-546798879c-l2qgj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID dc2424b58e19021d7f37588319f5e9ba; Tue, 18 Apr 2023 19:37:40 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 Message-Id: <963B0620-5342-4EC3-AA54-52DDD70D9E3D@yahoo.com> Date: Tue, 18 Apr 2023 12:37:30 -0700 To: =?utf-8?B?Sm9zw6kgUMOpcmV6?= , Current FreeBSD X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <963B0620-5342-4EC3-AA54-52DDD70D9E3D.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.205:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org] X-Rspamd-Queue-Id: 4Q1Dj71kRQz3Jg0 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Jos=C3=A9_P=C3=A9rez wrote on Date: Tue, 18 Apr 2023 16:59:03 UTC : > El 2023-04-17 21:59, Pawel Jakub Dawidek escribi=C3=B3: > > Jos=C3=A9, > >=20 > > I can only speak of block cloning in details, but I'll try to = address > > everything. > >=20 > > The easiest way to avoid block_cloning-related corruption on the > > kernel after the last OpenZFS merge, but before e0bb199925 is to set > > the compress property to 'off' and the sync property to something > > other than 'disabled'. This will avoid the block_cloning-related > > corruption and zil_replaying() panic. > >=20 > > As for the other corruption, unfortunately I don't know the details, > > but my understanding is that it is happening under higher load. Not > > sure I'd trust a kernel built on a machine with this bug present. = What > > I would do is to compile the kernel as of 068913e4ba somewhere else, > > boot the problematic machine in single-user mode and install the = newly > > built kernel. > >=20 > > As far as I can tell, contrary to some initial reports, none of the > > problems introduced by the recent OpenZFS merge corrupt the pool > > metadata, only file's data. You can locate the files modified with = the > > bogus kernel using find(1) with a proper modification time, but you > > have to decide what to do with them (either throw them away, restore > > them from backup or inspect them). >=20 > Sharing my experience on how to get out of the worst case scenario = with=20 > a building machine that is affected by the bug. >=20 > CAVEAT: this is my experience, take it at your own risk. It worked for=20= > me, there is no guarantee that it will work for your. You may create=20= > corrupted files and make your system harder to recover or definitely=20= > brick it. Don't blame me, you have been warned. YMMV. >=20 > Boot in single user mode and check if your pool has block cloning in=20= > use: > # zpool get feature@block_cloning zroot > NAME PROPERTY VALUE SOURCE > zroot feature@block_cloning active local >=20 > In this case it does because the value is "active". If it's "enabled"=20= > you do not need to do anything. Well, if block_cloning is disabled it would not become active. But, if it is enabled, it can automatically become active by creating a first entry in the involved Block Reference Table during any activity meets the criteria for such. If the FreeBSD vintage in place is one that corrupts zfs data for any reason, one would still want to progress to a vintage that does not corrupt zfs data, even if block_cloning is enabled but not active just before starting such an update sequence. So, in progressing past the vintage that corrupt zfs data, one could end up with block_cloning enabled in the process. At least, that is my understanding of the issue. May be only a subset of the "causes data corruption" range of vintages would have to worry about block_cloning becoming active during the effort to get past all the sources of corruptions. (If so, I've no clue what range that would be.) I expect that the "you do not need to do anything" for block_cloning being "enabled" instead of "active" may be too strong of a claim, depending on the specific starting-vintage inside the range with zfs data corruption problems. (=46rom what I've read, when the last Block Reference Table entry is removed for any reason, the matching block_cloning changes back from being indicated as active to being indicated as enabled.) > 1) When in single user mode set compression property to "off" on any = zfs=20 > active dataset that has compression other than "off" and the sync=20 > property to something other than "disabled". > 2) Boot multiuser and update your current sources, e.g. > git update --rebase > 3) Build and install a new kernel without too much pressure (e.g. with=20= > -j 1): > make -j 1 kernel > 4) Reboot with the new kernel > 5) Now you have to reinstall the kernel with > make installkernel > This is because the new kernel files were written by the old kernel=20 > and need to be removed. > 6) Find out when the pool was upgraded (I used command history) and=20 > create a file with that date, in my case: > touch -t 2304161957 /tmp/from > 7) Find out when you booted the new kernel (I used fgrep Copyright=20 > /var/log/messages | tail -n 1) and create a file with that date, in my=20= > case: > touch -t 2304172142 /tmp/to > 8) Find the files/firs created between the two dates: > find / -newerBm /tmp/from -and -not -newerBm /tmp/to >=20 > /tmp/filelist.txt > 9) Inspect /tmp/filelist.txt and save any important items. If the=20 > important files are not corrupted you can do: > cp important_file new; mv new important_file > NOTA BENE: "touch important_file" would not work, you do need to=20 > re-create the file. > 10) Delete the remaining files/dirs in /tmp/filelist.txt. If you did = 5)=20 > you will remove /boot/kernel.old files, but not /boot/kernel files. > 11) Restore your compression and sync properties where appropiate. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com