From nobody Wed Apr 12 17:37:59 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PxVKm583cz458PG; Wed, 12 Apr 2023 17:38:04 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PxVKm2X9Yz3lRH; Wed, 12 Apr 2023 17:38:04 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; none Received: from shw-obgw-4001a.ext.cloudfilter.net ([10.228.9.142]) by cmsmtp with ESMTP id mZ8bp7qvVjvm1mePzp7wCB; Wed, 12 Apr 2023 17:38:03 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id mePxpFViHHFsOmePypRsBg; Wed, 12 Apr 2023 17:38:03 +0000 X-Authority-Analysis: v=2.4 cv=XZqaca15 c=1 sm=1 tr=0 ts=6436ec7b a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=IkcTkHD0fZMA:10 a=dKHAf1wccvYA:10 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=EkcXrb_YAAAA:8 a=Rrb4mQHHm2gU3NfK1bwA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from [127.0.0.1] (S01060c8ddb6bf355.gv.shawcable.net [24.69.190.13]) by spqr.komquats.com (Postfix) with ESMTPSA id 4903942E; Wed, 12 Apr 2023 10:38:01 -0700 (PDT) Date: Wed, 12 Apr 2023 10:37:59 -0700 From: Cy Schubert To: Charlie Li CC: Rick Macklem , Martin Matuska , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 In-Reply-To: <64e4af2a-5273-6219-c146-f867160f09ac@freebsd.org> References: <202304031513.333FD6qw014903@gitrepo.freebsd.org> <20230403235851.84C0467@slippy.cwsent.com> <20230404052811.DA2172C1@slippy.cwsent.com> <7c75b934-cb0a-b32e-bc19-b1e15e8cf3aa@freebsd.org> <20230409154042.0685a273@cschubert.com> <707e4671-d746-aa23-e340-6eb8f50f78c6@freebsd.org> <20230409205826.7802259d@cschubert.com> <4e85eb84-f0cc-2f8c-d3d9-1e016ede042a@freebsd.org> <20230410165406.51bcd958@cschubert.com> <70739834-4eea-db30-63be-556bcfd881a1@freebsd.org> <464cc8cd-2bf6-b7e5-3823-89227d842458@freebsd.org> <64e4af2a-5273-6219-c146-f867160f09ac@freebsd.org> Message-ID: <00780E30-8E72-4746-B651-8A9A048C9EE4@cschubert.com> List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-CMAE-Envelope: MS4xfK+FtYA4Tjj91OVElqh36swKJ1Bk0Pe4FDKUo2trqFTbBn46LuR7jsH83oqn0R/0VCb+Iq2zAn59GYhCHJrF3HrExNDRQ4Ns7Jn+5pjbBF7JSSEB4Wje qkoGtQgfZ8/GzfHyaDo3xSjT/aig49EO2D37gG+BOsPaLUs1wMrN4mAwW5evbs/PM/Z3gOG9Pgi1IO1gQp3VBpHcbYUJdPPng3C3sqt6TvkIyq1oPPdZaP3e NywS5UgYAKWk8trIu99aweQRI4yeNcJqoG86JNOzZoXikr+h3OjRH9zvd62YiLczQgqri/jjn7R+usTHVoZLbF6I0UHLjJatR5vcLJb9/scqxqeb8qngY+tR QiqK2w1gOGRgf71ko0re3mA1BiZR8hJRf1k/w/5rsdF9Fw2SIV0= X-Rspamd-Queue-Id: 4PxVKm2X9Yz3lRH X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On April 12, 2023 10:22:25 AM PDT, Charlie Li wrote= : >Charlie Li wrote: >> Cy Schubert wrote: >>> On April 12, 2023 8:51:09 AM PDT, Charlie Li w= rote: >>>> Cy Schubert wrote: >>>>> I have a "sandhbox" pool, called t, used for /usr/obj and ports wrkd= irs, and other writes I can easily recreate on my laptop=2E Here are the re= sults of my tests=2E >>>>>=20 >>>>> Method: >>>>>=20 >>>>> Initially I copied my /usr/obj from my two build machines (one amd64= =2Eamd64 and an i386=2Ei386) to my "sandbox" zpool=2E >>>>>=20 >>>>> Next, with block_cloning disabled I did cp -R of the /usr/obj test f= iles=2E Then a diff -qr=2E They source and target directories were the same= =2E >>>>>=20 >>>>> Next, I cleaned up (rm -rf) the target directory to prepare for the >>>>> block_clone enabled test=2E >>>>>=20 >>>>> Next, I did zpool checkpoint t=2E After this, zpool upgrade t=2E Poo= l t now has block_cloning enabled=2E >>>>>=20 >>>>> I repeated the cp -R test from above followed by a diff -qr=2E Almos= t >>>>> every file was different=2E The pool was corrupted=2E >>>>>=20 >>>>> I restored the pool by the following removing the corruption: >>>>>=20 >>>>>=20 >>>>> slippy# zpool export t >>>>> slippy# zpool import --rewind-to-checkpoint t >>>>> slippy# >>>>>=20 >>>>> It is recommended that people avoid upgrading their zpools until the >>>>> problem is fixed=2E >>>>>=20 >>>> As of af7624ed3145, I just did this with an md(4)-backed test pool, t= hough with the second `cp -R` landing in a separate dataset, created and de= stroyed for each test=2E No corruption either way=2E However, my poudriere = builds still output/package corrupted files (particularly those with null c= haracters), probably after install(1) invocations (not cp(1))=2E >>>>=20 >>>=20 >>> You need to copy from/to the same dataset to reproduce the problem=2E = Copying from a source dataset to a different dataset will avoid block_cloni= ng=2E >>>=20 >> Got the corruption now=2E >>=20 >Clarify: no corruption without block_cloning, corruption with=2E > >What is still a mystery to me is how corruption happens even without bloc= k_cloning in the poudriere scenario=2E cp(1)/install(1) always happen withi= n the same dataset, as this test=2E > This is because your pool has previously corrupted blocks=2E Even when you= backed up the old pool, created a new pool without block_cloning and resto= red your data, because the backup contained corrupted blocks from your old = pool, they were restored as is=2E ZFS can only fix corruption if the checks= um says it's corrupt=2E As far as ZFS was concerned at the time those block= s were not corrupted=2E You will need to delete the files with corruption a= nd recreate them=2E Even after this regression is fixed and people build/install kernel, whate= ver was corrupted will remain until corrupted files are either removed and = recreated or fixed manually=2E This regression will have long lasting effects=2E Like Kirk McKusick has reiterated many times, back in the old days people = didn't trust EXT*FS because of the data corruption experienced=2E Sadly ZFS= will need to earn people's trust back again=2E This is unfortunate=2E --=20 Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD=2Eorg NTP: Web: https://nwtime=2Eorg e^(i*pi)+1=3D0 Pardon the typos=2E Small keyboard in use=2E