From nobody Fri Nov 12 12:47:38 2021 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 604B81858349 for ; Fri, 12 Nov 2021 12:47:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HrJJp287cz4kw5 for ; Fri, 12 Nov 2021 12:47:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 25FFD1F7ED for ; Fri, 12 Nov 2021 12:47:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1ACClck9098056 for ; Fri, 12 Nov 2021 12:47:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1ACClcGi098055 for bugs@FreeBSD.org; Fri, 12 Nov 2021 12:47:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs Date: Fri, 12 Nov 2021 12:47:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dch@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259796 Bug ID: 259796 Summary: [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs Product: Base System Version: 13.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: dch@freebsd.org It's not entirely clear to me what's happened to get to this point, so first I report the observed issue, and then try to recall the steps. some file contents are different after transferring zstd-9 datasets between zpools, between 13.0-RELEASE & 12.3-BETA1 + openzfs-kmod, using `zfs send -LevcR` & `zfs recv -Fuvs` flags. sha comparisons and git repo diffs confirm this, as does hd output. file size appears correct, but files are not in fact same length according to hd(1). -rw-r--r-- 1 dch wheel 461B Oct 30 22:32 base.yml -rw-r--r-- 1 dch wheel 392B Oct 30 22:32 bootstrap.yml -rw-r--r-- 1 dch wheel 280B Oct 30 22:32 certs.yml -rw-r--r-- 1 dch wheel 75B Oct 30 22:32 common.yml -rw-r--r-- 1 dch wheel 797B Oct 30 22:32 dns.yml -rw-r--r-- 1 dch wheel 741B Oct 30 22:32 jails.yml -rw-r--r-- 1 dch wheel 68B Oct 30 22:32 lb.yml -rw-r--r-- 1 dch wheel 401B Oct 30 22:32 mgmt.yml -rw-r--r-- 1 dch wheel 80B Oct 30 22:32 provision.yml -rw-r--r-- 1 dch wheel 266B Oct 30 22:32 site.yml -rw-r--r-- 1 dch wheel 452B Oct 30 22:32 toggle-backends.yml -rw-r--r-- 1 dch wheel 200B Oct 30 22:32 update.yml -rw-r--r-- 1 dch wheel 221B Oct 30 22:32 upgrade.yml -rw-r--r-- 1 dch wheel 93B Oct 30 22:32 zen.yml ## sha256 of yml files from pre & post send filesystems uniq -c |egrep -v '\s+2' 1 SHA256 (dns.yml) =3D 18681a16ee52ec4f57aef197a28a442157485db38529e2c818a5f2f841a330a0 1 SHA256 (dns.yml) =3D 7dfdbec88da403910b89aa41601650c2eb1b5e2a580e34d043c39e91cf138849 1 SHA256 (jails.yml) =3D 4387866be6ce796d321f3738a4a7a2a310bd256f902dd11eca87db700bf2f8c0 1 SHA256 (jails.yml) =3D 6b6d50f5cd9e52fe3da39a44d0560e5f9b80b9778f8f52d72b7b7c0bb34f97c9 ... # sample file ## corrupt jails.yml 00000000 00 00 00 f5 f0 19 2d 2d 2d 0a 2d 20 68 6f 73 74 |......---.- ho= st| 00000010 73 3a 20 73 65 72 65 6e 69 74 79 5f 6a 61 69 6c |s: serenity_ja= il| 00000020 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 0e 00 |s. become: ye= ..| 00000030 a0 67 61 74 68 65 72 5f 66 61 63 2d 00 f5 02 6e |.gather_fac-..= .n| 00000040 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. = -| 00000050 3f 00 16 0a 52 00 4f 65 64 65 6e 4e 00 24 73 64 |?...R.OedenN.$= sd| 00000060 6e 73 64 69 73 74 5c 00 00 49 00 03 0b 00 97 7a |nsdist\..I....= .z| 00000070 6f 6e 65 32 6a 73 6f 6e 68 00 7f 63 6f 75 63 68 |one2jsonh..cou= ch| 00000080 64 62 6b 00 24 03 3e 00 07 50 00 3f 77 77 77 4c |dbk.$.>..P.?ww= wL| 00000090 00 24 20 68 32 18 01 00 0f 01 37 77 77 77 52 00 |.$ h2.....7www= R.| 000000a0 2f 6d 75 51 00 24 27 6d 75 46 00 1f 66 e4 00 26 |/muQ.$'muF..f.= .&| 000000b0 19 66 e0 00 6f 6d 61 74 72 69 78 92 00 24 87 64 |.f..omatrix..$= .d| 000000c0 65 6e 64 72 69 74 65 98 00 9f 61 76 61 6c 61 6e |endrite...aval= an| 000000d0 63 68 65 53 00 24 05 40 00 07 54 00 9f 73 65 61 |cheS.$.@..T..s= ea| 000000e0 77 65 65 64 66 73 54 00 24 05 40 00 2f 0a 00 01 |weedfsT.$.@./.= ..| 000000f0 00 ff 03 50 00 00 00 00 00 00 00 00 00 00 00 00 |...P..........= ..| 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |..............= ..| * 000002e0 too short!! ## valid jails.yml 00000000 2d 2d 2d 0a 2d 20 68 6f 73 74 73 3a 20 73 65 72 |---.- hosts: s= er| 00000010 65 6e 69 74 79 5f 6a 61 69 6c 73 0a 20 20 62 65 |enity_jails. = be| 00000020 63 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 |come: yes. ga= th| 00000030 65 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 |er_facts: no. = r| 00000040 6f 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 72 65 |oles:. - se= re| 00000050 6e 69 74 79 0a 0a 2d 20 68 6f 73 74 73 3a 20 65 |nity..- hosts:= e| 00000060 64 65 6e 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f |den_jails. be= co| 00000070 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 |me: yes. gath= er| 00000080 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c |_facts: no. r= ol| 00000090 65 73 3a 0a 20 20 20 20 2d 20 64 6e 73 64 69 73 |es:. - dnsd= is| 000000a0 74 0a 20 20 20 20 2d 20 65 64 65 6e 0a 20 20 20 |t. - eden. = | 000000b0 20 2d 20 7a 6f 6e 65 32 6a 73 6f 6e 0a 0a 2d 20 | - zone2json..= - | 000000c0 68 6f 73 74 73 3a 20 63 6f 75 63 68 64 62 5f 6a |hosts: couchdb= _j| 000000d0 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 |ails. become:= y| 000000e0 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 |es. gather_fa= ct| 000000f0 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 |s: no. roles:= . | 00000100 20 20 20 2d 20 63 6f 75 63 68 64 62 0a 0a 2d 20 | - couchdb..= - | 00000110 68 6f 73 74 73 3a 20 77 77 77 5f 6a 61 69 6c 73 |hosts: www_jai= ls| 00000120 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a 20 |. become: yes= . | 00000130 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 6e | gather_facts:= n| 00000140 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. = -| 00000150 20 68 32 6f 0a 20 20 20 20 2d 20 77 77 77 0a 0a | h2o. - www= ..| 00000160 2d 20 68 6f 73 74 73 3a 20 6d 75 5f 6a 61 69 6c |- hosts: mu_ja= il| 00000170 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a |s. become: ye= s.| 00000180 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 | gather_facts= : | 00000190 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 |no. roles:. = | 000001a0 2d 20 6d 75 0a 0a 2d 20 68 6f 73 74 73 3a 20 66 |- mu..- hosts:= f| 000001b0 64 62 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d |db_jails. bec= om| 000001c0 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f |e: yes. gathe= r_| 000001d0 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 |facts: no. ro= le| 000001e0 73 3a 0a 20 20 20 20 2d 20 66 64 62 0a 0a 2d 20 |s:. - fdb..= - | 000001f0 68 6f 73 74 73 3a 20 6d 61 74 72 69 78 5f 6a 61 |hosts: matrix_= ja| 00000200 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 |ils. become: = ye| 00000210 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 |s. gather_fac= ts| 00000220 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 |: no. roles:.= | 00000230 20 20 2d 20 64 65 6e 64 72 69 74 65 0a 0a 2d 20 | - dendrite..= - | 00000240 68 6f 73 74 73 3a 20 61 76 61 6c 61 6e 63 68 65 |hosts: avalanc= he| 00000250 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a |_jails. becom= e:| 00000260 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 | yes. gather_= fa| 00000270 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a |cts: no. role= s:| 00000280 0a 20 20 20 20 2d 20 61 76 61 6c 61 6e 63 68 65 |. - avalanc= he| 00000290 0a 0a 2d 20 68 6f 73 74 73 3a 20 73 65 61 77 65 |..- hosts: sea= we| 000002a0 65 64 66 73 5f 6a 61 69 6c 73 0a 20 20 62 65 63 |edfs_jails. b= ec| 000002b0 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 |ome: yes. gat= he| 000002c0 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f |r_facts: no. = ro| 000002d0 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 61 77 65 |les:. - sea= we| 000002e0 65 64 66 73 0a |edfs.| 000002e5 ## original issue, with "approximate facts" - 13.0-RELEASE zpool is incompatible with 12.3-BETA because of upgraded poo= l, also wrong ashift - lets just add a new 12.3-BETA1 zpool and zfs send the bits across - re-create the 13.0-RELEASE pool with corrected ashift - zfs send the stuff back again - on booting, we see that the loader is very angry and wont boot - loader won't boot from this new pool zstd-9 compressed bootfs (since confirmed by kevans as by design) - when moving bootfs back to lz4 again we can boot - all other datasets suffer from the same corruption issues, as seen from 13.0-RELEASE ## zpool history | sort 2021-10-31.00:24:03 zpool create -o failmode=3Dcontinue -O atime=3Doff -O compression=3Dzstd-9 -O checksum=3Dskein -R /mnt puffy /dev/gpt/puffy 2021-10-31.00:38:25 zfs recv -Fuvs -o compression=3Dzstd-9 -o checksum=3Dsk= ein -o atime=3Doff puffy 2021-10-31.00:41:28 zpool set bootfs=3Dpuffy/ROOT/13.0-RELEASE puffy 2021-10-31.00:46:21 zpool create -o failmode=3Dcontinue -O atime=3Doff -O compression=3Dzstd-9 -O checksum=3Dskein -R /mnt zroot /dev/gpt/zfs 2021-10-31.00:56:10 zfs send -LevcPR puffy@20211031-0024 2021-10-31.00:56:16 zfs recv -Fuvs zroot 2021-10-31.00:58:53 zpool set bootfs=3Dzroot/ROOT/13.0-RELEASE zroot 2021-10-31.00:59:02 zpool set bootfs=3D puffy 2021-10-31.01:00:19 zfs set mountpoint=3D/ zroot/ROOT/12.3 2021-10-31.01:00:25 zfs set mountpoint=3D/ zroot/ROOT/13.0-RELEASE 2021-10-31.01:09:19 zpool set bootfs=3Dzroot/ROOT/12.3 zroot 2021-10-31.02:05:03 zfs set compression=3Dlz4 zroot/ROOT 2021-10-31.02:12:58 zpool import -R /mnt -N puffy 2021-10-31.02:13:03 zpool set bootfs=3Dpuffy/ROOT/13.0-RELEASE puffy 2021-10-31.02:13:13 zpool set bootfs=3D zroot 2021-10-31.02:17:33 zpool import -f -R /mnt -N zroot 2021-10-31.02:19:03 zfs destroy -vrf zroot/ROOT 2021-10-31.02:24:00 zfs send -LvPR puffy/ROOT@20211031-0024 2021-10-31.02:24:03 zfs recv -Fuvs -o compression=3Dlz4 zroot/ROOT 2021-10-31.02:24:18 zpool set bootfs=3D puffy 2021-10-31.02:24:23 zpool set bootfs=3Dzroot/ROOT/13.0-RELEASE zroot 2021-10-31.08:48:14 zpool import -fR /mnt -N puffy 2021-10-31.08:48:42 zpool set bootfs=3Dpuffy/ROOT/12.3 puffy 2021-10-31.08:48:51 zpool set bootfs=3D zroot 2021-10-31.08:49:05 zpool set bootfs=3Dpuffy/ROOT/13.0-RELEASE puffy 2021-10-31.16:31:48 zpool import -f -R /mnt -N zroot 2021-10-31.21:20:24 zfs send -LvcPR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:20:28 zfs recv -Fuvs zroot/ROOT/12.3-lvcpr 2021-10-31.21:22:44 zfs create -o canmount=3Doff -o mountpoint=3Dnone -o compression=3Dzstd-9 zroot/WOOT 2021-10-31.21:23:08 zfs destroy -vrRf zroot/ROOT/12.3-lvcpr 2021-10-31.21:26:21 zfs send -LevcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:26:23 zfs recv -Fuvs zroot/WOOT/12.3-levcr 2021-10-31.21:29:13 zfs send -cR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:29:14 zfs recv -Fus zroot/WOOT/12.3-vcr 2021-10-31.21:32:23 zfs send -LvcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:32:25 zfs recv -Fus zroot/WOOT/12.3-lvcr 2021-10-31.21:35:12 zfs send -evcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:35:14 zfs recv -Fus zroot/WOOT/12.3-evcr 2021-10-31.21:43:04 zfs send -vR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:43:35 zfs recv -Fus zroot/WOOT/12.3-vr 2021-10-31.21:44:15 zfs rename zroot/WOOT zroot/thirteens 2021-10-31.22:02:54 zpool import -f -R /mnt -N zroot 2021-10-31.22:04:06 zfs create -o canmount=3Doff -o mountpoint=3Dnone -o compression=3Dzstd-9 zroot/twelve 2021-10-31.22:07:24 zfs send -cR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:07:29 zfs recv -Fus zroot/twelve/12.3-vcr 2021-10-31.22:10:04 zfs send -LevcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:10:09 zfs recv -Fuvs zroot/twelve/12.3-levcr 2021-10-31.22:12:43 zfs send -LvcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:12:49 zfs recv -Fuvs zroot/twelve/12.3-lvcr 2021-10-31.22:15:23 zfs send -evcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:15:28 zfs recv -Fus zroot/twelve/12.3-evcr 2021-10-31.22:37:18 zfs rename zroot/thirteens zroot/thirteen 2021-10-31.22:40:54 zfs send -vR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:41:52 zfs recv -Fus zroot/twelve/12.3-vr 2021-11-08.08:14:57 zpool import -f -R /mnt -N zroot 2021-11-08.08:37:09 zfs destroy -vrRf zroot/ROOT/12.3 2021-11-08.08:37:28 zfs snapshot -r puffy/ROOT@fixit 2021-11-11.09:54:15 zpool import -f -R /mnt -N zroot 2021-11-11.09:54:47 zfs snapshot -r puffy@20211111-0954 2021-11-11.09:54:47 zfs snapshot -r zroot@20211111-0954 2021-11-11.09:59:42 zfs destroy -vrRf zroot/thirteen 2021-11-11.10:00:12 zfs recv -Fuvs zroot/12.3-RELEASE 2021-11-11.10:00:15 zfs send -LvcR puffy/ROOT/12.3@20211111-0954 2021-11-11.10:00:24 zfs destroy -vrRf zroot/twelve 2021-11-11.10:00:54 zfs rename zroot/12.3-RELEASE zroot/ROOT/12.3-RELEASE 2021-11-11.10:01:35 zpool set bootfs=3Dzroot/ROOT/12.3-RELEASE zroot 2021-11-11.10:01:42 zpool set bootfs=3D puffy 2021-11-11.10:04:51 zfs destroy -vrf zroot/usr/home/dch 2021-11-11.10:05:35 zfs send -LvcR puffy/usr/home/dch@20211111-0954 2021-11-11.10:05:37 zfs recv -Fuvs zroot/usr/home/dch 2021-11-11.10:06:39 zfs set compression=3Dlz4 zroot 2021-11-11.11:46:45 zpool import -R /mnt -N -f zroot 2021-11-11.11:47:32 zfs destroy -vr zroot@fixit 2021-11-11.11:47:35 zfs snapshot -r zroot@fixit 2021-11-11.12:07:49 zpool import -R /mnt2 -N -f puffy 2021-11-11.12:21:42 zfs rename zroot/ROOT zroot/WOOT 2021-11-11.12:22:01 zfs destroy -r puffy@fixit 2021-11-11.12:22:09 zfs snapshot -r puffy@fixit 2021-11-11.12:26:52 zfs send -LevR puffy/ROOT@fixit 2021-11-11.12:26:58 zfs recv -Fuvs -o compression=3Dlz4 zroot/ROOT 2021-11-11.12:27:33 zpool set bootfs=3Dzroot/ROOT/13.0-RELEASE zroot 2021-11-11.12:27:41 zpool set bootfs=3D puffy 2021-11-11.12:27:53 zpool export puffy 2021-11-11.12:27:56 zpool export zroot 2021-11-11.12:33:05 zfs destroy -vr zroot/WOOT 2021-11-11.12:38:45 zpool import -R /mnt -N puffy 2021-11-11.12:52:47 zfs destroy -vr puffy/usr/home@fixit 2021-11-11.12:53:00 zfs snapshot -r puffy/usr/home@fixit 2021-11-11.12:53:48 zfs rename zroot/usr/home/dch zroot/usr/home/borked 2021-11-11.12:54:41 zfs recv -o compression=3Dlz4 -Fuvs zroot/usr/home/dch 2021-11-11.12:54:41 zfs send -vR puffy/usr/home/dch@fixit ... --=20 You are receiving this mail because: You are the assignee for the bug.=