[Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs
- Reply: bugzilla-noreply_a_freebsd.org: "[Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs"
- Reply: bugzilla-noreply_a_freebsd.org: "[Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs"
- Reply: bugzilla-noreply_a_freebsd.org: "[Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 12 Nov 2021 12:47:38 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259796 Bug ID: 259796 Summary: [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs Product: Base System Version: 13.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: dch@freebsd.org It's not entirely clear to me what's happened to get to this point, so first I report the observed issue, and then try to recall the steps. some file contents are different after transferring zstd-9 datasets between zpools, between 13.0-RELEASE & 12.3-BETA1 + openzfs-kmod, using `zfs send -LevcR` & `zfs recv -Fuvs` flags. sha comparisons and git repo diffs confirm this, as does hd output. file size appears correct, but files are not in fact same length according to hd(1). -rw-r--r-- 1 dch wheel 461B Oct 30 22:32 base.yml -rw-r--r-- 1 dch wheel 392B Oct 30 22:32 bootstrap.yml -rw-r--r-- 1 dch wheel 280B Oct 30 22:32 certs.yml -rw-r--r-- 1 dch wheel 75B Oct 30 22:32 common.yml -rw-r--r-- 1 dch wheel 797B Oct 30 22:32 dns.yml -rw-r--r-- 1 dch wheel 741B Oct 30 22:32 jails.yml -rw-r--r-- 1 dch wheel 68B Oct 30 22:32 lb.yml -rw-r--r-- 1 dch wheel 401B Oct 30 22:32 mgmt.yml -rw-r--r-- 1 dch wheel 80B Oct 30 22:32 provision.yml -rw-r--r-- 1 dch wheel 266B Oct 30 22:32 site.yml -rw-r--r-- 1 dch wheel 452B Oct 30 22:32 toggle-backends.yml -rw-r--r-- 1 dch wheel 200B Oct 30 22:32 update.yml -rw-r--r-- 1 dch wheel 221B Oct 30 22:32 upgrade.yml -rw-r--r-- 1 dch wheel 93B Oct 30 22:32 zen.yml ## sha256 of yml files from pre & post send filesystems uniq -c |egrep -v '\s+2' 1 SHA256 (dns.yml) = 18681a16ee52ec4f57aef197a28a442157485db38529e2c818a5f2f841a330a0 1 SHA256 (dns.yml) = 7dfdbec88da403910b89aa41601650c2eb1b5e2a580e34d043c39e91cf138849 1 SHA256 (jails.yml) = 4387866be6ce796d321f3738a4a7a2a310bd256f902dd11eca87db700bf2f8c0 1 SHA256 (jails.yml) = 6b6d50f5cd9e52fe3da39a44d0560e5f9b80b9778f8f52d72b7b7c0bb34f97c9 ... # sample file ## corrupt jails.yml 00000000 00 00 00 f5 f0 19 2d 2d 2d 0a 2d 20 68 6f 73 74 |......---.- host| 00000010 73 3a 20 73 65 72 65 6e 69 74 79 5f 6a 61 69 6c |s: serenity_jail| 00000020 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 0e 00 |s. become: ye..| 00000030 a0 67 61 74 68 65 72 5f 66 61 63 2d 00 f5 02 6e |.gather_fac-...n| 00000040 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. -| 00000050 3f 00 16 0a 52 00 4f 65 64 65 6e 4e 00 24 73 64 |?...R.OedenN.$sd| 00000060 6e 73 64 69 73 74 5c 00 00 49 00 03 0b 00 97 7a |nsdist\..I.....z| 00000070 6f 6e 65 32 6a 73 6f 6e 68 00 7f 63 6f 75 63 68 |one2jsonh..couch| 00000080 64 62 6b 00 24 03 3e 00 07 50 00 3f 77 77 77 4c |dbk.$.>..P.?wwwL| 00000090 00 24 20 68 32 18 01 00 0f 01 37 77 77 77 52 00 |.$ h2.....7wwwR.| 000000a0 2f 6d 75 51 00 24 27 6d 75 46 00 1f 66 e4 00 26 |/muQ.$'muF..f..&| 000000b0 19 66 e0 00 6f 6d 61 74 72 69 78 92 00 24 87 64 |.f..omatrix..$.d| 000000c0 65 6e 64 72 69 74 65 98 00 9f 61 76 61 6c 61 6e |endrite...avalan| 000000d0 63 68 65 53 00 24 05 40 00 07 54 00 9f 73 65 61 |cheS.$.@..T..sea| 000000e0 77 65 65 64 66 73 54 00 24 05 40 00 2f 0a 00 01 |weedfsT.$.@./...| 000000f0 00 ff 03 50 00 00 00 00 00 00 00 00 00 00 00 00 |...P............| 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000002e0 too short!! ## valid jails.yml 00000000 2d 2d 2d 0a 2d 20 68 6f 73 74 73 3a 20 73 65 72 |---.- hosts: ser| 00000010 65 6e 69 74 79 5f 6a 61 69 6c 73 0a 20 20 62 65 |enity_jails. be| 00000020 63 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 |come: yes. gath| 00000030 65 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 |er_facts: no. r| 00000040 6f 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 72 65 |oles:. - sere| 00000050 6e 69 74 79 0a 0a 2d 20 68 6f 73 74 73 3a 20 65 |nity..- hosts: e| 00000060 64 65 6e 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f |den_jails. beco| 00000070 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 |me: yes. gather| 00000080 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c |_facts: no. rol| 00000090 65 73 3a 0a 20 20 20 20 2d 20 64 6e 73 64 69 73 |es:. - dnsdis| 000000a0 74 0a 20 20 20 20 2d 20 65 64 65 6e 0a 20 20 20 |t. - eden. | 000000b0 20 2d 20 7a 6f 6e 65 32 6a 73 6f 6e 0a 0a 2d 20 | - zone2json..- | 000000c0 68 6f 73 74 73 3a 20 63 6f 75 63 68 64 62 5f 6a |hosts: couchdb_j| 000000d0 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 |ails. become: y| 000000e0 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 |es. gather_fact| 000000f0 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 |s: no. roles:. | 00000100 20 20 20 2d 20 63 6f 75 63 68 64 62 0a 0a 2d 20 | - couchdb..- | 00000110 68 6f 73 74 73 3a 20 77 77 77 5f 6a 61 69 6c 73 |hosts: www_jails| 00000120 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a 20 |. become: yes. | 00000130 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 6e | gather_facts: n| 00000140 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. -| 00000150 20 68 32 6f 0a 20 20 20 20 2d 20 77 77 77 0a 0a | h2o. - www..| 00000160 2d 20 68 6f 73 74 73 3a 20 6d 75 5f 6a 61 69 6c |- hosts: mu_jail| 00000170 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a |s. become: yes.| 00000180 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 | gather_facts: | 00000190 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 |no. roles:. | 000001a0 2d 20 6d 75 0a 0a 2d 20 68 6f 73 74 73 3a 20 66 |- mu..- hosts: f| 000001b0 64 62 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d |db_jails. becom| 000001c0 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f |e: yes. gather_| 000001d0 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 |facts: no. role| 000001e0 73 3a 0a 20 20 20 20 2d 20 66 64 62 0a 0a 2d 20 |s:. - fdb..- | 000001f0 68 6f 73 74 73 3a 20 6d 61 74 72 69 78 5f 6a 61 |hosts: matrix_ja| 00000200 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 |ils. become: ye| 00000210 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 |s. gather_facts| 00000220 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 |: no. roles:. | 00000230 20 20 2d 20 64 65 6e 64 72 69 74 65 0a 0a 2d 20 | - dendrite..- | 00000240 68 6f 73 74 73 3a 20 61 76 61 6c 61 6e 63 68 65 |hosts: avalanche| 00000250 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a |_jails. become:| 00000260 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 | yes. gather_fa| 00000270 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a |cts: no. roles:| 00000280 0a 20 20 20 20 2d 20 61 76 61 6c 61 6e 63 68 65 |. - avalanche| 00000290 0a 0a 2d 20 68 6f 73 74 73 3a 20 73 65 61 77 65 |..- hosts: seawe| 000002a0 65 64 66 73 5f 6a 61 69 6c 73 0a 20 20 62 65 63 |edfs_jails. bec| 000002b0 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 |ome: yes. gathe| 000002c0 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f |r_facts: no. ro| 000002d0 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 61 77 65 |les:. - seawe| 000002e0 65 64 66 73 0a |edfs.| 000002e5 ## original issue, with "approximate facts" - 13.0-RELEASE zpool is incompatible with 12.3-BETA because of upgraded pool, also wrong ashift - lets just add a new 12.3-BETA1 zpool and zfs send the bits across - re-create the 13.0-RELEASE pool with corrected ashift - zfs send the stuff back again - on booting, we see that the loader is very angry and wont boot - loader won't boot from this new pool zstd-9 compressed bootfs (since confirmed by kevans as by design) - when moving bootfs back to lz4 again we can boot - all other datasets suffer from the same corruption issues, as seen from 13.0-RELEASE ## zpool history | sort 2021-10-31.00:24:03 zpool create -o failmode=continue -O atime=off -O compression=zstd-9 -O checksum=skein -R /mnt puffy /dev/gpt/puffy 2021-10-31.00:38:25 zfs recv -Fuvs -o compression=zstd-9 -o checksum=skein -o atime=off puffy 2021-10-31.00:41:28 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy 2021-10-31.00:46:21 zpool create -o failmode=continue -O atime=off -O compression=zstd-9 -O checksum=skein -R /mnt zroot /dev/gpt/zfs 2021-10-31.00:56:10 zfs send -LevcPR puffy@20211031-0024 2021-10-31.00:56:16 zfs recv -Fuvs zroot 2021-10-31.00:58:53 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot 2021-10-31.00:59:02 zpool set bootfs= puffy 2021-10-31.01:00:19 zfs set mountpoint=/ zroot/ROOT/12.3 2021-10-31.01:00:25 zfs set mountpoint=/ zroot/ROOT/13.0-RELEASE 2021-10-31.01:09:19 zpool set bootfs=zroot/ROOT/12.3 zroot 2021-10-31.02:05:03 zfs set compression=lz4 zroot/ROOT 2021-10-31.02:12:58 zpool import -R /mnt -N puffy 2021-10-31.02:13:03 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy 2021-10-31.02:13:13 zpool set bootfs= zroot 2021-10-31.02:17:33 zpool import -f -R /mnt -N zroot 2021-10-31.02:19:03 zfs destroy -vrf zroot/ROOT 2021-10-31.02:24:00 zfs send -LvPR puffy/ROOT@20211031-0024 2021-10-31.02:24:03 zfs recv -Fuvs -o compression=lz4 zroot/ROOT 2021-10-31.02:24:18 zpool set bootfs= puffy 2021-10-31.02:24:23 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot 2021-10-31.08:48:14 zpool import -fR /mnt -N puffy 2021-10-31.08:48:42 zpool set bootfs=puffy/ROOT/12.3 puffy 2021-10-31.08:48:51 zpool set bootfs= zroot 2021-10-31.08:49:05 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy 2021-10-31.16:31:48 zpool import -f -R /mnt -N zroot 2021-10-31.21:20:24 zfs send -LvcPR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:20:28 zfs recv -Fuvs zroot/ROOT/12.3-lvcpr 2021-10-31.21:22:44 zfs create -o canmount=off -o mountpoint=none -o compression=zstd-9 zroot/WOOT 2021-10-31.21:23:08 zfs destroy -vrRf zroot/ROOT/12.3-lvcpr 2021-10-31.21:26:21 zfs send -LevcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:26:23 zfs recv -Fuvs zroot/WOOT/12.3-levcr 2021-10-31.21:29:13 zfs send -cR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:29:14 zfs recv -Fus zroot/WOOT/12.3-vcr 2021-10-31.21:32:23 zfs send -LvcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:32:25 zfs recv -Fus zroot/WOOT/12.3-lvcr 2021-10-31.21:35:12 zfs send -evcR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:35:14 zfs recv -Fus zroot/WOOT/12.3-evcr 2021-10-31.21:43:04 zfs send -vR puffy/ROOT/12.3@20211031-0024 2021-10-31.21:43:35 zfs recv -Fus zroot/WOOT/12.3-vr 2021-10-31.21:44:15 zfs rename zroot/WOOT zroot/thirteens 2021-10-31.22:02:54 zpool import -f -R /mnt -N zroot 2021-10-31.22:04:06 zfs create -o canmount=off -o mountpoint=none -o compression=zstd-9 zroot/twelve 2021-10-31.22:07:24 zfs send -cR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:07:29 zfs recv -Fus zroot/twelve/12.3-vcr 2021-10-31.22:10:04 zfs send -LevcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:10:09 zfs recv -Fuvs zroot/twelve/12.3-levcr 2021-10-31.22:12:43 zfs send -LvcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:12:49 zfs recv -Fuvs zroot/twelve/12.3-lvcr 2021-10-31.22:15:23 zfs send -evcR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:15:28 zfs recv -Fus zroot/twelve/12.3-evcr 2021-10-31.22:37:18 zfs rename zroot/thirteens zroot/thirteen 2021-10-31.22:40:54 zfs send -vR puffy/ROOT/12.3@20211031-0024 2021-10-31.22:41:52 zfs recv -Fus zroot/twelve/12.3-vr 2021-11-08.08:14:57 zpool import -f -R /mnt -N zroot 2021-11-08.08:37:09 zfs destroy -vrRf zroot/ROOT/12.3 2021-11-08.08:37:28 zfs snapshot -r puffy/ROOT@fixit 2021-11-11.09:54:15 zpool import -f -R /mnt -N zroot 2021-11-11.09:54:47 zfs snapshot -r puffy@20211111-0954 2021-11-11.09:54:47 zfs snapshot -r zroot@20211111-0954 2021-11-11.09:59:42 zfs destroy -vrRf zroot/thirteen 2021-11-11.10:00:12 zfs recv -Fuvs zroot/12.3-RELEASE 2021-11-11.10:00:15 zfs send -LvcR puffy/ROOT/12.3@20211111-0954 2021-11-11.10:00:24 zfs destroy -vrRf zroot/twelve 2021-11-11.10:00:54 zfs rename zroot/12.3-RELEASE zroot/ROOT/12.3-RELEASE 2021-11-11.10:01:35 zpool set bootfs=zroot/ROOT/12.3-RELEASE zroot 2021-11-11.10:01:42 zpool set bootfs= puffy 2021-11-11.10:04:51 zfs destroy -vrf zroot/usr/home/dch 2021-11-11.10:05:35 zfs send -LvcR puffy/usr/home/dch@20211111-0954 2021-11-11.10:05:37 zfs recv -Fuvs zroot/usr/home/dch 2021-11-11.10:06:39 zfs set compression=lz4 zroot 2021-11-11.11:46:45 zpool import -R /mnt -N -f zroot 2021-11-11.11:47:32 zfs destroy -vr zroot@fixit 2021-11-11.11:47:35 zfs snapshot -r zroot@fixit 2021-11-11.12:07:49 zpool import -R /mnt2 -N -f puffy 2021-11-11.12:21:42 zfs rename zroot/ROOT zroot/WOOT 2021-11-11.12:22:01 zfs destroy -r puffy@fixit 2021-11-11.12:22:09 zfs snapshot -r puffy@fixit 2021-11-11.12:26:52 zfs send -LevR puffy/ROOT@fixit 2021-11-11.12:26:58 zfs recv -Fuvs -o compression=lz4 zroot/ROOT 2021-11-11.12:27:33 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot 2021-11-11.12:27:41 zpool set bootfs= puffy 2021-11-11.12:27:53 zpool export puffy 2021-11-11.12:27:56 zpool export zroot 2021-11-11.12:33:05 zfs destroy -vr zroot/WOOT 2021-11-11.12:38:45 zpool import -R /mnt -N puffy 2021-11-11.12:52:47 zfs destroy -vr puffy/usr/home@fixit 2021-11-11.12:53:00 zfs snapshot -r puffy/usr/home@fixit 2021-11-11.12:53:48 zfs rename zroot/usr/home/dch zroot/usr/home/borked 2021-11-11.12:54:41 zfs recv -o compression=lz4 -Fuvs zroot/usr/home/dch 2021-11-11.12:54:41 zfs send -vR puffy/usr/home/dch@fixit ... -- You are receiving this mail because: You are the assignee for the bug.