[Bug 259796] [zfs] datasets corrupted when moving between openzfs on 12.3-BETA* and 13.0-RELEASE zfs

From: <bugzilla-noreply_at_freebsd.org>
Date: Fri, 12 Nov 2021 12:47:38 UTC 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259796

            Bug ID: 259796
           Summary: [zfs] datasets corrupted when moving between openzfs
                    on 12.3-BETA* and 13.0-RELEASE zfs
           Product: Base System
           Version: 13.0-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: dch@freebsd.org

It's not entirely clear to me what's happened to get to this
point, so first I report the observed issue, and then try to
recall the steps.

some file contents are different after transferring zstd-9
datasets between zpools, between 13.0-RELEASE & 12.3-BETA1
+ openzfs-kmod, using `zfs send -LevcR` & `zfs recv -Fuvs`
flags.

sha comparisons and git repo diffs confirm this, as does hd
output. file size appears correct, but files are not in fact
same length according to hd(1).

-rw-r--r--  1 dch  wheel   461B Oct 30 22:32 base.yml
-rw-r--r--  1 dch  wheel   392B Oct 30 22:32 bootstrap.yml
-rw-r--r--  1 dch  wheel   280B Oct 30 22:32 certs.yml
-rw-r--r--  1 dch  wheel    75B Oct 30 22:32 common.yml
-rw-r--r--  1 dch  wheel   797B Oct 30 22:32 dns.yml
-rw-r--r--  1 dch  wheel   741B Oct 30 22:32 jails.yml
-rw-r--r--  1 dch  wheel    68B Oct 30 22:32 lb.yml
-rw-r--r--  1 dch  wheel   401B Oct 30 22:32 mgmt.yml
-rw-r--r--  1 dch  wheel    80B Oct 30 22:32 provision.yml
-rw-r--r--  1 dch  wheel   266B Oct 30 22:32 site.yml
-rw-r--r--  1 dch  wheel   452B Oct 30 22:32 toggle-backends.yml
-rw-r--r--  1 dch  wheel   200B Oct 30 22:32 update.yml
-rw-r--r--  1 dch  wheel   221B Oct 30 22:32 upgrade.yml
-rw-r--r--  1 dch  wheel    93B Oct 30 22:32 zen.yml

## sha256 of yml files from pre & post send filesystems

uniq -c |egrep -v '\s+2'
   1 SHA256 (dns.yml) =
18681a16ee52ec4f57aef197a28a442157485db38529e2c818a5f2f841a330a0
   1 SHA256 (dns.yml) =
7dfdbec88da403910b89aa41601650c2eb1b5e2a580e34d043c39e91cf138849
   1 SHA256 (jails.yml) =
4387866be6ce796d321f3738a4a7a2a310bd256f902dd11eca87db700bf2f8c0
   1 SHA256 (jails.yml) =
6b6d50f5cd9e52fe3da39a44d0560e5f9b80b9778f8f52d72b7b7c0bb34f97c9
...

# sample file

## corrupt jails.yml

00000000  00 00 00 f5 f0 19 2d 2d  2d 0a 2d 20 68 6f 73 74  |......---.- host|
00000010  73 3a 20 73 65 72 65 6e  69 74 79 5f 6a 61 69 6c  |s: serenity_jail|
00000020  73 0a 20 20 62 65 63 6f  6d 65 3a 20 79 65 0e 00  |s.  become: ye..|
00000030  a0 67 61 74 68 65 72 5f  66 61 63 2d 00 f5 02 6e  |.gather_fac-...n|
00000040  6f 0a 20 20 72 6f 6c 65  73 3a 0a 20 20 20 20 2d  |o.  roles:.    -|
00000050  3f 00 16 0a 52 00 4f 65  64 65 6e 4e 00 24 73 64  |?...R.OedenN.$sd|
00000060  6e 73 64 69 73 74 5c 00  00 49 00 03 0b 00 97 7a  |nsdist\..I.....z|
00000070  6f 6e 65 32 6a 73 6f 6e  68 00 7f 63 6f 75 63 68  |one2jsonh..couch|
00000080  64 62 6b 00 24 03 3e 00  07 50 00 3f 77 77 77 4c  |dbk.$.>..P.?wwwL|
00000090  00 24 20 68 32 18 01 00  0f 01 37 77 77 77 52 00  |.$ h2.....7wwwR.|
000000a0  2f 6d 75 51 00 24 27 6d  75 46 00 1f 66 e4 00 26  |/muQ.$'muF..f..&|
000000b0  19 66 e0 00 6f 6d 61 74  72 69 78 92 00 24 87 64  |.f..omatrix..$.d|
000000c0  65 6e 64 72 69 74 65 98  00 9f 61 76 61 6c 61 6e  |endrite...avalan|
000000d0  63 68 65 53 00 24 05 40  00 07 54 00 9f 73 65 61  |cheS.$.@..T..sea|
000000e0  77 65 65 64 66 73 54 00  24 05 40 00 2f 0a 00 01  |weedfsT.$.@./...|
000000f0  00 ff 03 50 00 00 00 00  00 00 00 00 00 00 00 00  |...P............|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000002e0

too short!!

## valid jails.yml

00000000  2d 2d 2d 0a 2d 20 68 6f  73 74 73 3a 20 73 65 72  |---.- hosts: ser|
00000010  65 6e 69 74 79 5f 6a 61  69 6c 73 0a 20 20 62 65  |enity_jails.  be|
00000020  63 6f 6d 65 3a 20 79 65  73 0a 20 20 67 61 74 68  |come: yes.  gath|
00000030  65 72 5f 66 61 63 74 73  3a 20 6e 6f 0a 20 20 72  |er_facts: no.  r|
00000040  6f 6c 65 73 3a 0a 20 20  20 20 2d 20 73 65 72 65  |oles:.    - sere|
00000050  6e 69 74 79 0a 0a 2d 20  68 6f 73 74 73 3a 20 65  |nity..- hosts: e|
00000060  64 65 6e 5f 6a 61 69 6c  73 0a 20 20 62 65 63 6f  |den_jails.  beco|
00000070  6d 65 3a 20 79 65 73 0a  20 20 67 61 74 68 65 72  |me: yes.  gather|
00000080  5f 66 61 63 74 73 3a 20  6e 6f 0a 20 20 72 6f 6c  |_facts: no.  rol|
00000090  65 73 3a 0a 20 20 20 20  2d 20 64 6e 73 64 69 73  |es:.    - dnsdis|
000000a0  74 0a 20 20 20 20 2d 20  65 64 65 6e 0a 20 20 20  |t.    - eden.   |
000000b0  20 2d 20 7a 6f 6e 65 32  6a 73 6f 6e 0a 0a 2d 20  | - zone2json..- |
000000c0  68 6f 73 74 73 3a 20 63  6f 75 63 68 64 62 5f 6a  |hosts: couchdb_j|
000000d0  61 69 6c 73 0a 20 20 62  65 63 6f 6d 65 3a 20 79  |ails.  become: y|
000000e0  65 73 0a 20 20 67 61 74  68 65 72 5f 66 61 63 74  |es.  gather_fact|
000000f0  73 3a 20 6e 6f 0a 20 20  72 6f 6c 65 73 3a 0a 20  |s: no.  roles:. |
00000100  20 20 20 2d 20 63 6f 75  63 68 64 62 0a 0a 2d 20  |   - couchdb..- |
00000110  68 6f 73 74 73 3a 20 77  77 77 5f 6a 61 69 6c 73  |hosts: www_jails|
00000120  0a 20 20 62 65 63 6f 6d  65 3a 20 79 65 73 0a 20  |.  become: yes. |
00000130  20 67 61 74 68 65 72 5f  66 61 63 74 73 3a 20 6e  | gather_facts: n|
00000140  6f 0a 20 20 72 6f 6c 65  73 3a 0a 20 20 20 20 2d  |o.  roles:.    -|
00000150  20 68 32 6f 0a 20 20 20  20 2d 20 77 77 77 0a 0a  | h2o.    - www..|
00000160  2d 20 68 6f 73 74 73 3a  20 6d 75 5f 6a 61 69 6c  |- hosts: mu_jail|
00000170  73 0a 20 20 62 65 63 6f  6d 65 3a 20 79 65 73 0a  |s.  become: yes.|
00000180  20 20 67 61 74 68 65 72  5f 66 61 63 74 73 3a 20  |  gather_facts: |
00000190  6e 6f 0a 20 20 72 6f 6c  65 73 3a 0a 20 20 20 20  |no.  roles:.    |
000001a0  2d 20 6d 75 0a 0a 2d 20  68 6f 73 74 73 3a 20 66  |- mu..- hosts: f|
000001b0  64 62 5f 6a 61 69 6c 73  0a 20 20 62 65 63 6f 6d  |db_jails.  becom|
000001c0  65 3a 20 79 65 73 0a 20  20 67 61 74 68 65 72 5f  |e: yes.  gather_|
000001d0  66 61 63 74 73 3a 20 6e  6f 0a 20 20 72 6f 6c 65  |facts: no.  role|
000001e0  73 3a 0a 20 20 20 20 2d  20 66 64 62 0a 0a 2d 20  |s:.    - fdb..- |
000001f0  68 6f 73 74 73 3a 20 6d  61 74 72 69 78 5f 6a 61  |hosts: matrix_ja|
00000200  69 6c 73 0a 20 20 62 65  63 6f 6d 65 3a 20 79 65  |ils.  become: ye|
00000210  73 0a 20 20 67 61 74 68  65 72 5f 66 61 63 74 73  |s.  gather_facts|
00000220  3a 20 6e 6f 0a 20 20 72  6f 6c 65 73 3a 0a 20 20  |: no.  roles:.  |
00000230  20 20 2d 20 64 65 6e 64  72 69 74 65 0a 0a 2d 20  |  - dendrite..- |
00000240  68 6f 73 74 73 3a 20 61  76 61 6c 61 6e 63 68 65  |hosts: avalanche|
00000250  5f 6a 61 69 6c 73 0a 20  20 62 65 63 6f 6d 65 3a  |_jails.  become:|
00000260  20 79 65 73 0a 20 20 67  61 74 68 65 72 5f 66 61  | yes.  gather_fa|
00000270  63 74 73 3a 20 6e 6f 0a  20 20 72 6f 6c 65 73 3a  |cts: no.  roles:|
00000280  0a 20 20 20 20 2d 20 61  76 61 6c 61 6e 63 68 65  |.    - avalanche|
00000290  0a 0a 2d 20 68 6f 73 74  73 3a 20 73 65 61 77 65  |..- hosts: seawe|
000002a0  65 64 66 73 5f 6a 61 69  6c 73 0a 20 20 62 65 63  |edfs_jails.  bec|
000002b0  6f 6d 65 3a 20 79 65 73  0a 20 20 67 61 74 68 65  |ome: yes.  gathe|
000002c0  72 5f 66 61 63 74 73 3a  20 6e 6f 0a 20 20 72 6f  |r_facts: no.  ro|
000002d0  6c 65 73 3a 0a 20 20 20  20 2d 20 73 65 61 77 65  |les:.    - seawe|
000002e0  65 64 66 73 0a                                    |edfs.|
000002e5

## original issue, with "approximate facts"

- 13.0-RELEASE zpool is incompatible with 12.3-BETA because of upgraded pool,
also wrong ashift
- lets just add a new 12.3-BETA1 zpool and zfs send the bits across
- re-create the 13.0-RELEASE pool with corrected ashift
- zfs send the stuff back again
- on booting, we see that the loader is very angry and wont boot
- loader won't boot from this new pool zstd-9 compressed bootfs (since
confirmed by kevans as by design)
- when moving bootfs back to lz4 again we can boot
- all other datasets suffer from the same corruption issues, as seen from
13.0-RELEASE

## zpool history | sort

2021-10-31.00:24:03 zpool create -o failmode=continue -O atime=off -O
compression=zstd-9 -O checksum=skein -R /mnt puffy /dev/gpt/puffy
2021-10-31.00:38:25 zfs recv -Fuvs -o compression=zstd-9 -o checksum=skein -o
atime=off puffy
2021-10-31.00:41:28 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.00:46:21 zpool create -o failmode=continue -O atime=off -O
compression=zstd-9 -O checksum=skein -R /mnt zroot /dev/gpt/zfs
2021-10-31.00:56:10 zfs send -LevcPR puffy@20211031-0024
2021-10-31.00:56:16 zfs recv -Fuvs zroot
2021-10-31.00:58:53 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-10-31.00:59:02 zpool set bootfs= puffy
2021-10-31.01:00:19 zfs set mountpoint=/ zroot/ROOT/12.3
2021-10-31.01:00:25 zfs set mountpoint=/ zroot/ROOT/13.0-RELEASE
2021-10-31.01:09:19 zpool set bootfs=zroot/ROOT/12.3 zroot
2021-10-31.02:05:03 zfs set compression=lz4 zroot/ROOT
2021-10-31.02:12:58 zpool import -R /mnt -N puffy
2021-10-31.02:13:03 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.02:13:13 zpool set bootfs= zroot
2021-10-31.02:17:33 zpool import -f -R /mnt -N zroot
2021-10-31.02:19:03 zfs destroy -vrf zroot/ROOT
2021-10-31.02:24:00 zfs send -LvPR puffy/ROOT@20211031-0024
2021-10-31.02:24:03 zfs recv -Fuvs -o compression=lz4 zroot/ROOT
2021-10-31.02:24:18 zpool set bootfs= puffy
2021-10-31.02:24:23 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-10-31.08:48:14 zpool import -fR /mnt -N puffy
2021-10-31.08:48:42 zpool set bootfs=puffy/ROOT/12.3 puffy
2021-10-31.08:48:51 zpool set bootfs= zroot
2021-10-31.08:49:05 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.16:31:48 zpool import -f -R /mnt -N zroot
2021-10-31.21:20:24 zfs send -LvcPR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:20:28 zfs recv -Fuvs zroot/ROOT/12.3-lvcpr
2021-10-31.21:22:44 zfs create -o canmount=off -o mountpoint=none -o
compression=zstd-9 zroot/WOOT
2021-10-31.21:23:08 zfs destroy -vrRf zroot/ROOT/12.3-lvcpr
2021-10-31.21:26:21 zfs send -LevcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:26:23 zfs recv -Fuvs zroot/WOOT/12.3-levcr
2021-10-31.21:29:13 zfs send -cR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:29:14 zfs recv -Fus zroot/WOOT/12.3-vcr
2021-10-31.21:32:23 zfs send -LvcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:32:25 zfs recv -Fus zroot/WOOT/12.3-lvcr
2021-10-31.21:35:12 zfs send -evcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:35:14 zfs recv -Fus zroot/WOOT/12.3-evcr
2021-10-31.21:43:04 zfs send -vR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:43:35 zfs recv -Fus zroot/WOOT/12.3-vr
2021-10-31.21:44:15 zfs rename zroot/WOOT zroot/thirteens
2021-10-31.22:02:54 zpool import -f -R /mnt -N zroot
2021-10-31.22:04:06 zfs create -o canmount=off -o mountpoint=none -o
compression=zstd-9 zroot/twelve
2021-10-31.22:07:24 zfs send -cR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:07:29 zfs recv -Fus zroot/twelve/12.3-vcr
2021-10-31.22:10:04 zfs send -LevcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:10:09 zfs recv -Fuvs zroot/twelve/12.3-levcr
2021-10-31.22:12:43 zfs send -LvcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:12:49 zfs recv -Fuvs zroot/twelve/12.3-lvcr
2021-10-31.22:15:23 zfs send -evcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:15:28 zfs recv -Fus zroot/twelve/12.3-evcr
2021-10-31.22:37:18 zfs rename zroot/thirteens zroot/thirteen
2021-10-31.22:40:54 zfs send -vR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:41:52 zfs recv -Fus zroot/twelve/12.3-vr
2021-11-08.08:14:57 zpool import -f -R /mnt -N zroot
2021-11-08.08:37:09 zfs destroy -vrRf zroot/ROOT/12.3
2021-11-08.08:37:28 zfs snapshot -r puffy/ROOT@fixit
2021-11-11.09:54:15 zpool import -f -R /mnt -N zroot
2021-11-11.09:54:47 zfs snapshot -r puffy@20211111-0954
2021-11-11.09:54:47 zfs snapshot -r zroot@20211111-0954
2021-11-11.09:59:42 zfs destroy -vrRf zroot/thirteen
2021-11-11.10:00:12 zfs recv -Fuvs zroot/12.3-RELEASE
2021-11-11.10:00:15 zfs send -LvcR puffy/ROOT/12.3@20211111-0954
2021-11-11.10:00:24 zfs destroy -vrRf zroot/twelve
2021-11-11.10:00:54 zfs rename zroot/12.3-RELEASE zroot/ROOT/12.3-RELEASE
2021-11-11.10:01:35 zpool set bootfs=zroot/ROOT/12.3-RELEASE zroot
2021-11-11.10:01:42 zpool set bootfs= puffy
2021-11-11.10:04:51 zfs destroy -vrf zroot/usr/home/dch
2021-11-11.10:05:35 zfs send -LvcR puffy/usr/home/dch@20211111-0954
2021-11-11.10:05:37 zfs recv -Fuvs zroot/usr/home/dch
2021-11-11.10:06:39 zfs set compression=lz4 zroot
2021-11-11.11:46:45 zpool import -R /mnt -N -f zroot
2021-11-11.11:47:32 zfs destroy -vr zroot@fixit
2021-11-11.11:47:35 zfs snapshot -r zroot@fixit
2021-11-11.12:07:49 zpool import -R /mnt2 -N -f puffy
2021-11-11.12:21:42 zfs rename zroot/ROOT zroot/WOOT
2021-11-11.12:22:01 zfs destroy -r puffy@fixit
2021-11-11.12:22:09 zfs snapshot -r puffy@fixit
2021-11-11.12:26:52 zfs send -LevR puffy/ROOT@fixit
2021-11-11.12:26:58 zfs recv -Fuvs -o compression=lz4 zroot/ROOT
2021-11-11.12:27:33 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-11-11.12:27:41 zpool set bootfs= puffy
2021-11-11.12:27:53 zpool export puffy
2021-11-11.12:27:56 zpool export zroot
2021-11-11.12:33:05 zfs destroy -vr zroot/WOOT
2021-11-11.12:38:45 zpool import -R /mnt -N puffy
2021-11-11.12:52:47 zfs destroy -vr puffy/usr/home@fixit
2021-11-11.12:53:00 zfs snapshot -r puffy/usr/home@fixit
2021-11-11.12:53:48 zfs rename zroot/usr/home/dch zroot/usr/home/borked
2021-11-11.12:54:41 zfs recv -o compression=lz4 -Fuvs zroot/usr/home/dch
2021-11-11.12:54:41 zfs send -vR puffy/usr/home/dch@fixit
...

-- 
You are receiving this mail because:
You are the assignee for the bug.