fsck -C -p: NO WRITE ACCESS

From: Andriy Gapon <avg_at_FreeBSD.org>
Date: Mon, 07 Feb 2022 12:15:57 UTC
I've got a problem where fsck behaves differently from my expectations.
The problem happens with a filesystem on a GELI encrypted ZVOL.
The volume has 4K block size and that's the GELI's sector size as well.
FreeBSD is stable/13 from mid January.

Let me demonstrate:

# fsck -C -p /dev/zvol/.../vault.eli
/dev/zvol/.../vault.eli: NO WRITE ACCESS
/dev/zvol/.../vault.eli: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

# fsck /dev/zvol/.../vault.eli
** /dev/zvol/.../vault.eli

SAVE DATA TO FIND ALTERNATE SUPERBLOCKS? [yn] n


ADD CYLINDER GROUP CHECK-HASH PROTECTION? [yn] n


ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /usr/home/avg/secret
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
136 files, 371 used, 253491 free (35 frags, 31682 blocks, 0.0% fragmentation)

***** FILE SYSTEM IS CLEAN *****

# fsck -C -p /dev/zvol/.../vault.eli
/dev/zvol/.../vault.eli: NO WRITE ACCESS
/dev/zvol/.../vault.eli: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

# mount /dev/zvol/.../vault.eli ~/secret
<success>

So, mount can mount the filesystem without problems.
Plain fsck suggests to enable some features, it does not find any problem and 
sees the filesystem as clean.
But fsck -C -p fails with a rather cryptic error.

I don't think I saw this before.

If I try to allow fsck to enable some things that it wants, it actually does 
fail to write:
# fsck /dev/zvol/.../vault.eli
** /dev/zvol/.../vault.eli

SAVE DATA TO FIND ALTERNATE SUPERBLOCKS? [yn] y


CANNOT WRITE BLK: 15
CONTINUE? [yn] n

With ktrace I see:
fsck_ufs CALL  pwrite(0x4,0x800aa0000,0x1000,0x1e00)
fsck_ufs RET   pwrite -1 errno 22 Invalid argument

So, a 4K write but the offset is not 4K aligned.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


A little bit more data:
# dumpfs -m /dev/zvol/.../vault.eli
# newfs command for /dev/zvol/.../vault.eli (/dev/zvol/.../vault.eli)
newfs -L vault -O 2 -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 
8192 -k 2616 -m 8 -o time -s 2097144 /dev/zvol/.../vault.eli

# geli list
Geom name: zvol/.../vault.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: software
Version: 7
UsedKey: 0
Flags: NONE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: zvol/.../vault.eli
    Mediasize: 1073737728 (1.0G)
    Sectorsize: 4096
    Mode: r0w0e0
Consumers:
1. Name: zvol/.../vault
    Mediasize: 1073741824 (1.0G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1

# tunefs -p /dev/zvol/.../vault.eli
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 disabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            2616
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 vault

# dumpfs /dev/zvol/.../vault.eli
magic   19540119 (UFS2)
last mounted time       Mon Feb  7 13:44:30 2022
last modified time      Mon Feb  7 13:51:18 2022
superblock location     65536   id      [ 581f28b7 8a03690e ]
ncg     4       size    262143  blocks  253862
bsize   32768   shift   15      mask    0xffff8000
fsize   4096    shift   12      mask    0xfffff000
frag    8       shift   3       fsbtodb 3
minfree 8%      optim   time    symlinklen 120
maxbsize 32768  maxbpg  4096    maxcontig 4     contigsumsize 4
nbfree  31682   ndir    27      nifree  130934  nffree  35
bpg     8192    fpg     65536   ipg     32768   unrefs  0
nindir  4096    inopb   128     maxfilesize     2252349704110079
sbsize  4096    cgsize  16384   csaddr  2088    cssize  4096
sblkno  24      cblkno  32      iblkno  40      dblkno  2088
cgrotor 0       fmod    0       ronly   0       clean   1
metaspace 2616  avgfpdir 64     avgfilesize 16384
flags   none

fsmnt   /usr/home/avg/secret
volname vault   swuid   0       providersize    262143

cs[].cs_(nbfree,ndir,nifree,nffree):
         (7905,3,32748,1) (7920,17,32691,10) (7928,3,32755,21) (7929,4,32740,3)
blocks in last group 8191


cg 0:
magic   90255   tell    20000   time    Fri Dec 17 00:25:26 2021
cgx     0       ndblk   65536   niblk   32768   initiblk 256    unrefs 0
nbfree  7905    ndir    3       nifree  32748   nffree  1
rotor   4872    irotor  19      frotor  4744
frsum   1       0       0       0       0       0       0
sum of frsum: 1
clusters 1-3:   0       0       0
clusters size 4 and over: 2
clusters free:  264-587, 611-8191
inodes used:    0-19
blks free:      2112-4703, 4751, 4888-65535

cg 1:
magic   90255   tell    10020000        time    Thu Oct 21 10:39:16 2021
cgx     1       ndblk   65536   niblk   32768   initiblk 256    unrefs 0
nbfree  7920    ndir    17      nifree  32691   nffree  10
rotor   4776    irotor  20      frotor  4776
frsum   5       1       1       0       0       0       0
sum of frsum: 10
clusters 1-3:   0       0       1
clusters size 4 and over: 2
clusters free:  0-2, 265-587, 598-8191
inodes used:    0-19, 21-66, 68, 70, 74-82
blks free:      0-23, 2118, 2120-4703, 4726, 4743, 4745,
         4757-4758, 4764, 4781-65535

cg 2:
magic   90255   tell    20020000        time    Tue Dec 28 18:40:22 2021
cgx     2       ndblk   65536   niblk   32768   initiblk 256    unrefs 0
nbfree  7928    ndir    3       nifree  32755   nffree  21
rotor   4720    irotor  6       frotor  4712
frsum   0       0       0       2       0       1       1
sum of frsum: 21
clusters 1-3:   0       0       1
clusters size 4 and over: 3
clusters free:  0-2, 262-587, 590-641, 645-8191
inodes used:    0-5, 8-9, 12-14, 17, 19
blks free:      0-23, 2090-4703, 4716-5135, 5144-5147, 5153-65535

cg 3:
magic   90255   tell    30020000        time    Wed Sep  1 12:52:02 2021
cgx     3       ndblk   65535   niblk   32768   initiblk 256    unrefs 0
nbfree  7929    ndir    4       nifree  32740   nffree  3
rotor   4736    irotor  26      frotor  4728
frsum   0       0       1       0       0       0       0
sum of frsum: 3
clusters 1-3:   0       0       1
clusters size 4 and over: 2
clusters free:  0-2, 261-587, 592-8190
inodes used:    0-25, 27-28
blks free:      0-23, 2088-4703, 4731-4733, 4736-65527


-- 
Andriy Gapon