Reading a corrupted file on ZFS

[Karl Denninger]

On 2/12/2021 09:53, Artem Kuchin wrote:
> I did a little experiment
>
> I have a mirror ZFS pool called ZDATA, i created a file and damanged 
> the same byte in file in it on both disks.
>
> File has ABCDEF string.
>
>
>         ZDATA       ONLINE       0     0     0
>           mirror-0  ONLINE       0     0     0
>             ada0    ONLINE       0     0     1
>             ada1    ONLINE       0     0     0
>
>
> destroy ZDATA
>
> on ada0 change A to Z
> on ada1 change A to Y
> reimport
>
> root at xigmanas:~# zpool status
>   pool: ZDATA
>  state: ONLINE
>   scan: scrub repaired 24K in 0 days 00:03:01 with 0 errors on Fri Feb 
> 12 16:48:46 2021
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         ZDATA       ONLINE       0     0     0
>           mirror-0  ONLINE       0     0     0
>             ada0    ONLINE       0     0     0
>             ada1    ONLINE       0     0     0
>
> errors: No known data errors
>
> Hmm. it repaired something. Read data from disks.
> ada0 - Z
> ada1 - Y
> so, that repair was not the rotten bytes.
>
> Now run scrub
>
> root at xigmanas:~# zpool status -v ZDATA
>   pool: ZDATA
>  state: ONLINE
> status: One or more devices has experienced an error resulting in data
>         corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore the
>         entire pool from backup.
>    see:http://illumos.org/msg/ZFS-8000-8A 
> <http://illumos.org/msg/ZFS-8000-8A>
>   scan: scrub repaired 0 in 0 days 00:03:02 with 1 errors on Fri Feb 
> 12 16:59:49 2021
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         ZDATA       ONLINE       0     0     1
>           mirror-0  ONLINE       0     0     2
>             ada0    ONLINE       0     0     2
>             ada1    ONLINE       0     0     2
>
> errors: Permanent errors have been detected in the following files:
>
>         /data/DATASET1/test.file
>
> the data is still Z and Y, not sync-ed.
>
> I cannot read or write this file. This is bad.
>
> zfs set checksum=off ZDATA
>
> did not help, still cannot read or write the file.
>
> zpool clear -F ZDATA
>
> still cannot r/w the file
>
> Now i am stuck in worst situation that with UFS - i cannot read what's 
> left of this file.
>
> I deleted the file and the scrubbed - error went away. But i lost the 
> file.
>
> So, the question is how to read the file with an error ? I googled for 
> 2 hours and still did not find a solution.
>
>
> Artem
>
With what you did both copies are damaged and non-repairable.

Neither block on the two mirror copies has a correct checksum therefore 
neither copy is good; both were damaged "at the same time".  The system 
knows the file is corrupt and thus the contents are not usable, but it 
has no way to fix it because both copies are damaged with no means of 
correction.

The correct answer to what to do is IMHO what ZFS does.  The only 
solution in such a case is to restore from a backup.

You *definitely* do not want the system to silently let you proceed in 
such an instance because now you will *propagate* the damaged file 
without knowing it was damaged, which will then wind up in your backup 
copies.  Over time even with a robust backup scheme this will eventually 
lead to you losing all "good" copies of the file since the backup media 
will be rotated and ultimately the data will be permanently lost with no 
means of recovery.

Blocking the read forces you to get the good copy off backup media and 
thus prevents that from happening.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20210212/bf7e65e1/attachment.bin>