ZFS errors with two HEX numbers.

From: Zaphod Beeblebrox <zbeeble_at_gmail.com>
Date: Sat, 25 Sep 2021 15:28:39 -0400
I've got a ZFS pool --- 3 x 8 x 4T disks in a RAIDZ2 configuration.  Every
few months I scrub them.  This time, curiously, it came up with an error.
All the disks in a vdev simultaneously give an error when this is
scanned.... thusly:

          raidz2-3      ONLINE       0     0     0
            gpt/v1-f0   ONLINE       0     0     1
            gpt/v1-f1   ONLINE       0     0     1
            gpt/v1-f2   ONLINE       0     0     1
            gpt/v1-f3   ONLINE       0     0     1
            gpt/v1-f4   ONLINE       0     0     1
            gpt/v1-f5   ONLINE       0     0     1
            gpt/v1-f6a  ONLINE       0     0     1
            gpt/v1-f7b  ONLINE       0     0     1

... and if I add -v to the status, I see the following right now:

errors: Permanent errors have been detected in the following files:
        <0x57b2>:<0x73ab46>

Now... since that happened, I have scrubbed twice more.  The first number
changes each time.  Last scrub it was: <0x4223>:<0x73ab46>.  Note the last
number is not changing.

I have done some googling and found that this might be something that got
corrupted on delete or somesuch.  My OCD would like to know what 0x4223
maps to (or now 0x57b2) s.t. I could find it and remove it.  I _think_ this
is a snapshot reference?

Anyways... any help in tracking this down would be appreciated.

For those who've read this far, here's the full zfs scrub -v output... If
you're all intensely curious, the last letter following the drive number
tracks the number of times a drive has been replaced.

  pool: vr1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 1 days 05:01:08 with 1 errors on Fri Sep 24
23:23:34 2021
config:

        NAME            STATE     READ WRITE CKSUM
        vr1             ONLINE       0     0     0
          raidz2-0      ONLINE       0     0     0
            gpt/v1-d0   ONLINE       0     0     0
            gpt/v1-d1   ONLINE       0     0     0
            gpt/v1-d2   ONLINE       0     0     0
            gpt/v1-d3   ONLINE       0     0     0
            gpt/v1-d4a  ONLINE       0     0     0
            gpt/v1-d5   ONLINE       0     0     0
            gpt/v1-d6a  ONLINE       0     0     0
            gpt/v1-d7a  ONLINE       0     0     0
          raidz2-2      ONLINE       0     0     0
            gpt/v1-e0c  ONLINE       0     0     0
            gpt/v1-e1c  ONLINE       0     0     0
            gpt/v1-e2b  ONLINE       0     0     0
            gpt/v1-e3b  ONLINE       0     0     0
            gpt/v1-e4b  ONLINE       0     0     0
            gpt/v1-e5a  ONLINE       0     0     0
            gpt/v1-e6a  ONLINE       0     0     0
            gpt/v1-e7d  ONLINE       0     0     0
          raidz2-3      ONLINE       0     0     0
            gpt/v1-f0   ONLINE       0     0     1
            gpt/v1-f1   ONLINE       0     0     1
            gpt/v1-f2   ONLINE       0     0     1
            gpt/v1-f3   ONLINE       0     0     1
            gpt/v1-f4   ONLINE       0     0     1
            gpt/v1-f5   ONLINE       0     0     1
            gpt/v1-f6a  ONLINE       0     0     1
            gpt/v1-f7b  ONLINE       0     0     1
        logs
          gpt/vr1log    ONLINE       0     0     0
        cache
          gpt/vr1cache  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x57b2>:<0x73ab46>
Received on Sat Sep 25 2021 - 19:28:39 UTC

Original text of this message