ZFS Checksum errors

Thu Jun 21 22:52:08 UTC 2012

Hi,

On Thu, Jun 21, 2012 at 2:48 PM,  <rondzierwa at comcast.net> wrote:
>
> ok, i ran a verify on the raid, and it completed, so I believe that, from
> the hardware standpoint, da0 should be a functioning, 12TB disk.
>
> i did a zpool clear and re-ran the scrub, and the results were almost
> identical:
[...]
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         zfsPool     ONLINE       0     0 6.20K
>           da0       ONLINE       0     0 12.5K  24K repaired

This is very likely be a hardware issue, or a driver issue (less
likely, since we have done extensive testing on this RAID card and the
problems are believed to fixed years ago).

There are however a few erratums from AMD that makes me feel quite concerned:

http://support.amd.com/us/Embedded_TechDocs/41322.pdf

Specifically speaking, #264, #298 seems quite serious.  How old is
your motherboard BIOS?  Are you using ECC memory by the way?

> errors: Permanent errors have been detected in the following files:
>
>         zfsPool/raid:<0x9e241>
>         zfsPool/Build:<0x0>
> phoenix#
>
> along with the 6,353 I/O errors, there were over 12,000 checksum mismatch
> errors on the console.
>
>
> The recommendation from ZFS is to restore the file in question.  At this
> point, I would just like to delete the two files.
> how do i do that?
>
> its these kind of antics that make me resistant to the thought of allowing
> ZFS to manage the raid.  it seems to be having problems just managing a big
> file system.  I don't want it to correct anything, or restore anything, just
> let me delete the files that hurt, fix up the free space list so it doesn't
> point outside the bounds of the disk, and get on with life.

Are you *really* sure that these are files?  The second one doesn't
seem to be a file, but rather some metadata.

If hardware issue have been ruled out, what I would do is to copy data
over to a different dataset (e.g. Build.new, then validate the data
copied, then destroy the current Build dataset, rename Build.new to
Build).

> if its finding corrupted files that appear to not have a directory entry
> associated with them (unlinked files), why doesn't it just delete them?
> fsck asks you if you want to delete unlinked files, why doesn't zfs do the
> same, or at least give you the option of deleting bad files when it finds
> them?

Normally, ZFS do tell you which files are corrupted, sometimes it
takes time since your file might be present in multiple snapshots, and
the current set of utilities only gives you one reference for the
file's name, and you may need to remove the file (or the snapshot
containing it), scrub, then remove the newly revealed reference, etc.

Your case seems to be very serious that I really think there are some
metadata corruption, which are serious enough that they are already
beyond fix.  ZFS replicates metadata into different locations, but
that does not prevent it from being corrupted in memory.  In these
situations you will have to use a backup.

> this is causing a lot of down time, and its making linux look very
> attractive in my organization. how do I get this untangled short of
> reformatting and starting over?

Linux does not have comparable end-to-end data validation ability that
ZFS offers.  Use caution if you go that route.

> ron.
>
>
> ________________________________
> From: "Xin LI" <delphij at gmail.com>
> To: rondzierwa at comcast.net
> Cc: "Steven Hartland" <killing at multiplay.co.uk>, freebsd-fs at freebsd.org
> Sent: Wednesday, June 20, 2012 6:56:09 PM
>
> Subject: Re: ZFS Checksum errors
>
> On Wed, Jun 20, 2012 at 1:55 PM,  <rondzierwa at comcast.net> wrote:
>> Steve.
>>
>> well, it got done, and it found another anonymous file with errors . any
>> idea how to get rid of these?
>
> Normally you need to "zpool clear zfsPool", and rerun zpool scrub.  If
> you see these numbers growing again, it's likely that there are some
> other problems with your hardware.  The recommended configuration is
> to use ZFS to manage disks, or at least split your RAID volumes into
> smaller ones by the way, since otherwise the volume is seen as a
> "single disk" to ZFS, making it impossible to repair data errors
> unless you add additional redundancy (zfs set copies=2, etc).
>
>>
>> thanks,
>> ron.
>>
>>
>>
>> phoenix# zpool status -v zfsPool
>> pool: zfsPool
>> state: ONLINE
>> status: One or more devices has experienced an error resulting in data
>> corruption. Applications may be affected.
>> action: Restore the file in question if possible. Otherwise restore the
>> entire pool from backup.
>> see: http://www.sun.com/msg/ZFS-8000-8A
>> scrub: scrub completed after 8h29m with 6276 errors on Wed Jun 20 16:18:01
>> 2012
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> zfsPool ONLINE 0 0 6.17K
>> da0 ONLINE 0 0 13.0K 1.34M repaired
>>
>> errors: Permanent errors have been detected in the following files:
>>
>> zfsPool/raid:<0x9e241>
>> zfsPool/Build:<0x0>
>> phoenix#
>>
>>
>>
>>
>> ----- Original Message -----
>> From: "Steven Hartland" <killing at multiplay.co.uk>
>> To: rondzierwa at comcast.net, freebsd-fs at freebsd.org
>> Sent: Wednesday, June 20, 2012 1:58:20 PM
>> Subject: Re: ZFS Checksum errors
>>
>> ----- Original Message -----
>> From: <rondzierwa at comcast.net>
>> ..
>>
>>> zpool status indicates that a file has errors, but doesn't tell me its
>>> name:
>>>
>>> phoenix# zpool status -v zfsPool
>>> pool: zfsPool
>>> state: ONLINE
>>> status: One or more devices has experienced an error resulting in data
>>> corruption. Applications may be affected.
>>> action: Restore the file in question if possible. Otherwise restore the
>>> entire pool from backup.
>>> see: http://www.sun.com/msg/ZFS-8000-8A
>>> scrub: scrub in progress for 5h27m, 18.71% done, 23h42m to go
>>
>> Try waiting for the scrub to complete and see if its more helpful after
>> that.
>>
>> Regards
>> Steve
>>
>> ================================================
>> This e.mail is private and confidential between Multiplay (UK) Ltd. and
>> the person or entity to whom it is addressed. In the event of misdirection,
>> the recipient is prohibited from using, copying, printing or otherwise
>> disseminating it or any information contained in it.
>>
>> In the event of misdirection, illegible or incomplete transmission please
>> telephone +44 845 868 1337
>> or return the E.mail to postmaster at multiplay.co.uk.
>>
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>
>
>
> --
> Xin LI <delphij at delphij.net> https://www.delphij.net/
> FreeBSD - The Power to Serve! Live free or die

-- 
Xin LI <delphij at delphij.net> https://www.delphij.net/
FreeBSD - The Power to Serve! Live free or die