ZFS i/o errors - which disk is the problem?

[Scott Long]

Bernd Walter wrote:
> On Mon, Jan 07, 2008 at 10:44:13AM +0800, Tz-Huan Huang wrote:
>> 2008/1/4, Brooks Davis <brooks at freebsd.org>:
>>> We've definitely seen cases where hardware changes fixed ZFS checksum errors.
>>> In once case, a firmware upgrade on the raid controller fixed it.  In another
>>> case, we'd been connecting to an external array with a SCSI card that didn't
>>> have a PCI bracket and the errors went away when the replacement one arrived
>>> and was installed.  The fact that there were significant errors caught by ZFS
>>> was quite disturbing since we wouldn't have found them with UFS.
>> Hi,
>>
>> We have a nfs server using zfs with the similar problem.
>> The box is i386 7.0-PRERELEASE with 3G ram:
>>
>> # uname -a
>> FreeBSD cml3 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #2:
>> Sat Jan  5 14:42:41 CST 2008 root at cml3:/usr/obj/usr/src/sys/CML2  i386
>>
>> The zfs pool contains 3 raids now:
>>
>> 2007-11-20.11:49:17 zpool create pool /dev/label/proware263
>> 2007-11-20.11:53:31 zfs create pool/project
>> ... (zfs create other filesystems) ...
>> 2007-11-20.11:54:32 zfs set atime=off pool
>> 2007-12-08.22:59:15 zpool add pool /dev/da0
>> 2008-01-05.21:20:03 zpool add pool /dev/label/proware262
>>
>> After a power loss yesterday, the zfs status shows
>>
>> # zpool status -v
>>   pool: pool
>>  state: ONLINE
>> status: One or more devices has experienced an error resulting in data
>>         corruption.  Applications may be affected.
>> action: Restore the file in question if possible.  Otherwise restore the
>>         entire pool from backup.
>>    see: http://www.sun.com/msg/ZFS-8000-8A
>>  scrub: scrub completed with 231 errors on Mon Jan  7 08:05:35 2008
>> config:
>>
>>         NAME                STATE     READ WRITE CKSUM
>>         pool                ONLINE       0     0   516
>>           label/proware263  ONLINE       0     0   231
>>           da0               ONLINE       0     0   285
>>           label/proware262  ONLINE       0     0     0
>>
>> errors: Permanent errors have been detected in the following files:
>>
>>         /system/database/mysql/flickr_geo/flickr_raw_tag.MYI
>>         pool/project:<0x0>
>>         pool/home/master/96:<0xbf36>
>>
>> The main problem is that we cannot mount pool/project any more:
>>
>> # zfs mount pool/project
>> cannot mount 'pool/project': Input/output error
>> # grep ZFS /var/log/messages
>> Jan  7 10:08:35 cml3 root: ZFS: zpool I/O failure, zpool=pool error=86
>> (repeat many times)
>>
>> There are many data in pool/project, probably 3.24T. zdb shows
>>
>> # zdb pool
>> ...
>> Dataset pool/project [ZPL], ID 33, cr_txg 57, 3.24T, 22267231 objects
>> ...
>>
>> (zdb is still running now, we can provide the output if helpful)
>>
>> Is there any way to recover any data from pool/project?
> 
> The data is corrupted by controller and/or disk subsystem.
> You have no other data sources for the broken data, so it is lost.
> The only garantied way is to get it back from backup.
> Maybe older snapshots/clones are still readable - I don't know.
> Nevertheless data is corrupted and that's the purpose for alternative
> data sources such as raidz/mirror and at last backup.
> You shouldn't have ignored those errors at first, because you are
> running with faulty hardware.
> Without ZFS checksumming the system would just process the broken
> data with unpredictable results.
> If all those errors are fresh then you likely used a broken RAID
> controller below ZFS, which silently corrupted syncronity and then
> blow when disk state changed.
> Unfortunately many RAID controllers are broken and therefor useless.
> 

Huh?  Could you be any more vague?  Which controllers are broken?  Have 
you contacted anyone about the breakage?  Can you describe the breakage?
I call bullshit, pure and simple.

Scott