Help needed! ZFS I/O error recovery?

[Solon Lutz]

Hi erverybody,

I'm faced with a 10TB ZFS pool on a 12TB RAID6 Areca controller.
And yes, I know, you shouldn't put a zpool on a RAID-device... =(

Due to problems with a sata-cable, some days ago the raid-controller
started to produce long timeouts to recover the resulting read errors.

The cable was replaced, a parity check was run on the RAID-Volume and
showed no errors, the zfs scrub however showed some 'defective' files.
After copying these files with 'dd -conv=noerror...' and comparing them
to the originals, they were error-free.

Yesterday however, three more defective cables forced the controller
to take the RAID6 volume offline. Now all cables were replaced and a parity
check was run on the RAID-Volume -> data integrity OK.

But now ZFS refuses to mount all volumes:

Solaris: WARNING: can't process intent log for temp/space1
Solaris: WARNING: can't process intent log for temp/space2
Solaris: WARNING: can't process intent log for temp/space3
Solaris: WARNING: can't process intent log for temp/space4

A scrub revealed to following:

errors: Permanent errors have been detected in the following files:

        temp:<0x0>
        temp/space1:<0x0>
        temp/space2:<0x0>
        temp/space3:<0x0>
        temp/space4:<0x0>


I tried to switch off checksums for this pool, but that didn't help in any
way. I also mounted the pool by hand and was faced with with 'empty' volumes
and 'I/O errors' when trying to list their contents...

Any suggestions? I'm offering some self-made blackberry jam and raspberry brandy
to the person who can help to restore or backup the data.

Tech specs:

FreeBSD 7.2-STABLE #21: Tue May  5 18:44:10 CEST 2009 (AMD64)
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1280-VOL#00 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: Command Queueing Enabled
da0: 10490414MB (21484367872 512 byte sectors: 255H 63S/T 1337340C)
ZFS filesystem version 6
ZFS storage pool version 6




Best regards,

Solon