ZFS I/O errors

[Olaf Seibert]

On Mon 30 May 2011 at 03:33:49 -0700, Jeremy Chadwick wrote:
> On Mon, May 30, 2011 at 12:10:51PM +0200, Olaf Seibert wrote:
> I'm not sure why this didn't actually map to a filename on the system
> however.  I've never quite understood what the hexadecimal values shown
> represent (I have ideas but it'd be useful to know what they meant).

The scrub is starting to add some filenames to the list. So far they are
two filenames in snapshots (where current versions of the file have been
modified since then).

> Try running without compression and see if that improves things.

That sounds like a good idea.

My theory so far is that it ran out of memory while compressing, with
incorrect compressed data written to the disk.

> It's important to note that the I/O errors shown happened on "random
> disks" (meaning more than just one device).  What you didn't disclose is
> what the disks are attached to.  "camcontrol devlist -v" would have been
> a good start, followed by any details of controller/driver/etc. you're
> using.  Possibly it's an underlying (silent) storage driver bug.

They are connected to an Areca host adapter:

arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 17 at device 14.0 on pci7
ARECA RAID ADAPTER0: Driver Version 1.20.00.19 2010-11-11 
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 
arcmsr0: [ITHREAD]
da0 at arcmsr0 bus 0 scbus0 target 0 lun 0
da1 at arcmsr0 bus 0 scbus0 target 0 lun 1
da2 at arcmsr0 bus 0 scbus0 target 0 lun 2
da3 at arcmsr0 bus 0 scbus0 target 0 lun 3
da4 at arcmsr0 bus 0 scbus0 target 0 lun 4
da5 at arcmsr0 bus 0 scbus0 target 0 lun 5
pass6 at arcmsr0 bus 0 scbus0 target 16 lun 0

camcontrol devlist -v:

scbus0 on arcmsr0 bus 0:
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 0 (pass0,da0)
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 1 (pass1,da1)
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 2 (pass2,da2)
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 3 (pass3,da3)
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 4 (pass4,da4)
<Seagate ST31500341AS R001>        at scbus0 target 0 lun 5 (pass5,da5)
<Areca RAID controller R001>       at scbus0 target 16 lun 0 (pass6)
<>                                 at scbus0 target -1 lun -1 ()
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun -1 (xpt0)

> Finally, and leaving the most important point for last: you didn't state
> what FreeBSD version you're using, and ceased to provide uname -a output
> (to see kernel build date, etc.).  It matters greatly.

Sorry. Here is it:

FreeBSD fourquid.cs.ru.nl 8.2-RELEASE FreeBSD 8.2-RELEASE #3: Tue Apr 19
13:02:11 CEST 2011 root at fourquid.cs.ru.nl:/usr/obj/usr/src/sys/FOURQUID  amd64

The kernel is built from unmodified 8.2-RELEASE sources, but I adapted
the configuration a bit to throw out irrelevant hardware support (this
does make a small difference):

-r-xr-xr-x  1 root  wheel  12612347 Apr 19 12:27 /boot/GENERIC/kernel*
-r-xr-xr-x  1 root  wheel   7031915 Apr 19 13:02 /boot/kernel/kernel*

The real original reason to make a custom kernel was to add the SCSI
passthrough device (needed by smartctl) , but istr that that is included
in GENERIC now.

Thanks so far,
-Olaf.
-- 
Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));