Another ZFS kernel panic on same block on every drive in raidz

[Mark Powell]

Hi,
   I am testing a 3 drive raidz1 array which has been built with 3 new WD 
500GB SATA drives /dev/ad1[468], bought from 2 different sources.
   I am being told that a dma error is occuring on the same block on all 3 
drives at the same time:

Aug 30 18:13:15 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:15 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:15 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: FAILURE - WRITE_DMA timed out LBA=268435340
Aug 30 18:13:46 echo kernel: ad14: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad18: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:46 echo kernel: ad16: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=268435340
Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad14s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad16s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:25 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad18s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad18s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad14s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path=/dev/ad16s2 offset=132076011520 size=65536 error=5
Aug 30 18:13:41 echo root: ZFS: vdev I/O failure, zpool=pool path= offset=396215451648 size=131072 error=5

And then the kernel panics:

panic: ZFS: I/O failure (write on <unknown> off 0: zio 0xffffff0013b0d000 
[L0 ZFS plain file] 20000L/20000P DVA[0]=<5:5c40480000:30000> fletcher2 
uncompressed LE contiguous birth=20167 fill=1 cksum=cfcfcfcfcfcfce00:cfcfcfcfcfcfce00:8a8a8a8a8a56e700:8a8a8a8a8a56e
cpuid = 0

   I think I saw someone else have a similar problem to this. There were 
told their hardware was probably flakey on to look for errors with geli.
   Just performing a scrub now to see what happens.
   Let me know if you need any further info.
   Cheers.

-- 
Mark Powell - UNIX System Administrator - The University of Salford
Information Services Division, Clifford Whitworth Building,
Salford University, Manchester, M5 4WT, UK.
Tel: +44 161 295 4837  Fax: +44 161 295 5888  www.pgp.com for PGP key