Marvell 88SX6081 timeouts, particularly when running 'zfs scrub' with regular I/O

[Michael DeMan]

Hi Jeremy,

Info is:

FreeBSD freenas.n.bli.openaccess.org 8.2-RELEASE-p2 FreeBSD 8.2-RELEASE-p2 #0: Tue Jul 12 12:11:37 PDT 2011     jpaetzel at servant.iXsystems.com:/b/home/jpaetzel/sf_freenas_build/obj.amd64/b/home/jpaetzel/sf_freenas_build/FreeBSD/src/sys/FREENAS.amd64  amd64


Also, the problem I am seeing with the mvs driver seems suspiciously close to a problem with ahci driver that is patched in 9-CURRENT apparently by disabling NCQ?

http://forums.freebsd.org/showthread.php?t=20412

The problem only occurs under high I/O - basically I can force it to occur by kicking off a zfs scrub while there is also about 300 IOPS of NFS traffic at the same time.



On Jul 20, 2011, at 9:40 PM, Jeremy Chadwick wrote:

> On Wed, Jul 20, 2011 at 08:59:32PM -0700, Michael DeMan wrote:
>> I've found a few posts around about this, but nothing conclusive.
>> 
>> We have been getting hit on this with two...
>> mvs0: <Marvell 88SX6081 SATA controller> port 0x9400-0x94ff mem 0xfc400000-0xfc4fffff irq 28 at device 1.0 on pci1
>> mvs1: <Marvell 88SX6081 SATA controller> port 0x9800-0x98ff mem 0xfc500000-0xfc5fffff irq 29 at device 3.0 on pci1
>> ...controllers.
>> 
>> I went through and did a few things (an older Opteron 285 box) and disabled super-pages and permutations on other device.hints, loader.conf and live sysctl settings -  all to no avail.
>> 
>> I also found a few things via Google about being to patch from 9-CURRENT, but the idea with this box was to be able to re-purpose some older equipment for proof of concept using FreeNAS8.
>> 
>> It is possible for me to build a version of that with the patches, etc - but I figured it would be better to post to the list first and gather feedback since this is pretty old/clunky hardware and newer patches may or may not solve the problem.
>> 
>> Thanks,
>> 
>> - mike deman
>> 
>> 
>> 
>> Jul 19 16:46:41 freenas kernel: mvsch11: Timeout on slot 0
>> Jul 19 16:46:41 freenas kernel: mvsch11: iec 02000000 sstat 00000123 serr 00000000 edma_s 00001023 dma_c 00000000 dma_s 00000000 rs 00000201 status 40
>> Jul 19 16:46:41 freenas kernel: mvsch11:  ... waiting for slots 00000200
>> Jul 19 16:46:43 freenas kernel: mvsch4: Timeout on slot 4
>> Jul 19 16:46:43 freenas kernel: mvsch4: iec 02000000 sstat 00000123 serr 00000000 edma_s 00001022 dma_c 00000000 dma_s 00000000 rs 00000010 status 40
>> Jul 19 16:46:45 freenas kernel: mvsch1: Timeout on slot 0
>> Jul 19 16:46:45 freenas kernel: mvsch1: iec 02000000 sstat 00000123 serr 00000000 edma_s 00001101 dma_c 00000000 dma_s 00000000 rs 00000001 status 40
>> Jul 19 16:46:46 freenas kernel: mvsch11: Timeout on slot 9
>> Jul 19 16:46:46 freenas kernel: mvsch11: iec 02000000 sstat 00000123 serr 00000000 edma_s 00001023 dma_c 00000000 dma_s 00000000 rs 00000201 status 40
>> Jul 19 16:46:47 freenas root: ZFS: checksum mismatch, zpool=zpmir1 path=/dev/gpt/ada12 offset=52341838336 size=131072
> 
> You didn't disclose what FreeBSD version you're running and what your
> kernel/world build date is.  It matters greatly since it will then give
> us some idea what exact source revision you're using for the mvs(4)
> driver.  "uname -a" output is sufficient.
> 
> Thanks.
> 
> -- 
> | Jeremy Chadwick                                jdc at parodius.com |
> | Parodius Networking                       http://www.parodius.com/ |
> | UNIX Systems Administrator                   Mountain View, CA, US |
> | Making life hard for others since 1977.               PGP 4BD6C0CB |
>