FreeBSD 10, ServeRAID M5210e, syspd corruption

[Borja Marcos]

(crossposting to -Stable just in case)

Hello,

I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID M5210e and 23 SSD disks.

uname -a
FreeBSD hostname 10.0-RELEASE FreeBSD 10.0-RELEASE #1: Fri Feb 14 09:35:12 CET 2014     toor@
hostname:/usr/obj/usr/src/sys/GENERIC  amd64


The server has a SAS backplane and a controller recognized by the mfi driver.

mfi0 Adapter:
    Product Name: ServeRAID M5210e
   Serial Number: 3CJ0SG          
        Firmware: 24.0.2-0013
     RAID Levels: JBOD, RAID0, RAID1, RAID10
  Battery Backup: not present
           NVRAM: 32K
  Onboard Memory: 0M
  Minimum Stripe: 64K
  Maximum Stripe: 64K


As I am intending to use ZFS, I need direct access to the disks, no need for fancy RAID features.

I have seen that the newest cards support a so-called "syspd" mode that gives direct acess to the disks.
However, in this configuration, syspd consistently corrupts data on the disks.

I have done tests with three models of disks:

- Samsung SSD 840 BB0Q (1 TB)
- OCZ-VERTEX4 1.5  (512 GB)
- SEAGATE ST9146803SS FS03 (136 GB)

In the three cases there is data corruption. Using FFS on the disks results in a panic if I run a benchmark, for example, bonnie++.
Using ZFS (I've been creating one disk pools to test) I don't get panics but the data  is consistently corrupted. The writes work, but whenever
there is read activity (either bonnie++ reaching the "rewrite" phase, or a ZFS scrub), ZFS detects data corruption.

Trying the über neat hw.mfi.allow_cam_disk_passthrough (which is great, because ZFS can detect the SSDs and issue TRIM commands)
I get the same result: data corruption.

However, I have tried to create a one-disk raid0 volume, and in that case it works like a charm, no corruption at all, so I can safely
assume that this is not a defective backplane, expander or cabling. 

So:

mfisyspd -> CORRUPT
da -> CORRUPT
mfid -> NOT CORRUPT

Any ideas? Could be a driver error or a firmware problem, I am clueless for now.

Anything I can test? The machine is not in production, I can try patches or whatever.




Thanks!!






Borja.