seeing data corruption with zfs trim functionality

[Jeremy Chadwick]

On Mon, Apr 29, 2013 at 03:51:43AM -0700, Jeremy Chadwick wrote:
> On Mon, Apr 29, 2013 at 09:22:06AM +0100, Steven Hartland wrote:
> > ----- Original Message ----- From: "Ajit Jain"
> > <ajit.jain at cloudbyte.com>
> > 
> > >I am running zfs with trim functionality (ported from head). Seeing data
> > >corruption when running iotest* with multiple threads (never saw data
> > >corruption with single thread).
> > >
> > >The patches merged to add trim support are as follows:
> > >1. 240868  (zfs trim patch)
> > >2. 230053 and 245252 (block device driver trim support)
> > >3. 239655 (fix an issue in patch 230053)
> > >
> > >I am "NOT" seeing data corruption in the following cases:
> > >1. Running iotest with single thread (Trim is enabled at entire io stack).
> > >2. Trim is enabled at zfs layer but disable at driver layer i.e. delete
> > >method is set to NONE (even with multiple threads).
> > >
> > >
> > >Since patch 240868 alone was not working as I pulled in additional zfs trim
> > >patches 244155, 244187, 244188, 248572 (however I am not using separate
> > >L2arc device), 248573, 248574, 248575 and 248576. Still I am seeing the
> > >same issue.
> > >
> > >Issue: After some time running with multiple thread write system call
> > >return sometimes with EIO or 122 (checksum error) error code.
> > >
> > >I looked at GEOM code a bit I think it already has the trim (DELETE)
> > >command support. Still I am doubtful if I have pulled in all required
> > >patches in the entire I/O stack.
> > >
> > >I am using a LSI SAS HBA card to connect to the SSD, firmware seems to
> > >claim the support for trim.
> > >
> > >*iotest: non standard freebsd FreeBSD utility, which creates files and does
> > >I/O on the files and can be invoked in single/multithread mode to do the
> > >I/O.
> > 
> > What version are you porting the changes to?
> > 
> > What SSD are you using?
> > 
> > What LSI controller are you using?
> 
> I'd also like to see "zpool status" (for every pool that involves this
> SSD) and "gpart show" against the disk itself.

Also, the controller involved is an mps(4) controller, which to the
underlying subsystem is SCSI.

TRIM (as it's called; the actual name per ATA standard is DATA SET
MANAGEMENT) is purely an ATA specification thing.

The SCSI equivalent is called UNMAP, or alternately WRITE SAME.

(This is not the case here, but just mentioning it: even in the cases of
SCSI controllers that have SATA disks attached, the OS ends up
submitting UNMAP/WRITE SAME, which the controller has to convert into
the relevant ATA DATA SET MANAGEMENT command.  If the controller
firmware screws this up there not much we can do about it)

References for FreeBSD:

http://lists.freebsd.org/pipermail/freebsd-current/2011-December/030714.html

PLEASE READ THE LAST PARAGRAPH OF THAT POST.

This brings into question whether or not relevant subsystems (ranging
from mps(4) to GEOM(4) to CAM(4)) actually have working UNMAP/WRITE SAME
or not, or if the controller itself is doing something stupid with them.

I'm CC'ing mav@ for what should be obvious reasons.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |