seeing data corruption with zfs trim functionality

[Steven Hartland]

----- Original Message ----- 
From: "Ajit Jain" <ajit.jain at cloudbyte.com>
To: "Steven Hartland" <killing at multiplay.co.uk>
Cc: "freebsd-fs" <freebsd-fs at freebsd.org>
Sent: Wednesday, May 15, 2013 4:18 PM
Subject: Re: seeing data corruption with zfs trim functionality


> Hi Steven,
> 
> Thanks for the update.
> It is surprising that there is no/less disk activity as the command is
> correct. May be we need to enable
> sync=always on the zfs dataset.
> 
> I will try again the once I update the cam code base.
> 
> regards,
> ajit
> 
> 
> On Wed, May 15, 2013 at 7:37 PM, Steven Hartland <killing at multiplay.co.uk>wrote:
> 
>> Unless you have the latest CAM patches, which is in current, you wont be
>> doing TRIM on SATA disk connected to an LSI controller.
>>
>> I've just tested using the following cmd under 8.3 with MFC'ed changes from
>> current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine
>> with Intel SSD and LSI controller.
>> ./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/
>>
>> I did however notice that your test is hardly doing any disk access apart
>> from when its "Initializing test file....", instead it seems to be CPU
>> bound,
>> so not sure if there's a problem with the iotest code or with my command
>> line args?
>>
>> Given this my current thinking is either:
>> 1. There's a problem with your patches 2. There's a bug in the FW of the
>> Seagate disk
>> 3. There's a problem with the UNMAP code which is being trigged by your
>> disk
>> only.
>>
>> I think #3 is quite unlikely.
>>
>> If you could install a recent version of current and test with that it
>> should rule out #1, leaving #2.

After initially seeing not issues, our overnight monitoring started moaning
big time on the test box. So we checked and there was zpool corruption as well
as a missing boot loader and a corrupt GPT, so I believe we have reproduced
your issue.

After recovering the machine I created 3 pools on 3 different disks each
running a different delete_method.

We then re-ran the tests which resulted in the pool running with delete_method
WS16 being so broken it had suspended IO. A reboot resulted in it once again
reporting no partition table via gpart.

A third test run again produced a corrupt pool for WS16.

I've conducted a preliminary review of the CAM WS16 code path along with SBC-3
spec which didn't identify any obvious issues.

Given we're both using LSI 2008 based controllers it could be FW issue specific
to WS16 but that's just speculation atm, so I'll continue to investigate.

If you could re-test you end without using WS16 to see if you can reproduce the
problem with either UNMAP or ATA_TRIM that would be a very useful data point.

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.