ZFS option vfs.zfs.cache_flush_disable=1

[Kevin Day]

On Aug 3, 2013, at 5:38 PM, Karl Denninger <karl at denninger.net> wrote:

> Hi folks;
> 
> I'm trying to run down a rather odd bit of ugly interaction between an
> ARECA 1680-IX adapter (in JBOD mode) and ZFS.
> 
> The adapter has both a fair bit of cache memory on it and a BBU unit. 
> If I turn off write-back caching ZFS performance goes straight into the
> toilet.  With it on, however, it appears that cache flushes are being
> honored and, what's worse, when they come down the pipe they force all
> further I/O to the adapter to cease until the ENTIRE cache is flushed.
> 
> This can and occasionally does lead to degenerate cases where very
> severe performance problems ensue.


Hey, Karl!

We had similar problems with the 3ware 9650SE cards. Cache flush operations would just kill performance to the entire array, even if ZFS was only being used on one volume on that card.

We've got vfs.zfs.cache_flush_disable=1 set on piles of servers, and so far we've only had one bit of weird ZFS corruption that I'm not even sure was related to that setting. While this isn't answering your question, I can say that having that flag enabled isn't causing a ton of corruption for us even with random hard reboots happening sometimes.

I will say though that the LSI 2008 based cards are substantially faster and less problematic for ZFS than anything else we've used. The "drives getting reordered" thing seems to only be happening on a few systems with a specific SATA/SAS backplane that doesn't appear to deterministically initialize things. gpart labels make that far less of a problem though, if you can use them. Failing that, it is possible to use "camcontrol inquiry daXX" to see the serial number of the drives, which you can then label the fronts of the drives with. Not as pretty, but being able to have a naked JBOD interface without controllers treating every drive as it's own 1-drive RAID0 volume is so much easier that I'll put up with a lot to have it.

-- Kevin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4891 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130803/883fe8ce/attachment.bin>