Filesystem, RAID Question

[Jeremy Chadwick]

On Thu, Oct 30, 2008 at 08:41:59PM -0500, Rich Winkel wrote:
> On Thu, Oct 30, 2008 at 04:38:49PM -0700, Jeremy Chadwick wrote:
> > On Thu, Oct 30, 2008 at 06:12:07PM -0500, Rich Winkel wrote:
> > > Doesn't hw.ata.wc affect only card-level caching?
> > 
> > hw.ata.wc causes the ata(4) subsystem to disable write caching on all
> > disks attached to the subsystem.  It does not disable card features.
> 
> I mean, the individual disks are invisible to the OS unless the
> card's driver (and the card itself) specifically supports it.

Correct.

With regards to ATA: ata(4) has support for pass-through on some RAID
cards, such as Promise.  FreeBSD will see the individual disks (e.g.
ad4, ad6, etc.) as well as the array (e.g. ar0).

With regards to SCSI: pass(4) provides this capability.  I don't think
in the case of SCSI that the disks will appear in FreeBSD (e.g. da0)
though.  Instead, pass(4) can be used to query individual disks on an
array, e.g. smartctl's -d flag (e.g. -d 3ware, -d marvell, etc.).

In both cases (ATA and SCSI), the card itself has to support
pass-through, *and* the FreeBSD driver has to have code to allow for
such, otherwise no go.

> > There's also the below PR, which extends atacontrol to permit disabling
> > and enabling write caching on a per-disk basis.
> > 
> > http://www.freebsd.org/cgi/query-pr.cgi?pr=127717
> 
> But not on disks which are behind hardware raid cards, correct?

Correct.  For FreeBSD to be able to disable write caching on disks
behind a RAID controller, one of two things is needed:

1) Pass-through support (see above),
2) A native CLI program that interfaces with the card directly (usually
   written by the vendor).

Sadly, #2 appears to be the most common choice when a RAID card is used.
I say "sadly" because many vendors do not support FreeBSD, and only
offer Linux CLI programs -- requiring an administrator to install Linux
emulation, Linux libraries, etc., and *hoping* that it works.

If the neither of the above options are available, then your only choice
is to go into the RAID card's BIOS and disable write caching in there,
assuming the option exists (on many cards it does).

> > What gives you the impression that during a power outage your data is
> > going to be intact?
> 
> One of the main functions of softupdates is to order disk updates in such
> a way that the fs organizational integrity is maintained at all times.

And we've recently found that this is simply not the case.  The benefits
of SU are applicable to very specific environments; desktop PCs are the
main ones, offering great performance improvements there.

But there's a known problem with the "background fsck" feature of
FreeBSD, which is only applicable to filesystems which use SU; sometimes
fsck does not correct all errors, causing the filesystem to be marked
clean, even though there are actual problems with it.  There's a thread
from about a month ago discussing why background_fsck="no" is highly
recommended when using SU.

> Of course this doesn't protect against actual sector corruption, but if
> the disk is between writes at the time it loses power, the fs structure
> is supposed to still be internally consistent.  At least that's my
> understanding of it.

Yep, that's how I understand it as well.  But this is a different topic
than what we were discussing 2-3 replies ago, talking about how a RAID
controller with cache + BBU is sufficient enough to guarantee data
integrity even when power is lost -- that's incorrect.

Back to write caching:

Disabling write caching on disks does not guarantee integrity in the
case of such failures either -- on the other hand, by disabling an extra
layer of caching, you've essentially diminished the risk by only a
nominal amount.

I personally believe disabling write caching is not a plausible option
for users; the performance hit is major (I have done tests) -- write
speeds drop to 12% of total capability.  Meaning: 70MB/sec with WC
enabled, 8.4MB/sec with WC disabled.  This is *without* a controller
that does caching of any kind.

Essentially you can use this to benchmark which is faster: write
caching disabled on disks + caching enabled on a controller, or
write caching enabled on disks + caching disabled on a controller.
It would be interesting to see benchmark comparisons of different
controllers.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |