ZFS RaidZ2 with 24 drives?

[James R. Van Artsdalen]

Matt Simerson wrote:

>> In general, it's always best to let ZFS handle the raid stuff and not
>> use the hardware raid settings.
>
> I'd like to see some evidence to back that statement up. The only time
> I've seen better ZFS performance numbers than what I'm getting with
> FreeBSD 8 ZFS + Areca RAID6 is when I tested OpenSolaris with them
> Marvell SATA controllers. But that was in August of 2008, and ZFS on
> FreeBSD performs much better now. Some updated benchmarks would be
> welcome.

You're fixating on throughput.  Moreover, Marvell is the wrong
controller for FreeBSD 8: use the SIIS driver with a controller based on
the Silicon Image 3124.

Doing redundancy in ZFS rather than the controller has two obvious
advantages:

1. ZFS has access the filesystem checksums, signatures and txtags
whereas RAID controllers do not.  ZFS can always tell when data recovery
is needed, which copy of the data is correct in a mirror, and which disk
in a striped-parity set needs reconstruction.  And ZFS can tell if no
recovery was possible.  A RAID controller can't always do this right.

2. More important, by letting ZFS handle RAID you can spread the risk
across controllers and drivers as well as disks.

For example, my home pool is an array of MIRRORs, each MIRROR carefully
arranged so that if one component is a Seagate disk, the other is *not*
a Seagate.  For every mirror the two disks are in different enclosures
(using different power supplies), each enclosure set connected to a
different host controller.  And I'm about to put half of the disks in
SAS enclosures connected to LSI 3801e controllers using the MPT driver:
once done I'll be fully redundant with respect to disk drives,
enclosures & disk power supplies, cables, controllers and drivers.