Raid + zfs performace.

[James R. Van Artsdalen]

Peter Ankerstål wrote:
> I have a question about raid and zfs. I Have a hardware-raid running. A mirror thats the only storage in my zfs pool. Im going to
> add another mirror to the machine and my question is, what is the best option performace-wise?
> Is it to add the other mirror to the same pool or create another separate pool for that mirror? 
> Btw. Today my disk are quite saturated r/w wise.

You should see if your disks are evenly saturated with reads & writes,
or heavily reads with few writes etc.  Whether the new mirror should be
added to the existing pool or placed in a new pool depends on your
unstated goals: do you want to reduce existing I/O saturation or add
storage that does not compete with existing I/O traffic?

If you are concerned about recovery scenarios then it is better to put
the RAID controller in JBOD mode using ZFS for a number of reasons:

1. When you replace a disk ZFS only rebuilds the areas of the disk that
are used.  The RAID controller must rebuild the entire disk, resulting
in extra I/O.

2. If a failure is not in the disk but rather somewhere else - perhaps a
disconnected cable - then ZFS can quickly rebuild the disk when
reconnected, often very quickly.  Last year I accidentally yanked out
the cable to an enclosure with 4 2TB disks (each in a different mirror),
and when I reconnected the enclosure ZFS took about *two seconds* to
rebuild everything.  A RAID controller would have had to rebuild all 8 TB.

3. With ZFS the different disks in a mirror need not be on the same
controller.  For a while I ran one "side" of each mirror on an LSI SAS
controller using the mpt driver and other "side" of each mirror on a
3124 SATA controller using the siis driver.  With hardware RAID you've
generally got the RAID controller and device driver as single points of
failure.

ZFS mirrors can be many-way mirrors.  I have not tested this but it
seems likely ZFS load balances reads in a mirror.  If the I/O load is
heavily slanted towards reads then adding more "ways" to the mirror
might add performance as well as redundancy.