ZFS or UFS for 4TB hardware RAID6?

[Maxim Khitrov]

On Mon, Jul 13, 2009 at 2:13 PM, Richard Mahlerwein<mahlerrd at yahoo.com> wrote:
>
> --- On Mon, 7/13/09, Maxim Khitrov <mkhitrov at gmail.com> wrote:
>
>> From: Maxim Khitrov <mkhitrov at gmail.com>
>> Subject: Re: ZFS or UFS for 4TB hardware RAID6?
>> To: mahlerrd at yahoo.com
>> Cc: "Free BSD Questions list" <freebsd-questions at freebsd.org>
>> Date: Monday, July 13, 2009, 2:02 PM
>> On Mon, Jul 13, 2009 at 1:46 PM,
>> Richard Mahlerwein<mahlerrd at yahoo.com>
>> wrote:
>> >>
>> >> Your mileage may vary, but...
>> >>
>> >> I would investigate either using more spindles if
>> you want
>> >> to stick to RAID6, or perhaps using another RAID
>> level if
>> >> you will be with 4 drives for a while.  The
>> reasoning
>> >> is that there's an overhead with RAID 6 - parity
>> blocks are
>> >> written to 2 disks, so in a 4 drive combination
>> you have 2
>> >> drives with data and 2 with parity.
>> >>
>> >> With 4 drives, you could get much, much higher
>> performance
>> >> out of RAID10 (which is alternatively called
>> RAID0+1 or
>> >> RAID1+0 depending on the manufacturer and on how
>> accurate
>> >> they wish to be, and on how they actually
>> implemented it,
>> >> too). This would also mean 2 usable drives, as
>> well, so
>> >> you'd have the same space available in RAID10 as
>> your
>> >> proposed RAID6.
>> >>
>> >> I would confirm you can, on the fly, convert from
>> RAID10 to
>> >> RAID6 after you add more drives.  If you can not,
>> then
>> >> by all means stick with RAID6 now!
>> >>
>> >> With 4 1 TB drives (for simpler examples)
>> >> RAID5 = 3 TB available, 1 TB worth used in
>> "parity".
>> >> Fast reads, slow writes.
>> >> RAID6 = 2 TB available, 2 TB worth used in
>> "parity".
>> >> Moderately fast reads, slow writes.
>> >> RAID10 = 2 TB available, 2TB in duplicate copies
>> (easier
>> >> work than parity calculations).  Very fast
>> reads,
>> >> moderately fast writes.
>> >>
>> >> When you switch to, say, 8 drives, the numbers
>> start to
>> >> change a bit.
>> >> RAID5 = 7TB available, 1 lost.
>> >> RAID6 = 6TB available, 2 lost.
>> >> RAID10 = 4TB available, 4 lost.
>> >>
>> >
>> > Sorry, consider myself chastised for having missed the
>> "Security is more important than performance" bit. I tend
>> toward solutions that show the most value, and with 4
>> drives, it seems that I'd stick with the same "data
>> security" only pick up the free speed of RAID10.  Change
>> when you get to 6 or more drives, if necessary.
>> >
>> > For data security, I can't answer for the UFS2 vs.
>> ZFS.  For hardware setup, let me amend everything I said
>> above with the following:
>> >
>> > Since you are seriously focusing on data integrity,
>> ignore everything I said but make sure you have good
>> backups!  :)
>> >
>> > Sorry,
>> > -Rich
>>
>> No problem :) I've been doing some reading since I posted
>> this
>> question and it turns out that the controller will actually
>> not allow
>> me to create a RAID6 array using only 4 drives. 3ware
>> followed the
>> same reasoning as you; with 4 drives use RAID10.
>>
>> I know that you can migrate from one to the other when a
>> 5th disk is
>> added, but RAID10 can only handle 2 failed drives if they
>> are from
>> separate RAID1 groups. In this way, it is just slightly
>> less resilient
>> to failure than RAID6. With this new information, I think I
>> may as
>> well get one more 2TB drive and start with 6TB of RAID6
>> space. This
>> will be less of a headache later on.
>>
>> - Max
>
> Just as a question: how ARE you planning on backing this beast up?  While I don't want to sound like a worry-wort, I have had odd things happen at the worst of times.  RAID cards fail, power supplies let out the magic smoke, users delete items they really want back... *sigh*

Rsync over ssh to another server. Most of the data stored will never
change after the first upload. A daily rsync run will transfer one or
two gigs at the most. History is not required for the same reason;
this is an append-only storage for the most part. A backup for the
previous day is all that is required, but I will keep a weekly backup
as well until I start running out of space.

> A bit of reading shows that ZFS, if it's stable enough, has some really great features that would be nice on such a large pile o' drives.
>
> See http://wiki.freebsd.org/ZFSQuickStartGuide
>
> I guess the last question I'll ask (as any more may uncover my ignorance) is if you need to use hardware RAID at all?  It seems both UFS2 and ZFS can do software RAID which seems to be quite reasonable with respect to performance and in many ways seems to be more robust since it is a bit more portable (no specialized hardware).

I've thought about this one a lot. In my case, the hard drives are in
a separate enclosure from the server and the two had to be connected
via SAS cables. The 9690SA-8E card was the best choice I could find
for accessing an external SAS enclosure with support for 8 drives.

I could configure it in JBOD mode and then use software to create a
RAID array. In fact, I will likely do this to compare performance of a
hardware vs. software RAID5 solution. The ZFS RAID-Z option does not
appeal to me, because the read performance does not benefit from
additional drives, and I don't think RAID6 is available in software.
For those reasons I'm leaning toward a hardware implementation.

If I go the hardware route, I'll try to purchase a backup controller
in a year or two. :)

> There are others who may respond with better information on that front.  I've been a strong proponent of hardware RAID, but have recently begun to realize many of the reasons for that are only of limited validity now.

Agreed, and many simple RAID setups (0, 1, 10) will give you much
better performance in software. In my case, I have to have some piece
of hardware just to get to the drives, and I'm guessing that hardware
RAID5/6 will be faster than the closest software equivalent. Maybe my
tests will convince me otherwise.

- Max