ZFS or UFS for 4TB hardware RAID6?

[Richard Mahlerwein]

--- On Mon, 7/13/09, Maxim Khitrov <mkhitrov at gmail.com> wrote:

> From: Maxim Khitrov <mkhitrov at gmail.com>
> Subject: Re: ZFS or UFS for 4TB hardware RAID6?
> To: mahlerrd at yahoo.com
> Cc: "Free BSD Questions list" <freebsd-questions at freebsd.org>
> Date: Monday, July 13, 2009, 3:23 PM
> On Mon, Jul 13, 2009 at 2:13 PM,
> Richard Mahlerwein<mahlerrd at yahoo.com>
> wrote:
> >
> > --- On Mon, 7/13/09, Maxim Khitrov <mkhitrov at gmail.com>
> wrote:
> >
> >> From: Maxim Khitrov <mkhitrov at gmail.com>
> >> Subject: Re: ZFS or UFS for 4TB hardware RAID6?
> >> To: mahlerrd at yahoo.com
> >> Cc: "Free BSD Questions list" <freebsd-questions at freebsd.org>
> >> Date: Monday, July 13, 2009, 2:02 PM
> >> On Mon, Jul 13, 2009 at 1:46 PM,
> >> Richard Mahlerwein<mahlerrd at yahoo.com>
> >> wrote:
> >> >>
> >> >> Your mileage may vary, but...
> >> >>
> >> >> I would investigate either using more
> spindles if
> >> you want
> >> >> to stick to RAID6, or perhaps using
> another RAID
> >> level if
> >> >> you will be with 4 drives for a while. 
> The
> >> reasoning
> >> >> is that there's an overhead with RAID 6 -
> parity
> >> blocks are
> >> >> written to 2 disks, so in a 4 drive
> combination
> >> you have 2
> >> >> drives with data and 2 with parity.
> >> >>
> >> >> With 4 drives, you could get much, much
> higher
> >> performance
> >> >> out of RAID10 (which is alternatively
> called
> >> RAID0+1 or
> >> >> RAID1+0 depending on the manufacturer and
> on how
> >> accurate
> >> >> they wish to be, and on how they
> actually
> >> implemented it,
> >> >> too). This would also mean 2 usable
> drives, as
> >> well, so
> >> >> you'd have the same space available in
> RAID10 as
> >> your
> >> >> proposed RAID6.
> >> >>
> >> >> I would confirm you can, on the fly,
> convert from
> >> RAID10 to
> >> >> RAID6 after you add more drives.  If you
> can not,
> >> then
> >> >> by all means stick with RAID6 now!
> >> >>
> >> >> With 4 1 TB drives (for simpler
> examples)
> >> >> RAID5 = 3 TB available, 1 TB worth used
> in
> >> "parity".
> >> >> Fast reads, slow writes.
> >> >> RAID6 = 2 TB available, 2 TB worth used
> in
> >> "parity".
> >> >> Moderately fast reads, slow writes.
> >> >> RAID10 = 2 TB available, 2TB in duplicate
> copies
> >> (easier
> >> >> work than parity calculations).  Very
> fast
> >> reads,
> >> >> moderately fast writes.
> >> >>
> >> >> When you switch to, say, 8 drives, the
> numbers
> >> start to
> >> >> change a bit.
> >> >> RAID5 = 7TB available, 1 lost.
> >> >> RAID6 = 6TB available, 2 lost.
> >> >> RAID10 = 4TB available, 4 lost.
> >> >>
> >> >
> >> > Sorry, consider myself chastised for having
> missed the
> >> "Security is more important than performance" bit.
> I tend
> >> toward solutions that show the most value, and
> with 4
> >> drives, it seems that I'd stick with the same
> "data
> >> security" only pick up the free speed of RAID10.
>  Change
> >> when you get to 6 or more drives, if necessary.
> >> >
> >> > For data security, I can't answer for the
> UFS2 vs.
> >> ZFS.  For hardware setup, let me amend everything
> I said
> >> above with the following:
> >> >
> >> > Since you are seriously focusing on data
> integrity,
> >> ignore everything I said but make sure you have
> good
> >> backups!  :)
> >> >
> >> > Sorry,
> >> > -Rich
> >>
> >> No problem :) I've been doing some reading since I
> posted
> >> this
> >> question and it turns out that the controller will
> actually
> >> not allow
> >> me to create a RAID6 array using only 4 drives.
> 3ware
> >> followed the
> >> same reasoning as you; with 4 drives use RAID10.
> >>
> >> I know that you can migrate from one to the other
> when a
> >> 5th disk is
> >> added, but RAID10 can only handle 2 failed drives
> if they
> >> are from
> >> separate RAID1 groups. In this way, it is just
> slightly
> >> less resilient
> >> to failure than RAID6. With this new information,
> I think I
> >> may as
> >> well get one more 2TB drive and start with 6TB of
> RAID6
> >> space. This
> >> will be less of a headache later on.
> >>
> >> - Max
> >
> > Just as a question: how ARE you planning on backing
> this beast up?  While I don't want to sound like a
> worry-wort, I have had odd things happen at the worst of
> times.  RAID cards fail, power supplies let out the magic
> smoke, users delete items they really want back... *sigh*
> 
> Rsync over ssh to another server. Most of the data stored
> will never
> change after the first upload. A daily rsync run will
> transfer one or
> two gigs at the most. History is not required for the same
> reason;
> this is an append-only storage for the most part. A backup
> for the
> previous day is all that is required, but I will keep a
> weekly backup
> as well until I start running out of space.
> 
> > A bit of reading shows that ZFS, if it's stable
> enough, has some really great features that would be nice on
> such a large pile o' drives.
> >
> > See http://wiki.freebsd.org/ZFSQuickStartGuide
> >
> > I guess the last question I'll ask (as any more may
> uncover my ignorance) is if you need to use hardware RAID at
> all?  It seems both UFS2 and ZFS can do software RAID
> which seems to be quite reasonable with respect to
> performance and in many ways seems to be more robust since
> it is a bit more portable (no specialized hardware).
> 
> I've thought about this one a lot. In my case, the hard
> drives are in
> a separate enclosure from the server and the two had to be
> connected
> via SAS cables. The 9690SA-8E card was the best choice I
> could find
> for accessing an external SAS enclosure with support for 8
> drives.
> 
> I could configure it in JBOD mode and then use software to
> create a
> RAID array. In fact, I will likely do this to compare
> performance of a
> hardware vs. software RAID5 solution. The ZFS RAID-Z option
> does not
> appeal to me, because the read performance does not benefit
> from
> additional drives, and I don't think RAID6 is available in
> software.
> For those reasons I'm leaning toward a hardware
> implementation.
> 
> If I go the hardware route, I'll try to purchase a backup
> controller
> in a year or two. :)
> 
> > There are others who may respond with better
> information on that front.  I've been a strong
> proponent of hardware RAID, but have recently begun to
> realize many of the reasons for that are only of limited
> validity now.
> 
> Agreed, and many simple RAID setups (0, 1, 10) will give
> you much
> better performance in software. In my case, I have to have
> some piece
> of hardware just to get to the drives, and I'm guessing
> that hardware
> RAID5/6 will be faster than the closest software
> equivalent. Maybe my
> tests will convince me otherwise.
> 
> - Max

I'd love to hear about any test results you may get comparing software with hardware raid.