large RAID volume partition strategy

[Boris Samorodov]

On Fri, 17 Aug 2007 17:42:55 -0400 Vivek Khera wrote:

> I have a shiny new big RAID array.  16x500GB SATA 300+NCQ drives
> connected to the host via 4Gb fibre channel.  This gives me 6.5Tb of
> raw disk.

> I've come up with three possibilities on organizing this disk.  My
> needs are really for a single 1Tb file system on which I will run
> postgres.  However, in the future I'm not sure what I'll really need.
> I don't plan to ever connect any other servers to this RAID unit.

> The three choices I've come with so far are:

> 1) Make one RAID volume of 6.5Tb (in a RAID6 + hot spare
> configuration), and make one FreeBSD file system on the whole
> partition.

> 2) Make one RAID volume of 6.5Tb (in a RAID6 + hot spare
> configuration), and make 6 FreeBSD partitions with one file system
> each.

> 3) Make 6 RAID volumes and expose them to FreeBSD as multiple drives,
> then make one partition + file system on each "disk".  Each RAID
> volume would span across all 16 drives, and I could make the volumes
> of differing RAID levels, if needed, but I'd probably stick with RAID6
> +spare.

> I'm not keen on option 1 because of the potentially long fsck times
> after a crash.

> What advantage/disadvantage would I have between 2 and 3?  The only
> thing I can come up with is that the disk scheduling algorithm in
> FreeBSD might not be optimal if the drives really are not truly
> independent as they are really backed by the same 16 drives, so
> option 2 might be better.  However, with option 3, if I do ever end
> up connecting another host to the array, I can assign some of the
> volumes to the other host(s).

> My goal is speed, speed, speed.

Seems that RAID[56] may be too sloooow. I'd suggest RAID10.

I have 6 SATA-II 300MB/s disks at 3WARE adapter. My (very!) simple
tests gave about 170MB/s for dd. BTW, I tested (OK, very fast)
RAID5, RAID6, gmirror+gstripe and noone get close to RAID10. (Well, as
expected, I suppose).

> I'm running FreeBSD 6.2/amd64 and
> using an LSI fibre card.

If you have time you may do your own tests... And in case RAID0 you
shouldn't have problems with long fsck. Leave a couple of your disks
for hot-swapping and you'll get 7Tb. ;-)

> Thanks for any opinions and recommendations.


WBR
-- 
bsam