RAID10 setup

[John Nielsen]

You're on the right track, additional comments inline.

On Saturday 22 August 2009 06:49:06 am Phil Lewis wrote:
> This question was asked a few weeks ago, but the original poster
> must have had their questions amswered. As follow-ups offered
> further assistance given more detail, I wonder if I could be so bold
> as to provide that detail for my own circumstances.
>
> I have six disks:
>
> ad4  - 500MB
> ad5  - 500MB
> ad6  - 500MB
> ad7  - 400MB
> ad8  - 500MB
> ad10 - 500MB
>
> These are SATA drives, with ad8 and ad10 on a PCIe SATA controller.
>
> ad7 was my first disk and currently contains FreeBSD7.2-RELEASE.
> I've been using that to gain some familiarity with FreeBSD, but it
> need not be preserved (in fact, I'd rather not preserve it!). When I
> built the machine, I just plugged the 400GB drive in any old slot,
> so it can move if that makes sense. When I got the new drives I tried
> to get identical to the 400GB drive, but couldn't. The 400GB drive
> currently has a single slice using the full drive.

Just make sure you have the disk(s) you plan to boot from on a controller 
that will boot in your machine. If the controllers have different 
performance characteristics then you probably want to share the wealth of 
the better one between multiple mirrors.

> What I'd like to end up with is a three-way stripe across three
> two-way mirrors, containing as much of the system as possible.

This is certainly do-able. If it were me I'd put the whole OS on 
the "spare change" partitions and leave the whole stripe for your serious 
data consumer(s): /home, /data, possibly /usr/local or some or all 
of /var, etc. Depends on your intended use of the storage naturally.

> I understand that you can't boot from a stripe, so some part of some
> disk will have to be outside the stripe. However, as the stripe will
> also be limited to the smallest disk, I'm going to have 5 x 100 GB
> bits left over anyway, so I guess /boot can go on one of these..?

Absolutely. I'd make a gmirror of two or three of them and put / on it. If 
you really want to be minimal w/ your use of the extra space then you 
could do /boot as you propose.

> If possible, I'd like set this up pre-install. If it has to be done
> post-install, or is easier to describe how to do post-install, then
> that's fine.

Either will work. Exactly how you do it depends on how much of the base 
system you want to end up on the stripe.

> >From here on in, this email becomes speculative.
>
> All of the examples I've seen for setting up GEOM stripes and mirrors
> have used the raw disk as the base-level provider. On the other hand,
> I've seen nothing that says that the bottom level cannot be a slice,
> rather than a raw disk, and given the way GEOM works, I suspect this
> is true.

Yes, you can use partitions, slices or any other GEOM providers as members 
of gstripe, gmirror and friends.

> My current plan, based on this assumption, is as follows:
>
> With my current FreeBSD installation, create 2 slices on each 500GB
> disk, 1 x ~400GB,  1 x ~100GB (the same size as the slice of my 400GB
> disk, and the rest of the disk).
>
> Boot from the FreeBSD 7.2-RELEASE dvd, and enter fixit mode. I'm
> not sure which would be best, or even if both are feasible for what I
> want to do. (I was at this point in my researchwhen I found this
> post!).
>
> >From here, kldload geom_stripe and kldload geom_mirror.
>
> Then, create the three mirrors:
>
> gmirror label -v main0 /dev/ad4s1 /dev/ad5s1
> gmirror label -v main1 /dev/ad6s1 /dev/ad571
> gmirror label -v main2 /dev/ad8s1 /dev/ad10s1
>
> This should give me /mirror/main0|main1|main2, right?

Right.

> Next create the stripe:
>
> gstripe label -v -s 131072 raid10 /dev/mirror/main0
> 			/dev/mirror/main1
> 			/dev/mirror/main2
> 	(that's all one line)
>
>
> If I'm right so far, then hopefully I should be able to boot to the
> install dvd again (or just rerun sysnstall?), and from there I should
> be able to choose a slice from outside 'raid10' to mount /boot, and
> use 'raid10' for everything else. Do I need anything else on a
> non-striped slice?

/boot or equivalent is the only thing required to smell like a normal disk 
(which gmirror is capable of but gstripe isn't). You may want to use some 
of the space for swap. The virtual memory system should do its own 
version of stripe or interleave if you feed it multiple swap devices.

> Maybe I could even create another mirror:
>
> gmirror label -v boot /dev/ad4s2 /dev/ad5s2
>
> and use that to mount /boot, leaving me with s2 on ad6,8 and 10 as
> 3 spare 100GB slices?
>
> Or am I just way off track?

You seem to be pretty well on track. It seems you've already parsed the 
gstripe and gmirror man pages. You should probably look at fdisk(8) and 
bsdlabel(8) as well in case sysinstall doesn't tie up all your loose 
ends. Additionally you could just reinstall to a plain disk (or use your 
existing installation) and use dump/restore (and/or rsync) to move your 
filesystems to their multi-disk destinations.

> PS. I can't believe I'm talking about 300 'spare' GB! My first disk
> was 20MB! And I never filed that!

Mine was too but it got full... :)

JN