Raid 0 + 1
Felipe Neuwald
felipe at neuwald.biz
Mon Oct 29 11:14:25 PDT 2007
Hi Kevin,
A bootable 0+1 RAID isn't my idea, cause I have one small disk to system
files, but I'll try use your documentation to make my 0+1 RAID.
I have one blog to put these documentation, if it's ok to you, please
take a look at ontheroadbrother.blogspot.com .
Cheers,
Felipe Neuwald
Kevin Thompson escreveu:
>> Hi Folks,
>>
>> I talked with my customer, and we decided to implement a 0 + 1 RAID,
>> with 4 disks of 250Gb each.
>>
>> Here is how my RAID is working now:
>>
>
> I have some rough instructions I wrote up about doing a bootable 0+1 Geom
> RAID, in wiki format. I don't have a public place to post them, so I've
> just copied them below. Feedback is welcome.
>
> Once you get it setup, I would highly recommend that you experiment with
> the setup before relying on it - make sure you know how to handle failed
> disk replacements, etc. VMWare/camcontrol/atacontrol is very handy in this
> regard.
>
> --Kevin Thompson
>
>
>
>
> ==Introduction==
>
> This article describes one method to build a RAID 0+1 array with 4 logical
> disks using the FreeBSD GEOM framework, and being able to boot off of such
> an array.
>
> Building a GEOM mirror and being able to directly boot off of it is a
> relatively simple task - this is due to the fact that when mirroring is
> done, all FreeBSD slices maintain their disk-level structure - GEOM stores
> its metadata at the end of the disk/partition/etc. Most importantly, in
> such a configuration the MBR and boot file system is untouched.
>
> The latter is not the case in RAID 0, RAID 0+1, or RAID 1+0 setups.
>
> In striping configurations, the boot filesystem is interleaved between the
> various disks (usually two), and as such, the filesystem appears to be
> corrupt if read from only one disk.
>
> Since the MBR program on the boot disk is incapable of understanding this
> physical block layout, it is unable to find the kernel in order to start
> FreeBSD.
>
> However, should the MBR program be able to read the boot filesystem, it
> can then load the kernel and related kernel modules, which would then
> allow the system to 'boot' from the raidset.
>
> Technically speaking, the only file system that has to remain untouched is
> the /boot file system. root (/), /usr, /var, /tmp, et cetera may all then
> be mounted from a raidset.
>
> In this example, for simplicity's sake, I protect all of the typical root
> file system, not just /boot. A more enterprising user may want to modify
> their approach.
>
> ==Instructions==
> Boot with a FreeBSD install disk, then start the fixit console. Next
> you'll need to clean up the environment (the fixit console needs a little
> updating)
> ln -s /dist/boot/kernel /boot/kernel
> ln -s /dist/lib /lib
> EDITOR=/mnt2/usr/bin/vi; export EDITOR
> PATH=$PATH':'/mnt2/sbin':'/mnt2/usr/bin':'/mnt2/usr/sbin
> export PATH
>
>
> Now load the kernel modules for the geom modules we're going to be using.
> glabel load
> gstripe load
> gmirror load
>
> Next, we're going to label each of the drives. The 'da0' drive name is
> typically assigned by the bios on boot, by way of some sort of metric such
> as chain location. The geom label module writes a fixed label to each
> drive, so that no matter where each drive goes, or if new devices are
> inserted and the numbering is reordered, the raid set will always work the
> same.
> glabel label geom0 da0
> glabel label geom1 da1
> glabel label geom2 da2
> glabel label geom3 da3
>
> Now install a mbr and basic partioning on each drive. This will create a
> single partition taking the entire drive for each drive.
> fdisk -vBI /dev/label/geom0
> fdisk -vBI /dev/label/geom1
> fdisk -vBI /dev/label/geom2
> fdisk -vBI /dev/label/geom3
>
> On the first disk slice of the first drive, install a simple disk label
> and bootstrap code.
> bsdlabel -wB /dev/label/geom0s1
>
> Now edit the generic label on that disk, setting it as you please. 'a' is
> commonly root, 'b' the swap partition and 'd' the rest. Don't create any
> more partitions other than a, b and d (d will be used as the provider for
> a future geom consumer).
>
> slave(/u9/antiduh) # bsdlabel -e /dev/label/geom0s1
> # size offset fstype [fsize bsize bps/cpg]
> a: 500M 16 4.2BSD
> b: 500M * swap
> c: # leave as is
> d: * * 4.2BSD
>
> The '*' for size means use whatever is left, and the '*' for offset means
> use the next logical offset.
>
> Once you're finished labeling the first drive, write the label for the
> drive out to a file, then use it to initialize the other three disks:
> bsdlabel /dev/label/geom0s1 > /file
> bsdlabel -R /dev/label/geom1s1 /file
> bsdlabel -R /dev/label/geom2s1 /file
> bsdlabel -R /dev/label/geom3s1 /file
>
>
> Now, create a GEOM mirror out of the 'a' partition of each drive. This
> will eventually be the root partition. The new device will be called
> '''boot''', and will enumerate in FreeBSD as '''/dev/mirror/boot'''
> gmirror label -vh boot /dev/label/geom0s1a /dev/label/geom1s1a
> /dev/label/geom2s1a /dev/label/geom3s1a
>
> Now we're going to pull together the d partion on each drive, create
> pairwise stripes, then mirror the new stripes together.
> gstripe label -vh -s 131072 st0 /dev/label/geom0s1d /dev/geom1s1d
> gstripe label -vh -s 131072 st1 /dev/label/geom2s1d /dev/geom3s1d
>
> We should now have two new devices '''/dev/stripe/st0''' and
> '''/dev/stripe/st1'''. Now mirror those two devices to create our final
> device that will next be used for the rest of our filesystems:
> gmirror label -vh gm0 /dev/stripe/st0 /dev/stripe/st1
>
> We now have our final device '''/dev/mirror/gm0''' that is going to serve
> as the base for our regular filesystems.
>
> Create a basic slicing of the new disk, then edit to taste. Note that we
> don't fdisk the raw gm0 device - we create slices directly on the raw
> device.
> slave(/u9/antiduh) # bsdlabel -wB /dev/mirror/gm0
> slave(/u9/antiduh) # bsdlabel -e /dev/mirror/gm0
> # size offset fstype [fsize bsize bps/cpg]
> a: 1 16 unused # that is just a bare '1', not '1M'. This is to get
> around a bug in bsdlabel
> c: #-- leave as is --
> e: 1000M * 4.2BSD # /var
> f: 500M * 4.2BSD # /tmp
> d: * * 4.2BSD # /usr, which just gets the rest
>
> Now create our filesystems on associated slices (the -U option enables
> soft updates)
> newfs /dev/mirror/boot
> newfs -U /dev/mirror/gm0d
> newfs -U /dev/mirror/gm0e
> newfs -U /dev/mirror/gm0f
>
> Now we finally get down to mounting the disks, installing the OS on it,
> setting up the OS to boot, and finally booting.
>
> Mount the root disk as /mnt
> mount /dev/mirror/boot /mnt
>
> Create mount points for our other file systems, then mount them:
> mkdir /mnt/usr
> mkdir /mnt/var
> mkdir /mnt/tmp
>
> mount /dev/mirror/gm0d /mnt/usr
> mount /dev/mirror/gm0e /mnt/var
> mount /dev/mirror/gm0f /mnt/tmp
>
> Now we're going to do a install the fun way:
> DESTDIR=/mnt
> export DESTDIR
> cd /dist/6.2-RELEASE/base; ./install.sh
> cd ../ports; ./install.sh
> cd ../manpages; ./install.sh
> cd ../kernels; ./install.sh GENERIC
> mv /mnt/boot/GENERIC/* /mnt/boot/kernel
>
>
> You now have a basic, blank, sorta bootable unconfigured FreeBSD install
> on the machine.
>
> Setup the boot loader to load the needed kernel modules, so that we can
> mount root from the mirror.
> echo 'geom_label_load="YES"' >> /mnt/boot/loader.conf
> echo 'geom_stripe_load="YES"' >> /mnt/boot/loader.conf
> echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf
>
> Set a kernel option to use more memory but make the stripe layer faster,
> otherwise stripes are unbearably slow.
> echo 'kern.geom.stripe.fast=1"' >> /mnt/boot/loader.conf
>
> Now setup our initial fstab file:
> echo '/dev/label/geom0s1b none swap sw 0 0' >> /mnt/etc/fstab
> echo '/dev/label/geom1s1b none swap sw 0 0' >> /mnt/etc/fstab
> echo '/dev/label/geom2s1b none swap sw 0 0' >> /mnt/etc/fstab
> echo '/dev/label/geom3s1b none swap sw 0 0' >> /mnt/etc/fstab
> echo '/dev/mirror/boot / ufs rw 1 1' >> /mnt/etc/fstab
> echo '/dev/mirror/gm0d /usr ufs rw 2 2' >> /mnt/etc/fstab
> echo '/dev/mirror/gm0e /var ufs rw 2 2' >> /mnt/etc/fstab
> echo '/dev/mirror/gm0f /tmp ufs rw 2 2' >> /mnt/etc/fstab
>
> Reboot and the machine should start the raid sets automatically and mount
> off of the raid array automatically. Keep that install CD handy in case
> you messed up.
>
> Login, change the root password, setup rc.conf (hostname, interfaces, ssh,
> linux binary compat... ), start installing stuff, etc.
>
> Enjoy.
>
> ==Block Diagram==
> <code>
> da0 --label-->geom0
> da1 --label-->geom1
> da2 --label-->geom2
> da3 --label-->geom3
>
> label/geom0s1a --|--mirror-->/dev/mirror/boot
> label/geom1s1a --|
> label/geom2s1a --|
> label/geom3s1a --|
>
> label/geom0s1d --|--stripe-->/dev/stripe/st0 --|--mirror-->/dev/mirror/gm0
> label/geom1s1d --| |
> |
> label/geom2s1d --|--stripe-->/dev/stripe/st1 --|
> label/geom3s1d --|
>
> mirror/boot --> /
>
> mirror/gm0s1d --> /usr
> mirror/gm0s1e --> /var
> mirror/gm0s1f --> /tmp
>
> label/geom0s1b --> swap
> label/geom1s1b --> swap
> label/geom2s1b --> swap
> label/geom3s1b --> swap
> </code>
>
>
>
>
>
>
More information about the freebsd-geom
mailing list