7.2-RELEASE-p4, IO errors & RAID1 failure

[Matthew Lear]

On Sun, 2010-06-27 at 09:36 +0100, Matthew Seaman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 27/06/2010 24:04:48, Matthew Lear wrote:
> > Incidentally, is there a way to easily migrate from a atacontrol created
> > array to a gmirror created array? I'm running FreeBSD 8.0 on another
> > machine with a gmirror created RAID1 array with no problem whatsoever (I
> > chose gmirror as the choice for this machine over atacontrol after
> > reading various postings about software RAID under recent releases of
> > FreeBSD). I was planning on upgrading the 7.2 machine to 8.0-RC1 anyway
> > so if I could easily move to using gmirror then I would. That said,
> > atacontrol should (I assume) function correctly with 8.x, shouldn't it,
> > or is support of it dwindling somewhat?
> 
> So long as your ataraid setup is a simple mirror, this is actually quite
> easy.
> 
> Lets assume you have two identical disks, ad0 and ad2 from which you
> create a RAID ar0.  To switch over to gmirror based RAID:
> 
>    * Edit /etc/fstab changing all references to ar0 to read ad0, so
>      eg.  change:
>        /dev/ar0s1a	/	ufs	rw	1	1
>      to read:
>        /dev/ad0s1a	/	ufs	rw	1	1
> 
>      and similarly for any other partitions on ar0 (don't forget the
>      swap area)
> 
>    * Edit /boot/loader.conf to load the necessary gmirror kernel modules
>      add:
>        geom_mirror_load="YES"
> 
>    * Reboot
>        If necessary, drop into the system BIOS and make sure that which
>        ever drive corresponding to your ad0 is set as the primary drive
>        to boot from.
> 
>        This should bring your system up on just your ad0 drive -- so no
>        resilience for a few minutes.
> 
>        (Effectively, you're back at the default system on a single
>        disk layout now)
> 
>     * Delete the old ar0 RAID:
>        # atacontrol delete ar0
> 
>       This should have no effect on the running system as all it does
>       is remove some meta-data stored on the disk.
> 
>    * Configure a new geom mirror device.  You need to know the sysctl
>      magic to let you futz with a mounted drive:
>        # sysctl kern.geom.debugflags=16
>        # gmirror label -v -b load gm0 /dev/ad0
> 
>      (If you're a fan of the classic article on OnLamp
>       http://onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html) which
>       recommends using round-robin as the balancing algorithm: that's
>       fine as well, but 'load' as a mechanism wasn't available when
>       that article was written, and it is generally recognised as
>       superior nowadays.)
> 
>    * Re-edit /etc/fstab to change all references from ad0 to mirror/gm0
>      so eg. change:
>        /dev/ad0s1a		/	ufs	rw	1	1
>      to:
>        /dev/mirror/gm0s1a 	/	ufs	rw	1	1
> 
>      and similarly for any other partitions on ad0
> 
>    * Reboot.  You should see messages towards the end of the kernel
>      initialisation sequence saying gmirror raid created successfully.
> 
>    * Now add your second drive to the mirror:
>        # gmirror insert gm0 /dev/ad2
> 
>    * The system will spend some time synchronizing the data from ad0 to
>      ad0 -- the system can be put back into use immediately if needed,
>      but it's better to let the resynch finish first, as you won't be
>      resilient until it has.  Also, active disk use will slow down the
>      synchronisation process.  Monitor synchronisation progress by
>      'gmirror status gm0' or by running gstat(8).
> 
>    * Once everything is synch'd it's all ready for immediate use.
> 
> > How easy is it to upgrade an array to use larger disks - atacontrol or
> > gmirror? Feel free to respond with RTFM :-) I suppose one possible
> > solution would be to use something like GpartEd (example Linux land
> > tool) to grow a certain partition on an array (eg the partition mounted
> > on /usr/local). That way both partitions on each of the separate array
> > subdisks would be grown transparently since the operation would be
> > performed on partition ar0s1<n> (ie, taken care of by atacontrol /
> > gmirror).
> 
> gmirror doesn't really help you with resizing disks or filesystems.
> Note the the recipe above assumes you want to mirror the entire contents
> of one disk onto another -- which is the usual idea for the system drive
> used by a server -- so ideally you want two identical drives.  If one of
> your drives fails and you need to replace it, but can't get the exact
> same model, you will have to use a *bigger* drive -- ie. a larger number
> of sectors.  Be wary when buying, as disks of different models but with
> nominally the same capacity can easily vary by a few thousand sectors.
> 
> Anyhow, it is certainly possible to replace one of the drives in a
> gmirror setup with another of much larger capacity, synchronise the
> drives, then replace the second drive with a similar larger one and
> synchronise back again.  If you have hot-swap capability, you can do all
> that without downtime.  However, you will end up with a system layout
> exactly the same size as the original disks.  In order to grow the
> filesystems to fill the drives, you will have to do all sorts of
> high-risk invasive procedures to rewrite partition tables and run
> growfs(8) to expand your UFS filesystems to fill the available space.
> Note: you can't shrink UFS partitions easily -- generally you have to
> backup the data, and then delete and recreate the partition as a smaller
> size and restore the data.
> 
> If you want to do this sort of migration from ar0 to a much bigger
> gmirror setup, probably the lowest risk, least effort route is to mount
> the two new drives on the system, configure a gmirror across the two,
> partition and create filesystems on the gmirror, use dump(8) and
> restore(8) to copy the filesystems from your live disks, then shutdown,
> take out the old ar0 disks and boot from the new gmirror.  Remember
> you're going to have to fix /etc/fstab on the gmirror before rebooting.
> 
> If you can't fit 4 drives in your system at once, then take out one
> drive (which will break the ar0 mirror), replace it with one of your new
> disks, build the gmirror on it as above, then shutdown, remove the 2nd
> ar0 drive, reboot, add the 2nd gmirror drive and resynch.
> 
> If you really want to be able to resize partitions easily or swap out
> the hardware and have the system utilize the increased space without a
> great deal of faffing around, then ZFS is your friend.  See:
> 
>     http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
> 
> You should be able to adapt those instructions to install a ZFS system
> on some fresh drives attached to an existing system which uses
> atacontrol or gmirror raid + UFS rather than booting from installation
> media, or you can split your original mirror and build half a ZFS mirror
> in the space that creates, then resynch disks afterwards.
> 
> It's more work to set up ZFS initially, but once it is done, it just
> ticks along happily and it makes any subsequent disk manipulations much
> easier and safer.
> 
Excellent info, Matthew. Thank you very much indeed.