Hardware Raid + hot-replace failed disk

[Jeremy Chadwick]

On Fri, Nov 07, 2008 at 10:35:39AM +0100, Pieter Donche wrote:
> On Thu, 6 Nov 2008, Wojciech Puchar wrote:
>
>>> Suppose you have a system with multiple disks managed by a
>>> hardware RAID controller in a RAID5 of RAID6 configuration,
>>> To FreeBSD it will look like e.g. a single large drive.
>>
>> what is "RAID5 of RAID6"???
> RAID5 or RAID6 (sorry, typing error)
>
>>> If you want to extend your disk space by plugging in an extra
>>> disk, the hardware RAID controller will probably detect it and
>>> add it in his management, but will it be seen by FreeBSD?
>> FreeBSD will see larger drive.
>
> With what command can you see that FreeBSD had 'seen' it ?

The answer is: "it depends".  The below applies to SATA, SAS, and SCSI
only; you cannot hot-swap PATA disks.

If you have a hot-swap enclosure or a hot-swap backplane, and are using
a hardware RAID controller (and I do mean *real* hardware RAID, not
BIOS-level RAID like Intel MatrixRAID or Adaptec HostRAID), then the
FreeBSD controller driver should report the disk falling off the bus (if
a disk is removed), or a disk appearing on the bus (if a disk is added).
If the driver does not handle this natively, you will have to rely on
command-line utilities from the RAID card vendor to manage this.

If you have a hot-swap enclosure or a hot-swap backplane, and are using
software/OS-based RAID (such as gvinum, ccd, or ZFS), then it depends
on the underlying type of disk you're using.

With SATA disks, you rely on the FreeBSD ata(4) layer.  You are at the
whim of the ata(4) layer and its support for your motherboard chipset,
assuming that's what you're using (there are exceptions; see below).

Removal of a SATA disk should show the disk falling off the bus, and you
will need to perform "atacontrol detach <channel>" to ensure the kernel
knows the disk has been removed (this is not done automatically, despite
what you see on the console; I recommend you do the "detach" prior to
disk removal).

Addition of a SATA disk will require you to perform "atacontrol attach
<channel>", and hopefully you will see the disk make and model show up
moments later.

With SCSI or SAS disks, you rely on the FreeBSD da(4) layer, backed by
the FreeBSD CAM(4) layer.  This layer is proven reliable, and even some
SATA RAID controllers use it (such as Areca controllers; yes, they're
SATA disks on a hardware RAID controller, but the FreeBSD driver for the
Areca card uses da(4) and CAM(4)).

Removal of a SCSI disk should show the disk falling off the bus.  You
can use "camcontrol" to examine the state of things; you may need to
use start/stop (it's been a while since I've used camcontrol).

Addition of a SCSI disk might require "camcontrol rescan"; again,
it's been a while since I've used camcontrol.

In general, there is no easy way to describe every single scenario under
the sun.  It greatly depends upon what hardware you're using, and what
kind of disk you're using.  If you choose to use a hardware RAID card,
the card user manual should describe *exactly* how to accomplish
additions and removals.

Chances are you're talking about generic SATA disks hooked up to your
generic motherboard.  You should be aware that FreeBSD is somewhat
"flaky" in this regard.  I've recently written about a disk swap gone
bad (while using a Promise TX4310 controller), which should give you
some idea of the chaos that can happen as a result of shoddy driver
support:

http://wiki.freebsd.org/JeremyChadwick/ZFS_disk_upgrade_gone_bad

This article is followed-up by a fully-working example when using
an Intel ICH-based board with Intel AHCI enabled (meaning, everything
worked flawlessly and exactly how it should've):

http://wiki.freebsd.org/JeremyChadwick/ZFS_disk_upgrade_gone_bad_part_2

I'm still in the process of writing the details that make up Part 2.

> Or is the the bsdlabel command?

bsdlabel(8) is what creates filesystems.  To format filesystems, you use
newfs(8).

> Is bsdlabel a partition management program (such as GParted, Partition
> Magic)?

No, that's fdisk(8).  FreeBSD calls these "slices", not "partitions",
but they're the same thing.

If you want to "keep it simple", I recommend you use sade(8), which is
the text-based interface for partitioning and filesystem creation that
you see when you install FreeBSD.  If you don't have the "sade"
command, just run "sysinstall" and choose post-configuration.

> Is there any document (besides the manual pages bsdlabel, growfs, ..)
> that describes step-by-step what to do to grow an existing file system
> of to add a new file system on newly added disk space ?

What everyone else is telling you is sending you on a wild goose chase.
I'm sitting here imagining you clicking your mouse at 6000 clicks per
second, eyeballs rolling around, sweating profusely.  :-)  I wish
FreeBSD mailing list people wouldn't do this to new folks, because all
it's doing is confusing you.

The simple answer is this: on FreeBSD, there is not a reliable way to
grow an existing filesystem without taking the machine down, bringing
it into single-user, or rebooting it.  In fact, I would go so far to
say there is not a reliable way on FreeBSD to grow a filesystem at all;
you are generally forced to back up all of your data, then recreate
the filesystem.

ZFS doesn't make this any easier, by the way -- you cannot "grow" a
zpool.  You can add disks in real-time (no rebooting/single-user) to a
separate zpool (think: separate filesystem), but you cannot add
disks to an existing zpool.

Yes, I am aware Linux has LVM, which provides all of this capability
for you, without having to reboot.

If you're really wanting something like a end-user NAS, which allows
you to remove/add disks as you please, and the filesystem shrinks/grows
with it, then you need to be looking at a product from Data Robotics
called Drobo.

Note that Drobo does not work under FreeBSD, as far as I know (it
probably shows up as a umass/da device, but if you remove/add disks,
I'm absolutely certain the filesystem does not grow.  Linux Drobo
users have been dealing with that problem for quite some time too,
and Data Robotics is *just now* getting around to addressing it)

Hope this helps.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |