ZFS on Hardware RAID

[Borja Marcos]

> On 22 Jan 2019, at 12:15, andy thomas <andy at time-domain.co.uk> wrote:
> 
> 
> Yesterday I set up a spare Dell 2950 with Perc 5/i Integrated HBA and six 73 GB SAS disks, with the first two disks configured as a RAID 1 system disk (/dev/mfid0) and the remaining 4 disks as RAID 0 (mfid1- mfid4). After adding a freebsd-zfs GPT partition to each of these 4 disks I then created a RAIDz1 pool using mfid1p1, mfid2p1 and mfid3p1 with mfid4p1 as a spare, followed by creating a simple ZFS filesystem.
> 
> After copying a few hundred MB of files to the ZFS filesystem, I yanked /dev/mfid3 out to simulate a disk failure. I was then able to manually detach the failed disk and replace it with the spare. Later, after pushing /dev/mfid3 back in followed by a reboot and scrubbing the pool, mfid4 automatically replaced the former mfid3 that was pulled out and mfid3 became the new spare.

You shouldn’t require a reboot. If the actual targets are exposed to the CAM layer your disks will appear as “da” (SAS backplane) and you can offline a device, 
hot plug a new one, at most do a “camcontrol rescan” to detect it and run a zfs replace (or whatever) without stopping the system. 

If your drives are “mfid” devices you may need either a reboot or some magic rituals using “sas2ircu” or “sas3ircu” to have the controller recognize the new drive
and accept it as a valid volume.

> So a spare disk replacing a failed disk seems to be semi-automatic in FreeBSD (this was version 10.3) although I have seen fully automatic replacement on a Solaris parc platform.

There are several stages at play here:

1- Starting up and recognizing a SAS or SATA drive. 

2- Having it recognized as a volume by a RAID card. With LSI cards and single disk RAID0 volumes it may require a reboot or using the sas2ircu/sas3ircu utility.

3- ZFS replacement, which on Solaris can be automatic and on FreeBSD is done manually (I haven’t tried zfsd yet).






Borja.