Raid + zfs performace.

[Peter Ankerstål]

On 30 okt 2010, at 20.23, Jeremy Chadwick wrote:

> On Sat, Oct 30, 2010 at 07:56:56PM +0200, Peter Ankerstål wrote:
>> 
>> 
>> On 30 okt 2010, at 19.39, Sean wrote:
>> 
>>>> I have a question about raid and zfs. I Have a hardware-raid running.
>>>> A mirror thats the only storage in my zfs pool. Im going to
>>>> add another mirror to the machine and my question is, what is the best option performace-wise?
>>> 
>>> The best performance option is to get rid of the hardware-raid, and
>>> present each disc to ZFS in a JBOD fashion.
>> 
>> Ok. RIght now thats not an option. I have a da0 device thats a hardware-raid mirror. and it is currently
>> the only device in the only pool on the machine.
>> 
>>> 
>>>> Is it to add the other mirror to the same pool or create another separate pool for that mirror?
>>>> Btw. Today my disk are quite saturated r/w wise.
>>> 
>>> RAID functionality only exists within the context of a single pool.
>>> You don't create a new pool and then try to mirror the two pools. You
>>> add the storage to an existing pool, unless you have a reason to start
>>> a new pool. When I already have a mirror, I like to add new mirror
>>> sets. It's the equivalent of a RAID 10 scenario.
>> 
>> Yes I know.
>> I thought maybe because the existing pool is kind of r/w saturated it should be better
>> to create a new independent pool for the new drives. In that way the heavy activity 
>> would not "spread" to the new drives.
>> 
>> Now you presented me with a third option. So you think I should skip to create
>> a new hardware-raid mirror and instead use two single drives and add these as
>> a mirror to the existing pool? How will zfs handle howswap of these drives?
>> I've seen a few crashes due to ata-detach in other systems.
> 
> As stated previously: ideally you should turn off all RAID capabilities
> on the controller itself and just use JBOD + let FreeBSD deal with the
> RAID'ing (in this case, ZFS).
> 
> I realise you can't do this for the current hardware RAID'd "da0"
> mirror.  I also understand the reason you might have done this.  I too
> would trust a hardware RAID controller to handle booting from a mirror
> more than the FreeBSD bootstraps.  (I choose not to deal with things
> like gptzfsboot, zfsloader, etc. etc. -- these should all just become
> part of the standard bootstraps and stop requiring people to jump
> through hoops to achieve such capabilities)
> 
Yes.

> As for hot-swapping of drives: there are well-established procedures for
> this sort of thing.  I blogged about it[1] not too long ago, with
> real-world server hardware.  I think it will address your concerns,
> since I spent the time to document *everything*, including all output.
> 
> What confuses me about your statement is that you're talking about
> "ata-detach" when your drives are appearing as daX, which implies use of
> CAM.  Are you talking about hot-swapping failing on a completely
> different system which was using ata(4) without AHCI?  If so, I'm not
> surprised.  You need AHCI to achieve this.

Yes, that was another machine.

> 
> Finally, you should be aware that there are two AHCI "methods"
> implemented in FreeBSD: ataahci.ko and ahci.ko.  ataahci is the older of
> the two and uses ata(4).  ahci.ko is the newer of the two and is
> significantly better, since it makes use of CAM, and thus provides
> performance improvements and handles hot-swapping a lot better (IMHO).
> ahci.ko "trumps" ataahci.ko (meaning if your kernel includes ataahci and
> you do "load ahci" from loader or loader.conf, ahci.ko will be used
> instead).  Be aware that ahci.ko disks appear as adaX (this is not a
> typo).
> 
> [1]: http://koitsu.wordpress.com/2010/07/22/freebsd-and-zfs-hot-swapping-sata-disks-with-ahci/
> 
> -- 
> | Jeremy Chadwick                                   jdc at parodius.com |
> | Parodius Networking                       http://www.parodius.com/ |
> | UNIX Systems Administrator                  Mountain View, CA, USA |
> | Making life hard for others since 1977.              PGP: 4BD6C0CB |
> 
>