[SO]HO Software RAID5 server: which implementation should I choice?

[Arnaud Houdelette]

Lev Serebryakov a écrit :
> Hello, freebsd-stable.
>
>   Does somebody use some software RAID5 on FreeBSD in real production
>   system?
>
>   I want to build storage server for my home: RAW photos, multi-layer
> PhotoShop files and FLAC-encoded music consume a lot of space, and
> they should be availible both from desktop & notebook.
>
>   Also, all photo-content is unique, so I need some insuranse from
> single HDD crash. I understand, that I will not safe from fire, PSU
> failure and thing slike this.
>
>   I selected hardware platform: Intel Q35-based MoBo with 6xSATA-II ports
> (all of them is chipset-based, so no SiliconImage/JMicron/Whatever
> crappy controllers), some low-end Core2Duo, 2Gb of memory.
>   Storage will be 5x500Gb WD HDDs for RAID + one small HDD for boot, system,
> swap, etc. I want to have 2Tb (ok, not real Tb, I know) of "protected"
> storage.
>   I want to have maximum speed via 1Gb network, because graphic files
> are big and should open fast. Not as fast as local ones, I understand
> that, but speeds about 12-15Mb/s is not enough for sure :)
>
>   Only problem I see: which software RAID5 solution should I prefer?
> FreeBSD-based, of course!
>
>   I see these variants:
>
>  (1) FreeBSD 6(7?) + graid3. Slow, one disk for checksums is bottleneck,
>      as far as I understand.
>
>  (2) FreeBSD 6(7?) + gvinum/radi5. Is it stable enough?! Is it complete?
>      when I try it about 6 months ago in VMWare installation with 5
>      virtual disks, I got panics and strange behaviour after "crashing"
>      one of virtual disks.
>
>  (3) FreeBSD 6(7?) + graid5. Again, is it stable enough? There are
>      THREE versions of it. Which one should I prefer? There was long
>      thread about it some times ago without any clear conclusion. Does
>      something changed?
>
>  (4) FreeBSD 7 + ZFS "zraid". And again: stability. Too many messages
>      about locks, crashes, etc. Code is experemental. Is it only for
>      32 bit systems?
>
>  (5) Do I miss something?
>
>  (6) Solaris + ZFS? I don't want it, I know a little about Solaris
>      administaration, and I already have FreeBSD servers and routers.
>
>   I know, that 3ware or Areca controllers are very good. I know, that
>   "gmirror" is very stable. But these variants are too expensive for
>   home server :(
>
>   Does somebody use some software RAID5 on FreeBSD in real production?
>   Any advices?
>   

Hi !

I personally use the 3 option for my personal Home File Server. I got 
approximatly the same usage for the file server (mostly video, music, 
photo). I built my own about 12 month ago.
I reviewed the about the same variants as the one you propose :
(1) Discarded for performance issues. Raid3 is slow. Really.
(2) raid5/vinum is also slow. And as I understood at that time,  
recovery from lost hard drive wasn't easy enough for the freebsd niubee 
I was  then.
(4) ZFS wasn't there yet. But I did test it on a test VMWare, and wasn't 
convinced (mostly stability and memory issues).

So I use geom_raid5. I sticked to the main distributions, which seemed 
more stable at the moment. The kernel module is fairly simple to 
build/install. Performance is (very) good for a software raid.
I successfully switched the raid array from an i386 6.2-RELEASE to an an 
amd64 7.0-RELEASE (with motherboard and CPU change) without any assle.

For the moment, I use one big UFS+SU (and snapshots) on the whole array. 
I successfuly tried unpplugging then replugging on of the drives, 
suddent power loss, using the array with a missing disk (degraded mode). 
All did work fine. (still, I use an UPS on the file server).

The sole issue I had is with ataidle. I had to patch ata-disc.c to 
increase the IO timeout. Without, the raid5 module detected temporary 
disk loss and constantly launched rebuilds of the array.

With 7.0, I wondered if I should use gjournal, but I'm not sure if it's 
really the way to go on a file system dedicated to store many big files. 
So I stick to soft updates.

Current configuration is :
/ on a 2GB usb key
/tmp on memory
ports and source trees (and some portsnap stuff) on a small disk
4x250 GB sata for the raid5 array.
AMD A64 3200+ and 512 GB DDRII
Realtek Gigabit nics.

Copy from raid5 to /dev/null gives about 100MB/s
Copy from /dev/random to raid5 about 40MB/s

I use samba shares. I get about 40MB/s in both ways from another 
computer on the network (enabling jumbo-frames gives a big boost).

Hope my own story can help you in any way.

Regards,

Arnaud Houdelette