SATA 3ware RAID review...sort of.

[Gerald]

Details of a web server upgrade using SATA RAID5 drives.

Old Machine:
FreeBSD 4.8 (CUSTOM), Dual Pentium III 850s, 1 GB RAM
ADAPTEC 3200S FW Rev. 370F
External disk case for SCSI SCA 10k RPM drives.

Problem: Not enough memory. No way to upgrade RAM on existing MB. 
Solution: Kill soon to be hard drive issue and memory problem at same
time with new machine. (Long story about "hard drive issue"s on old 
machine.)

After shopping around we decided to try the SATA RAID setup for this
web server. Upon advice from this list, I went for the 3ware 9000 series
controller.

New Machine:
Freebsd 5.3 (GENERIC), Dual Xeon 3.06GHz, 4GB RAM 
3ware 9500S8CH 
Disks: (5x) 7200RPM 8MB cache SATA150 RAID5'ed

Some information about this server: It's totally dedicated to serving
web pages for one site. Site traffic during down months has a 95th
percentile of about 15 Mb. During Feb and Sep it jumps to 25 Mb and
this month we expect 30 Mb 95th percentile. For those that do transfer
numbers, the last week of September 04 saw 895.96 gigabytes total
transferred and 127.99 gigabytes average per day. (7 days)

On the old server I watched systat -v 1 for about 30 minutes recently
to see how busy the individual disks were during normal load. Most
partitions bounced around from 20% busy to 75% busy. This was a SCSI
setup though so I was concerned about moving over to SATA for something
that uses disk I/O so much.

The new setup is 1 TB of usable space. The new setup allows all of the
data to be on one partition that is RAID5'ed. I switched to the new
server Wed night at midnight and peak traffic hit today. They went from
15 mb to 30 Mb between 6AM and 8AM. (INCH has redundant 100 Mb upstream
connections.)

Systat and top now show the machine taking the load in stride. 
LA:  0.94  0.52  0.58 
Disk usage: hovers around 50% (40-55% predominantly)
CPU: stays about 80% Idle

I obviously don't have the apache defaults to serve up this many pages,
but I wanted to share the SATA success on a high usage web server. At
least from my perspective you can easily do 30 Mb of web traffic on a
properly configured RAID5 SATA system. If I had to guess, you could make
it to somewhere in the neighborhood of 75-100 Mb of traffic before you
would run in to I/O problems configured the way I have it. I wouldn't do
that much on it though since a drive failure would put your processes
waiting on the Disk I/O.

Problems related to drives:

The twa driver would not let me create more than I think 7 partitions.
I'm pretty sure it was 7. I've slept since then. It tried to create the
like "/dev/...i" as /dev/X literally. I just reworked my FS layout to
work around this. This might just be the 5.3 installer though. I don't
know. I didn't spend too much time on this.

Concessions:
- There are many many changes between FreeBSD 4.8 and 5.3. I'm
benefiting from quite a few improvements in that upgrade.

- UFS -> UFS2 probably plays quite a bit in to the above information.

- If I could have upgraded the RAM on the old machine, I could have made
do with a memory upgrade for an indeterminate amount of time. The memory
is what I needed the most.

- I've not tested (even for the fun of it) a drive failure and the
ensuing load should such a situation happen. I feel comfortable that I
have enough breathe room with the current load on the disks.

- I did stress test the disks with bonnie++ from the ports and it seemed
to do well.

- You can squeeze even more out of this setup with 10k RPM drives, but I
had to do 7200s to keep within customer's budget and it's handling the
load quite well.

- YMMV, This information comes with no warranty either expressed or
implied...etc etc. Contact me if you want a server setup the same way.

Gerald Coon
System Administrator
Internet Channel