Slow Performance: Intel SRC16 RAID controller (amr driver)

[SharkTECH Internet Services]

Hello, 

I'm e-mailing you as a last ditch of effort to resolve this problem which really have left us hopeless. We have recently acquired a high-end server and have been beta testing it to bring it to production, but we are experiencing SERIOUS problems with the performance of the raid controller read/write performance which we are coming to you hoping for a resolution. 

System Specs: 
2 x AMD Opteron 244
2GB DDR-3200 ECC REG RAM
Tyan S2882 Board (most recent bios release)
Intel SRCS16 Raid Controller (Most recent bios release)
5 x 200GB 7.2KRPM WD 8MBuffer HDD's
Intel Pro/1000 SC Fiber

O/S: 
FreeBSD 5.4 RELEASE (amd64)

Raid is configured as Raid-5 with 64Kbyte and pretty much default settings all the way on that part. FreeBSD detects the Raid and installs it just fine. When it starts installing the O/S from CD it bursts to ~2000KB/s and go as low as 40KB/s while installing ports. Once system is installed we tried dd/bonnie/bonnie++ and a small script we even made to benchmark it yet all show really bad performance. Logs below will show tests made: 

bonnie: 
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100  5116  4.6  3412  0.7  9850  1.6 91008 99.8 1530299 99.9 118084.7 98.7

bonnie++: 
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
godzilla.hosts 300M   326  98  4824   1  4459   1   749  99 +++++ +++  1261   3
Latency             34053us     299ms     291ms   24724us     653us    6277ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
godzilla.hosts.bc1. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3216   9 +++++ +++ +++++ +++  2582   6 +++++ +++ +++++ +++
Latency              1757ms     299us      64us    1811ms     159us     253us
1.93c,1.93c,godzilla.hosts.bc1.bresnan.net,1,1116728125,300M,,326,98,4824,1,4459,1,749,99,
+++++,+++,1261,3,16,,,,,3216,9,+++++,+++,+++++,+++,2582,6,+++++,+++,+++++,
+++,34053us,299ms,291ms,24724us,653us,6277ms,1757ms,299us,64us,1811ms,159us,253us

dd: 
dd if=/dev/amrd0s1a of=/dev/null
^C36267+0 records in
36267+0 records out
18568704 bytes transferred in 9.330515 secs (1990105 bytes/sec)

Custom script making continuous copies of 1 file (223M):
Sat May 21 18:58:24 MDT 2005: 291072 KB/min (4851.20 KB/sec)
Sat May 21 18:59:25 MDT 2005: 286064 KB/min (4767.73 KB/sec)
Sat May 21 19:00:25 MDT 2005: 285808 KB/min (4763.46 KB/sec)
Sat May 21 19:01:25 MDT 2005: 279728 KB/min (4662.13 KB/sec)

dmesg information:
amr0: <LSILogic MegaRAID 1.51> mem 0xff5f0000-0xff5fffff irq 28 at device 3.0 on pci3
amr0: <LSILogic Intel(R) RAID Controller SRCS16> Firmware 713N, BIOS G401, 64MB RAM
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 763108MB (1562845184 sectors) RAID 5 (optimal)

We have already tried to play around with enabling/disabling the cache on the hard drives, we even used a different set of 3 80GB 7.2KRPM WD drives instead to see if their is a disfunctional drive causing this yet nothing. This problem has been really frustrating for us and we even got a couple of people emailing us with the same problem they are having I hope you understand we've been using FreeBSD for the last 7 years and we really don't wanna stop using it now, as the problem does not occur in other O/S (ie MSWindows).

If any driver developers that can modify amr driver would like to test it on our system, feel free to contact me back via e-mail to provide you SSH access.

I really hope you would be able to help us out, really appreciate your time on this.

Thank you
Tim Timrawi
SharkTECH Internet Services.