LSI Megaraid (amr) performance woes

[Sven Willenberger]

On Thu, 2006-02-23 at 15:53 -0500, Kris Kennaway wrote: 
> On Thu, Feb 23, 2006 at 03:41:06PM -0500, Sven Willenberger wrote:
> > I am having some issues getting any (write) performance out of an LSi
> > Megaraid (320-1) SCSI raid card (using the amr driver). The system is an
> > i386 (p4 xeon) with on-board adaptec scsi controllers and a SUPER GEM318
> > Saf-te backplane with 6 ea 146GB U320 10k rpm Hitachi drives.
> 
> Try again with 6.1, performance should be much better.
> 
> Kris

I cvsupped a 6.1 prerelease and found no performance improvements. I did
some further tests and the performance issues seem very specific to the
mirroring aspect of the raid:

The server has 6 drives so I set up a single-drive Raid0, dual-drive
raid0 and triple-drive raid0 using the lsi configuration tool. Since
these were simple stripes I would expect increasing performance with
each additional drive and the results matched these expectations. Write
speeds were roughly 50MB/sec, 100MB/sec, and 150MB/sec for the
single,dual, and triple drive stripes respectively with read speeds on
the order of 110MB/s or so.

I then went back to the beginning and setup a simple 2-drive mirror, and
a 4-drive raid 10 (spanning/striping over 2 mirrors). After reinstalling
the OS I ended up with the following results:
on the 2-drive mirror write speeds were an abysmal 7MB/sec and on the
4-disk raid10 write speeds were 8MB/sec. Looking at iostat during the
write (using dd if=/dev/zero of=filename bs=64k count=32768) I saw that
the 2-drive mirror seemed to jump between 35 tps and 65 tps with and
average 128kb per transaction while the 4-drive array maintained a more
consistent 65 tps. Read speeds on the 4-drive array were around
110MB/sec.

I cannot rule out the possiblity that the card itself is bad but in an
effort to try and do so I tried these tests using an install of
slackware 10.2 (using the 2.6 kernel and megaraid.ko module) and
reiserfs filesystem. On the 2-drive mirror I achieved write speeds of
22MB/sec and the 4-drive array saw write speeds of about 45MB/sec. Read
speeds on the 4-drive array were roughly 150MB/sec.

In summary, it would seem that the amr driver (which I also tested on a
tyan transport system (using the same hard drives) using FreeBSD amd64
(as it is an AMD system) and had the same results as on the i386 system)
seems to have issues when any type of data-duplication (mirror) scheme
is in place. Is there anything I can try and do to troubleshoot this on
a lower level? Is it possible that the card is actually just defective?

Sven