Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

[Alexander Goncharov]

On Tue, May 27, 2008 at 1:10 AM, Gary Palmer <gpalmer at freebsd.org> wrote:

> On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander Goncharov wrote:
> > Hello world, I have faced with following issue on my dedicated server:
> >
> > 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10
> >
> > FreeBSD 7.0-generic, 64 bit version
> >
> >
> >
> > IO performance and behavior is very strange:
> >
> > 1)       No other process are running:
> >
> > Memory stat:
> >
> > Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free
> >
> > Copy 3GB file first time
> >
> > dd if=/home/3gb_file of=/home/3gb_file2
> > 6291456+0 records in
> > 6291456+0 records out
> > 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec)
> >
> > 20MBS is very poor?
> >
> > Memory stat now:
> >
> > Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free
> >
> > 2)       Copy the same file again:
> >
> >  dd if=/home/3gb_file of=/home/3gb_file2
> > 6291456+0 records in
> > 6291456+0 records out
> > 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec)
> >
> > 100MBs ? much better
> >
>
> The "Inact" (Inactive) went up - the 3GiB file is now cached in
> memory.  So the second (and subsequent) runs are going from cached
> memory so your 100MiB/sec transfer is actually just testing write
> speed, not read/write speed.  This is the same for your other
> tests too
>
> "Inact" is memory that has been used and is being kept around incase
> it is used again, in other words its caching file data in the
> "Inact" region in top.  It'll be reused if something else needs
> the memory, but until then it sticks around.
>
> Remember - you are copying the file from and to the same
> filesystem - this is always going to appear slow relative to
> pure read or pure write tests.  A pure write test is effectively
> what you have when you're getting your 100MiB/sec test result since
> its just writing out from cache memory.
>
> Regards,
>
> Gary
>

Hi Gary,

Big thanks for your quick reply. Read only and write only speed is good
~100MBs. But I am not sure if it really good for hw RAID 10 8x 15k rpm
drives.
I am worried about read/write speed usecase which is most used at real tasks
(data base). I was surprised 20 MBs speed, this value is likely to one drive
speed. But I have 8 high speed drives. Something isn't right here.

I am waiting for freebsd community help.

Thanks in advance.
Alexander