Poor write performance with LSI 320-2 on 6.1-STABLE

[Albert Chin]

I have an Intel S875PWP1 motherboard with a Pentium4 CPU at 2.40GHz. PCI
bus is 33Mhz, 32-bit. I recently purchased an LSI 320-2/128MB on eBay
(though the card really looks like a PERC4/DS) and just ran some
bonnie++ tests on a RAID 1 array between two U320 drives for the first
channel and on a RAID 0 array between one U320 drive for the second
channel. The 320-2 has the latest LSI firmware, 1L47.

  #1. (RAID 1, two U320 drives, channel 1)
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
maetel.il.thew 300M   270  99 16917   5 15734   5   514  99 +++++ +++  4237  48
Latency             33357us     130ms   86843us   33195us    1063us     120ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
maetel.il.thewritte -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  9214  26 +++++ +++ +++++ +++  6687  19 +++++ +++ +++++ +++
Latency               424ms     129us    2629us     475ms     115us      82us

  #2. (RAID 1, two U320 drives, one on channel 1, one on channel 2)
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
maetel.il.thew 300M   272  99 16787   4 15949   5   515  99 +++++ +++  4575  50
Latency             31339us     139ms    1925ms   29314us     848us    1623ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
maetel.il.thewritte -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  7497  22 +++++ +++ +++++ +++  5850  17 +++++ +++ +++++ +++
Latency               516ms     123us    3381us     577ms   14425us      50us

  #3. (RAID 0, one U320 drive, channel 2)
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
maetel.il.thew 300M   270  99 17347   5 16347   5   530  99 +++++ +++  4510  46
Latency             34033us     131ms     100ms   34354us    1061us    1575ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
maetel.il.thewritte -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  8401  24 +++++ +++ +++++ +++  6489  19 +++++ +++ +++++ +++
Latency               574ms     127us    7480us     640ms   16607us      46us

  #4. I also testing an Adaptec 39160 card with one U320 drive:
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
maetel.il.thew 300M   259  99 62293  18 36054  12   522  99 +++++ +++  5747  58
Latency             33370us   26379us   43979us   34708us    1064us    1264ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
maetel.il.thewritte -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16761  49 +++++ +++ +++++ +++ 13569  40 +++++ +++ +++++ +++
Latency               142ms     141us      49us     203ms     123us      43us


I don't understand how #1 gives 16MB/s (RAID 1) while #3 gives only
17MB/s. I'd expect #3 to be about x2 the performance of #1.

The arrays are configured with "write-thru" write policy, "adaptive"
read policy, and "cachedio" cache policy.

Anything I can do to improve the speed. Even though the bus is 33Mhz,
I should definitely be doing better than 16MB/s.

I'm running bonnie++ as:
  $ cd [some tmp dir on UFS2 file system]
  $ bonnie++

The system runs FreeBSD 6.1-STABLE:
  $ uname -a
  FreeBSD maetel.il.thewrittenword.com 6.1-STABLE FreeBSD 6.1-STABLE #0: Mon May  8 22:25:41 CDT 2006
  $ dmesg | grep amr0
  amr0: <LSILogic MegaRAID 1.53> mem 0xf47f0000-0xf47fffff irq 21 at device 0.0 on pci3
  amr0: delete logical drives supported by controller
  amr0: <LSILogic MegaRAID SCSI 320-2> Firmware 1L47, BIOS G121, 128MB RAM
  amr0: delete logical drives supported by controller


-- 
albert chin (china at thewrittenword.com)