More Controllers != Higher Through Put

[Stephen Sanders]

We found that in order to get the dual controller case up to speed we
had to set vfs.lorunningspace=1MB and vfs.hirunningspace=8MB.

On 07/07/2010 10:12 AM, Ivan Voras wrote:
> On 07/07/10 01:35, Stephen Sanders wrote:
>   
>> I'm wondering if anyone has heard of this.
>>
>> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives
>> and the motherboard servicing 2 7200 RPM Segate 1TB drives.
>>     
> So far so good.
>
>   
>> The 4 disk array is RAID 6 while the 2 disk array is RAID 1.  The drives
>> should deliver about 100MB/s.
>>     
> Ok, so you've reduced the 4-drive array's write performance nearly to
> equivalent of 2 drives and the 2-drive array to 1 drive. It should be
> even worse for random IOs.
>
> Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID?
>
> For the 2-disk RAID-1 : you are probably using software RAID, right?
> (on-board "SATA RAID" controllers usually are just software
> implementations).
>
>   
>> 1.  The most the 4 disk array is developing is 250MB/s write performance
>>     
> This is too much. It almost looks like something is caching what
> shouldn't be cached. How did you get this result?
>
> I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with
> 100 MB/s drives.
>
>   
>> while the 2 disk array is coming in at 90MB/s write performance.  
>>     
> This is as expected - write performance of any size RAID-1 is equivalent
> of 1 drive or less.
>
>   
>> The 4 disk array seems slow.
>>     
> Nope - the contrary should be true. It looks like you are doing
> something you shouldn't if you get that much performance, or your test
> is overly simplistic (e.g. you're testing cache).
>
>   
>> 2. Attempting to write to both arrays simultaneously causes the rate on
>> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s
>>     
> Are you running on an Atom CPU? What kind of system are you using?
>
>   
>> I'd expect the 4 disk array should look more like 300+MB/s while the 2
>> disk array is about right.
>>     
> No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction.
>
> Think about it: all the data needs to be a) written as-is to 2 of the
> drives, then b) parity/ECCs calculated and c) the same amount of data
> written to 2 more drives. You cannot get write performance of more than
> 2 drives equivalent in this scheme, and will probably be worse. For
> reads, only if your RAID controller is very, very smart (meaning: it
> probably isn't), you can recover some performance by using this
> parity/ECC data to reconstruct more data than is read from the two
> "plain" drives. I think ZFS does this in a limited way.
>
>   
>> I don't get why there should be a 'coupling' between the rates on
>> separate controllers. 
>>     
> This is the only thing which is puzzling a bit. I
>
>   
>> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and
>> the test program is using O_DIRECT for writes in order to avoid the page
>> daemon.
>>     
> You should use some benchmark which knows how to deal with OS cache, for
> example bonnie++ from the ports.
>
> Use a benchmark with random IO to see just how horrible your RAID-6
> performance will be for random writes.
>
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
>
>