More Controllers != Higher Through Put
ivoras at freebsd.org
Wed Jul 7 14:12:47 UTC 2010
On 07/07/10 01:35, Stephen Sanders wrote:
> I'm wondering if anyone has heard of this.
> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives
> and the motherboard servicing 2 7200 RPM Segate 1TB drives.
So far so good.
> The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives
> should deliver about 100MB/s.
Ok, so you've reduced the 4-drive array's write performance nearly to
equivalent of 2 drives and the 2-drive array to 1 drive. It should be
even worse for random IOs.
Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID?
For the 2-disk RAID-1 : you are probably using software RAID, right?
(on-board "SATA RAID" controllers usually are just software
> 1. The most the 4 disk array is developing is 250MB/s write performance
This is too much. It almost looks like something is caching what
shouldn't be cached. How did you get this result?
I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with
100 MB/s drives.
> while the 2 disk array is coming in at 90MB/s write performance.
This is as expected - write performance of any size RAID-1 is equivalent
of 1 drive or less.
> The 4 disk array seems slow.
Nope - the contrary should be true. It looks like you are doing
something you shouldn't if you get that much performance, or your test
is overly simplistic (e.g. you're testing cache).
> 2. Attempting to write to both arrays simultaneously causes the rate on
> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s
Are you running on an Atom CPU? What kind of system are you using?
> I'd expect the 4 disk array should look more like 300+MB/s while the 2
> disk array is about right.
No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction.
Think about it: all the data needs to be a) written as-is to 2 of the
drives, then b) parity/ECCs calculated and c) the same amount of data
written to 2 more drives. You cannot get write performance of more than
2 drives equivalent in this scheme, and will probably be worse. For
reads, only if your RAID controller is very, very smart (meaning: it
probably isn't), you can recover some performance by using this
parity/ECC data to reconstruct more data than is read from the two
"plain" drives. I think ZFS does this in a limited way.
> I don't get why there should be a 'coupling' between the rates on
> separate controllers.
This is the only thing which is puzzling a bit. I
> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and
> the test program is using O_DIRECT for writes in order to avoid the page
You should use some benchmark which knows how to deal with OS cache, for
example bonnie++ from the ports.
Use a benchmark with random IO to see just how horrible your RAID-6
performance will be for random writes.
More information about the freebsd-performance