Very low disk performance on 5.x

[Steven Hartland]

----- Original Message ----- 
From: "Poul-Henning Kamp" <phk at phk.freebsd.dk>

>>Ok from what your saying it sounds like RAID on FreeBSD is useless
>>apart to create large disks. Now to the damaging facts the results
>>from my two days worth of testing:
> 
> Now, cool down a moment and lets talk about what you _really_ have
> measured.
> 
> You have measured the end to end performance: you include /dev/zero,
> dd(1), filesystem, disk device driver, hardware and disks.
> 
> As such that is a fair end-user benchmark, but unfortunately it
> doesn't really tell us anything useful for the purpose of this
> discussion.

Yes but the end-user performance is really the only thing that matters.
There are two killer issues here:
1. Write performance being nearly 3x that of read performance
2. Read performance only equalling that of single disk.

> If you are going to do a high-performance setup, you should not
> just take the "out-of-box" settings, you should optimize for
> the configurations particular features.

I'm quite willing to test and optimise things but so far no one has
had any concrete suggestions on that to try. As such I've done
all the benchmarks I've done varying the one key thing I could
find stripe size. If there are others please do tell and I'll test.
N.B. I need to get this machine into production so I only have
a limited time span available, I've already sacrificed a full install
with 700GB of data to run the previous tests.

This is just me though I think we do need to strive for good out
the box performance in these types of senarios

> Testing end-to-end means that we have very little to go from to
> find out where things went wrong in any one instance.

To eliminate various parts of the subsystems I've just tested:
dd if=/dev/da0 of=/dev/null bs=64k count=100000
Read: 220Mb/s

dd if=/dev/da0s1 of=/dev/null bs=64k count=100000
Read: 220Mb/s

dd if=/dev/da0s1f of=/dev/null bs=64k count=100000
Read: 220Mb/s

Compared with:
dd if=/usr/testfile of=/dev/null bs=64k count=100000
Read: 152Mb/s

So looks like the FS is adding quite an overhead ~70Mb/s ( 60% )
although from the linux tests we know the disks are capable
of at least another 40Mb/s

> But anyway: here are some questions which I wonder about ?

> Does the linux and FreeBSD filesystem offer the same semantics
> with respect to integrity ?  Ie: if one is asyncronous mode
> the comparison is not a fair comparison.   Just because you
> chose the default in each case doesn't mean you got the same
> thing.

FreeBSD using default UFS no special options in fstab.
Linux using Reiser again no special options.
I'm not familiar with the intrecuies of either FS so cant answer
the direct question here.

> Does any of the drivers change the settings/modes of the controllers
> use of cache ?  (this may be hard to determine without looking 
> at driver sources.

FreeBSD used both the downloadable module and the built in
module. Linux used the downloadable module. If these vary in any
way I dont know may be Paul could answer that as he did the driver
port.

> In particular, are you sure that the RAID-5 logical device was
> in the same exact state and location for each run ?

Yep I created the RAID5 logical volume the same way via the BIOS
for each test the only varient being the stripe size.

> Did you remember to disable all the debugging in FreeBSD 6-Current ?
> (see top of src/UPDATING)

Yep all debugging was disabled on my second run on current. Made
3MB/s difference IIRC ( reported value was without debugging ).
FreeBSD was tried with and without SMP no difference Linux was
SMP for all tests.
N.B. Current had at least on out of order lock issue while I was using
it but not while the tests where going on.

    Steve


================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137
or return the E.mail to postmaster at multiplay.co.uk.