8.0-RELEASE/amd64 - full ZFS install - low read and write disk performance

[Bob Friesenhahn]

On Sun, 24 Jan 2010, Dan Naumov wrote:
>
> This works out to 1GB in 36,2 seconds / 28,2mb/s in the first test and
> 4GB in 143.8 seconds / 28,4mb/s and somewhat consistent with the
> bonnie results. It also sadly seems to confirm the very slow speed :(
> The disks are attached to a 4-port Sil3124 controller and again, my
> Windows benchmarks showing 65mb/s+ were done on exact same machine,
> with same disks attached to the same controller. Only difference was
> that in Windows the disks weren't in a mirror configuration but were
> tested individually. I do understand that a mirror setup offers
> roughly the same write speed as individual disk, while the read speed
> usually varies from "equal to individual disk speed" to "nearly the
> throughput of both disks combined" depending on the implementation,
> but there is no obvious reason I am seeing why my setup offers both
> read and write speeds roughly 1/3 to 1/2 of what the individual disks
> are capable of. Dmesg shows:

There is a mistatement in the above in that a "mirror setup offers 
roughly the same write speed as individual disk".  It is possible for 
a mirror setup to offer a similar write speed to an individual disk, 
but it is also quite possible to get 1/2 (or even 1/3) the speed. 
ZFS writes to a mirror pair requires two independent writes.  If these 
writes go down independent I/O paths, then there is hardly any 
overhead from the 2nd write.  If the writes go through a 
bandwidth-limited shared path then they will contend for that 
bandwidth and you will see much less write performance.

As a simple test, you can temporarily remove the mirror device from 
the pool and see if the write performance dramatically improves. 
Before doing that, it is useful to see the output of 'iostat -x 30' 
while under heavy write load to see if one device shows a much higher 
svc_t value than the other.

> expectations, I was seeing 65-70mb/s. Note: again, exact same
> hardware, exact same disks attached to the exact same controller. In
> my knowledge, Solaris/OpenSolaris has an issue where they have to
> automatically disable disk cache if ZFS is used on top of partitions
> instead of raw disks, but to my knowledge (I recall reading this from
> multiple reputable sources) this issue does not affect FreeBSD.

Does FreeBSD always enable the disk write caches?

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/