ZFS performance of various vdevs (long post)

Bradley W. Dutton brad at duttonbros.com
Tue Jun 8 00:32:20 UTC 2010


Quoting Bob Friesenhahn <bfriesen at simple.dallas.tx.us>:

> On Mon, 7 Jun 2010, Bradley W. Dutton wrote:
>> So the normal vdev performs closest to raw drive speeds. Raidz1 is  
>> slower and raidz2 even more so. This is observable in the dd tests  
>> and viewing in gstat. Any ideas why the raid numbers are slower?  
>> I've tried to account for the fact that the raid vdevs have fewer  
>> data disks. Would a faster CPU help here?
>
> The sequential throughput on your new drives is faster than the old  
> drives, but it is likely that the seek and rotational latencies are  
> longer.  ZFS is transaction-oriented and must tell all the drives to  
> sync their write cache before proceeding to the next transaction  
> group.  Drives with more latency will slow down this step.   
> Likewise, ZFS always reads and writes full filesystem blocks  
> (default 128K) and this may cause more overhead when using raidz.

The details are little lacking on the Hitachi site but the  
HDS722020ALA330 says 8.2 seek time.
http://www.hitachigst.com/tech/techlib.nsf/techdocs/5F2DC3B35EA0311386257634000284AD/$file/USA7K2000_DS7K2000_OEMSpec_r1.2.pdf

The WDC drives say 8.9 so we should be in the same ballpark on seek times.
http://www.wdc.com/en/products/products.asp?driveid=399

I thought the NCQ vs no NCQ might tip the scales in favor of the  
Hitachi array as well.

Are there any tools to check the latencies of the disks?


> Using 'dd' from /dev/zero is not a very good benchmark test since  
> zfs could potentially compress zero-filled blocks down to just a few  
> bytes (I think recent versions of zfs do this) and of course Unix  
> supports files with holes.

I know it's pretty simple but for checking throughput I thought it  
would be ok. I don't have compression on and based on the drive lights  
and gstat, the drives definitely aren't idle.


> The higher CPU usage might be due to the device driver or the  
> interface card being used.

Definitely a plausible explanation. If this was the case would the 8  
parallel dd processes exhibit the same behavior? or is the type of IO  
affecting how much CPU the driver is using?


> If you could afford to do so, you will likely see considerably  
> better performance by using mirrors instead of raidz since then 128K  
> blocks will be sent to each disk and with fewer seeks.

I agree with you but at this poing I value the extra space more as I  
don't have a lot of random IO. I read the following and decided to  
stick with raidz2 when ditching my old raidz1 setup:
http://blogs.sun.com/roch/entry/when_to_and_not_to


Thanks for the feedback,
Brad




More information about the freebsd-fs mailing list