ZFS performance of various vdevs (long post)

[Bradley W. Dutton]

Quoting Jeremy Chadwick <freebsd at jdc.parodius.com>:

>> On Mon, Jun 07, 2010 at 05:32:18PM -0700, Bradley W. Dutton wrote:
>> I know it's pretty simple but for checking throughput I thought it
>> would be ok. I don't have compression on and based on the drive
>> lights and gstat, the drives definitely aren't idle.
>
> Try disabling prefetch (you have it enabled) and try setting
> vfs.zfs.txg.timeout="5".  Some people have reported a "sweet spot" with
> regards to the last parameter (needing to be adjusted if your disks are
> extremely fast, etc.), as otherwise ZFS would be extremely "bursty" in
> its I/O (stalling/deadlocking the system at set intervals).  By
> decreasing the value you essentially do disk writes more regularly (with
> less data), and depending upon the load and controller, this may even
> out performance.

I tested some of these settings. With the timeout set to 5 not much  
changed write wise. (keep in mind these results are the Nvidia/WDRE2  
combo):

With txg=5 and prefetch disabled I saw read speeds go down considerably:
# normal/jbod txg=5 no prefetch
zpool create bench /dev/ad4 /dev/ad6 /dev/ad10 /dev/ad12 /dev/ad14
dd if=/bench/test.file of=/dev/null bs=1m
12582912000 bytes transferred in 59.330330 secs (212082286 bytes/sec)
compared to
12582912000 bytes transferred in 34.668165 secs (362952928 bytes/sec)

zpool create bench raidz /dev/ad4 /dev/ad6 /dev/ad10 /dev/ad12 /dev/ad14
dd if=/bench/test.file of=/dev/null bs=1m
12582912000 bytes transferred in 71.135696 secs (176886046 bytes/sec)
compared to
12582912000 bytes transferred in 45.825533 secs (274582993 bytes/sec)

Running the same tests on the raidz2 Supermicro/Hitachi setup didn't  
yield any difference in writes, the reads were slower:
zpool create tank raidz2 /dev/da0 /dev/da1 /dev/da2 /dev/da3 /dev/da4  
/dev/da5 /dev/da6 /dev/da7
dd if=/tank/test.file of=/dev/null bs=1m
12582912000 bytes transferred in 44.118409 secs (285207745 bytes/sec)
compared to
12582912000 bytes transferred in 32.911291 secs (382328118 bytes/sec)

I rebooted and reran these numbers just to make sure they were consistent.


>> >The higher CPU usage might be due to the device driver or the
>> >interface card being used.
>>
>> Definitely a plausible explanation. If this was the case would the 8
>> parallel dd processes exhibit the same behavior? or is the type of
>> IO affecting how much CPU the driver is using?
>
> It would be the latter.
>
> Also, I believe this Supermicro controller has been discussed in the
> past.  I can't remember if people had outright failures/issues with it
> or if people were complaining about sub-par performance.  I could also
> be remembering a different Supermicro controller.
>
> If I had to make a recommendation, it would be to reproduce the same
> setup on a system using an Intel ICH9/ICH9R or ICH10/ICH10R controller
> in AHCI mode (with ahci.ko loaded, not ataahci.ko) and see if things
> improve.  But start with the loader.conf tunables I mentioned above --
> segregate each test.
>
> I would also recommend you re-run your tests with a different blocksize
> for dd.  I don't know why people keep using 1m (Linux websites?).  Test
> the following increments: 4k, 8k, 16k, 32k, 64k, 128k, 256k.  That's
> about where you should stop.

I tested with 8, 16, 32, 64, 128, 1m and the results all looked  
similar. As such I stuck with bs=1m because it's easier to change count.


> Otherwise, consider installing ports/benchmarks/bonnie++ and try that.
> That will also get you concurrent I/O tests, I believe.

I may give this a shot but I'm most interested in less concurrency as  
I have larger files with only a couple of readers/writers. As Bob  
noted a bunch of mirrors in the pool would definitely be faster for  
concurrent IO.


Thanks for the help,
Brad