ZFS read performance

[markham breitbach]

   Hi,
   I'm trying to figure out a ZFS read performance issue that I am seeing
   on FreeBSD9.2 (amd64).
   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
   K8-class CPU)
   real memory  = 4294967296 (4096 MB)
   avail memory = 4059762688 (3871 MB)
   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
   with the drives installed JBOD.  These are intended for use as a
   storage pool.  The main system drives are using the onboard SATA
   controller for the system.

   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
   Write speeds using dd as follows:
   WRITE:
   dd if=/dev/zero of=/vol0/test bs=1m count=10000
   READ
   dd if=/vol0/test of=/dev/null
   When I create a zfs pool with a single drive:
   zpool create vol0 /dev/mfisyspd0
   using dd again, I can write ~175MBps, but my read speed is only about
   60MBps.
   I have set
   vfs.zfs.prefetch_disable = 0
   and this improves reads to about 70MBps, but no much beyond that.
   Now, this gets even more interesting.
   If I add a drive to the pool:
   zpool add vol0 /dev/mfisyspd1
   My write speeds increase to ~280MBps, but my read speed is still about
   60MBps.
   Using gstat, I can see that the load is shared pretty equally amongst
   the drives.
   This trend continues with the addition of a 3rd and 4th drive with
   write speeds ramping up to about 500MBps (I didn't test past that), but
   read speeds stuck around 60MBps.
   I have done similar testing under 9.1 making sure to align 4K sectors
   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
   I have looked through the wiki and handbook for tuning and done some
   googling around, but I'm pretty much out of ideas now.  Everything I
   have read seems to indicate that ZFS should run fine on 4GB without any
   tuning, and at this point I haven't even begun any serious stress
   testing.
   Any ideas of where to go from here would be greatly appreciated.
   Thanks,
   -Markham

   ---
   [1]Markham Breitbach
   Network Operations
   SSi   People, Ideas, Technology
   - - - - - - - - - - - - - - - - - - - - -
   +1 867 669 7500 work
   +1 867 669 7510 fax
   [2]markham_breitbach at ssimicro.com
   [3]www.ssimicro.com
   356B Old Airport Road
   Yellowknife , NT X1A 3T4
   Canada
   - - - - - - - - - - - - - - - - - - - - -
   Visit some of our other networks
   [4]www.qiniq.com   &   [5]www.airware.ca

References

   1. http://www.ssimicro.com/
   2. mailto:markham_breitbach at ssimicro.com
   3. http://www.ssimicro.com/
   4. http://www.qiniq.com/
   5. http://www.airware.ca/