ZFS read performance

[Steven Hartland]

First off avoid using mfi its a RAID card which adds a extra layer
you dont want even in jbob mode.

Whats the mfi max_cmd set to? If you havent already set it to -1
/boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max.

Next 4GB on a machine avoid it if possible, thats whats disabling
prefetch by default, I'd recommend a min of 8Gb more if posslbe.

When you "add" a disk your creating a stripe which is bad news if
any single disk fails, so avoid that at all costs.

Finally your issue with read performance is likely your block size,
you set it for write to 1mb (which is reasonable) but you didn't
on read so will be 512b which will likely cpu limit in the dd
process instead of bottlenecking on the disk / FS performance.

    Regards
    Steve
----- Original Message ----- 
From: "markham breitbach" <markham_breitbach at ssimicro.com>
To: <freebsd-performance at freebsd.org>
Sent: Wednesday, January 29, 2014 12:02 AM
Subject: ZFS read performance


>   Hi,
>   I'm trying to figure out a ZFS read performance issue that I am seeing
>   on FreeBSD9.2 (amd64).
>   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
>   K8-class CPU)
>   real memory  = 4294967296 (4096 MB)
>   avail memory = 4059762688 (3871 MB)
>   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
>   with the drives installed JBOD.  These are intended for use as a
>   storage pool.  The main system drives are using the onboard SATA
>   controller for the system.
> 
>   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
>   Write speeds using dd as follows:
>   WRITE:
>   dd if=/dev/zero of=/vol0/test bs=1m count=10000
>   READ
>   dd if=/vol0/test of=/dev/null
>   When I create a zfs pool with a single drive:
>   zpool create vol0 /dev/mfisyspd0
>   using dd again, I can write ~175MBps, but my read speed is only about
>   60MBps.
>   I have set
>   vfs.zfs.prefetch_disable = 0
>   and this improves reads to about 70MBps, but no much beyond that.
>   Now, this gets even more interesting.
>   If I add a drive to the pool:
>   zpool add vol0 /dev/mfisyspd1
>   My write speeds increase to ~280MBps, but my read speed is still about
>   60MBps.
>   Using gstat, I can see that the load is shared pretty equally amongst
>   the drives.
>   This trend continues with the addition of a 3rd and 4th drive with
>   write speeds ramping up to about 500MBps (I didn't test past that), but
>   read speeds stuck around 60MBps.
>   I have done similar testing under 9.1 making sure to align 4K sectors
>   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
>   I have looked through the wiki and handbook for tuning and done some
>   googling around, but I'm pretty much out of ideas now.  Everything I
>   have read seems to indicate that ZFS should run fine on 4GB without any
>   tuning, and at this point I haven't even begun any serious stress
>   testing.
>   Any ideas of where to go from here would be greatly appreciated.
>   Thanks,
>   -Markham
> 
>   ---
>   [1]Markham Breitbach
>   Network Operations
>   SSi   People, Ideas, Technology
>   - - - - - - - - - - - - - - - - - - - - -
>   +1 867 669 7500 work
>   +1 867 669 7510 fax
>   [2]markham_breitbach at ssimicro.com
>   [3]www.ssimicro.com
>   356B Old Airport Road
>   Yellowknife , NT X1A 3T4
>   Canada
>   - - - - - - - - - - - - - - - - - - - - -
>   Visit some of our other networks
>   [4]www.qiniq.com   &   [5]www.airware.ca
> 
> References
> 
>   1. http://www.ssimicro.com/
>   2. mailto:markham_breitbach at ssimicro.com
>   3. http://www.ssimicro.com/
>   4. http://www.qiniq.com/
>   5. http://www.airware.ca/
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.