[ZFS] Raid 10 performance issues

[Pierre Lemazurier]

I add my /boot/loader.conf for more information :

zfs_load="YES"
vm.kmem_size="22528M"
vfs.zfs.arc_min="20480M"
vfs.zfs.arc_max="20480M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"

Le 07/06/2013 17:07, Pierre Lemazurier a écrit :
> Hi, i think i suffer of write and read performance issues on my zpool.
>
> About my system and hardware :
>
> uname -a
> FreeBSD bsdnas 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4
> 09:23:10 UTC 2012
> root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>
> sysinfo -a : http://www.privatepaste.com/b32f34c938
>
> - 24 (4gbx6) GB DDR3 ECC :
> http://www.ec.kingston.com/ecom/configurator_new/partsinfo.asp?ktcpartno=KVR16R11D8/4HC
>
> - 14x this drive :
> http://www.wdc.com/global/products/specs/?driveID=1086&language=1
> - server :
> http://www.supermicro.com/products/system/1u/5017/sys-5017r-wrf.cfm?parts=show
>
> - CPU :
> http://ark.intel.com/fr/products/64594/Intel-Xeon-Processor-E5-2620-15M-Cache-2_00-GHz-7_20-GTs-Intel-QPI
>
> - chassis :
> http://www.supermicro.com/products/chassis/4u/847/sc847e16-rjbod1.cfm
> - HBA sas connector :
> http://www.lsi.com/products/storagecomponents/Pages/LSISAS9200-8e.aspx
> - Cable between chassis and server :
> http://www.provantage.com/supermicro-cbl-0166l~7SUPA01R.htm
>
> I use this command for test write speed :dd if=/dev/zero of=test.dd
> bs=2M count=10000
> I use this command for test read speed :dd if=test.dd of=/dev/null bs=2M
> count=10000
>
> Of course no compression on zfs dataset.
>
> Test on one of this disk format with UFS :
>
> Write :
> some gstat raising : http://www.privatepaste.com/dd31fafaa6
> speed around 140 mo/s and something like 1100 iops
> dd result : 20971520000 bytes transferred in 146.722126 secs (142933589
> bytes/sec)
>
> Read :
> I think I read on RAM (20971520000 bytes transferred in 8.813298 secs
> (2379531480 bytes/sec)).
> Then I make the test on all the drive (dd if=/dev/gpt/disk14.nop
> of=/dev/null bs=2M count=10000)
> some gstat raising : http://www.privatepaste.com/d022b7c480
> speed around 140 mo/s again an near 1100+ iops
> dd reslut : 20971520000 bytes transferred in 142.895212 secs (146761530
> bytes/sec)
>
>
> ZFS - I make my zpool on this way : http://www.privatepaste.com/e74d9cc3b9
>
> zpool status : http://www.privatepaste.com/0276801ef6
> zpool get all : http://www.privatepaste.com/74b37a2429
> zfs get all : http://www.privatepaste.com/e56f4a33f8
> zfs-stats -a : http://www.privatepaste.com/f017890aa1
> zdb : http://www.privatepaste.com/7d723c5556
>
> With this setup I hope to have near 7x more speed for write and near 14x
> for
> read than the UFS device alone. Then for be realistic, something like
> 850 mo/s for write and 1700 mo/s for read.
>
>
> ZFS – test :
>
> Write :
> gstat raising : http://www.privatepaste.com/7cefb9393a
> zpool iostat -v 1 of a fastest try : http://www.privatepaste.com/8ade4defbe
> dd result : 20971520000 bytes transferred in 54.326509 secs (386027381
> bytes/sec)
>
> 386 mo/s more than twice less than I expect.
>
>
> Read :
> I export and import the pool for limit the ARC effect. I don't know how
> to do better, I hope that sufficient.
> gstat raising : http://www.privatepaste.com/130ce43af1
> zpool iostat -v 1 : http://privatepaste.com/eb5f9d3432
> dd result : 20971520000 bytes transferred in 30.347214 secs (691052563
> bytes/sec)
>
> 690 mo/s 2,5x less than I expect.
>
>
> It's appear to not be an hardware issue, when I do a dd test of each
> whole disk at the same time with the command dd if=/dev/gpt/diskX
> of=/dev/null bs=1M count=10000, I have this gstat raising :
> http://privatepaste.com/df9f63fd4d
>
> Near 130 mo/s for each device, something like I expect.
>
> In your opinion where the problem come from ?
>
>
> Forgive me for my English, please keep easy language, i'm not realy easy
> with English.
> I can give you more information if you need.
>
> Many thanks for your help.
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"