ZFS performance strangeness

[Damien Fleuriot]

On 4/12/11 1:33 PM, Lars Wilke wrote:
> Hi,
> 
> There are quite a few threads about ZFS and performance difficulties,
> but i did not find anything that really helped :)
> Therefor any advice would be highly appreciated.
> I started to use ZFS with 8.1R, only tuning i did was setting
> 
> vm.kmem_size_scale="1"
> vfs.zfs.arc_max="40000M"
> 
> The machines are supermicro boards with 48 GB ECC RAM and 15k RPM SAS
> drives. Local read/write performance was and is great.
> But exporting via NFS was a mixed bag in 8.1R.
> Generally r/w speed over NFS was ok, but large reads or writes took
> ages. Most of the reads and writes were small, so i did not bother.
> 
> Now i upgraded one machine to 8.2R and i get very good write performance
> over NFS but read performance drops to a ridiciously low value, around
> 1-2 MB/s. While writes are around 100MB/s. The network is a dedicated
> 1GB Ethernet. The zpool uses RAIDZ1 over 7 drives, one vdev.
> The filesystem has compression enabled. Turning it off made no
> difference AFAICT
> 
> Now i tried a few of the suggested tunables and my last try was this
> 
> vfs.zfs.txg.timeout="5"
> vfs.zfs.prefetch_disable="1"
> vfs.zfs.txg.synctime=2
> fs.zfs.vdev.min_pending="1"
> fs.zfs.vdev.max_pending="1"
> 
> still no luck. Writting is fast, reading is not. Even with enabled
> prefetching. The only thing i noticed is, that reading for example 10MB
> is fast (on a freshly mounted fs) but when reading larger amounts, i.e.
> couple hundred MBs, the performance drops and zpool iostat or iostat -x
> show that there is not much activity on the zpool/hdds.
> 
> It seems as if ZFS does not care that someone wants to read data, also idle
> time of the reading process happily ticks up and gets higher and higher!?
> When trying to access the file during this time, the process blocks and
> sometimes is difficult to kill, i.e. ls -la on the file.
> 
> I read and write with dd and before read tests i umount and mount the
> NFS share again.
> 
> dd if=/dev/zero of=/mnt/bla size=1M count=X
> dd if=/mnt/bla of=/dev/null size=1M count=Y
> 
> mount is done with this options from two centos 5 boxes:
> rw,noatime,tcp,bg,intr,hard,nfsvers=3,noacl,nocto
> 
> thanks
>    --lars
> 
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"



I refer you to this post by Jeremy Chadwick with tuning values *AND*
their actual explanation.

http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/061642.html