Fwd: Re: Strange ZFS performance

[Mikle Krutov]

On Mon, Apr 05, 2010 at 12:30:59AM -0700, Jeremy Chadwick wrote:
> I'm not sure why this mail didn't make it to the mailing list (I do see
> it CC'd).  The attachments are included inline.
> 
> SMART stats for the disk look fine, so the disk is unlikely to be
> responsible for this issue.  OP, could you also please provide the
> output of "atacontrol cap ad12"?
> 
> The arcstats entry that interested me the most was this (prior to the
> reboot):
> 
> > kstat.zfs.misc.arcstats.memory_throttle_count: 39958287
> 
> The box probably needs tuning in /boot/loader.conf to relieve this
> problem.
> 
> Below are values I've been using on our production systems for a month
> or two now.  These are for machines with 8GB RAM installed.  The OP may
> need to adjust the first two parameters (I tend to go with RAM/2 for
> vm.kmem_size and then subtract a bit more for arc_max (in this case
> 512MB less than kmem_size)).
> 
> # Increase vm.kmem_size to allow for ZFS ARC to utilise more memory.
> vm.kmem_size="4096M"
> vfs.zfs.arc_max="3584M"
> 
> # Disable ZFS prefetching
> # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
> # Increases overall speed of ZFS, but when disk flushing/writes occur,
> # system is less responsive (due to extreme disk I/O).
> # NOTE: 8.0-RC1 disables this by default on systems <= 4GB RAM anyway
> # NOTE: System has 8GB of RAM, so prefetch would be enabled by default.
> vfs.zfs.prefetch_disable="1"
> 
> # Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
> # should increase throughput and decrease the "bursty" stalls that
> # happen during immense I/O with ZFS.
> # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
> # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
> vfs.zfs.txg.timeout="5"
I've tried that tuning, now i have:
vm.kmem_size="1024M"
vfs.zfs.arc_max="512M"
vfs.zfs.txg.timeout="5"
No change in perfomance. Also, now reading directrly from hdd is slow, too (22-30MB/s), so that shows me that this could be some hardware problem (sata controller? but than the other disks were in same situation too. also, i've thought that that could be sata cable - and changed it - no speed after this).
Additional information for dd:
dd if=/dev/zero of=./file bs=4M count=10
41943040 bytes transferred in 0.039295 secs (1067389864 bytes/sec)

dd if=/dev/zero of=./file bs=4M count=20
83886080 bytes transferred in 0.076702 secs (1093663943 bytes/sec)

dd if=/dev/zero of=./file bs=4M count=30
125829120 bytes transferred in 0.114576 secs (1098216647 bytes/sec)

dd if=/dev/zero of=./file bs=4M count=40 
167772160 bytes transferred in 0.174362 secs (962206293 bytes/sec)

dd if=/dev/zero of=./file bs=4M count=50 
209715200 bytes transferred in 45.636052 secs (4595384 bytes/sec)
done without any delay, all the time zpool iostat & gstat are showing me 100KB/s-3MB/s
Any information i could provide to help us know what's the source of the problem?

-- 
Wbr,
Krutov Mikle