ZFS performance degradation over time

[Garrett Moore]

I upgraded my system to 8GB of ram to see if that would help. It hasn't made
much of a difference. After having rTorrent running for a while, my
performance again tanked. Around 6.5GB of memory was showing as 'Active'
according to top. Copying a file from the zpool to another location on the
zpool had the following performance (from zpool iostat 1):

tank        2.57T  8.31T     11    215   273K   745K
tank        2.57T  8.31T     22     97   816K   540K
tank        2.57T  8.31T      6     67  13.5K   887K
tank        2.57T  8.31T      5    101   639K   190K
tank        2.57T  8.31T     14     81  1.14M   154K
tank        2.57T  8.31T      6    152  12.5K   897K
tank        2.57T  8.31T     11    153   143K   790K
tank        2.57T  8.31T      3    172   134K   566K
tank        2.57T  8.31T      3    105  7.48K   699K
tank        2.57T  8.31T      5    136   138K   383K

Combined read/write of <2MB/s -- pretty pathetic. I then tried Artem's
trick, running ` perl -e '$x="x"x3000000000'` to force a swapout. This
command completed in about 10 seconds, and I then had >5GB of memory showing
as 'Free' according to top. Checking zpool iostat 1 again showed:

tank        2.57T  8.31T    375    477  46.1M  57.9M
tank        2.57T  8.31T     18    472  1.75M  44.8M
tank        2.57T  8.31T    129      0  16.1M      0
tank        2.57T  8.31T    428      0  53.2M      0
tank        2.57T  8.31T    262    947  31.8M   103M
tank        2.57T  8.30T     80    105  9.61M   196K
tank        2.57T  8.30T    612      0  75.8M      0
tank        2.57T  8.30T    155    951  18.4M   103M
tank        2.57T  8.30T    662      0  82.1M      0
tank        2.57T  8.30T    176    945  21.1M   103M

Which is obviously much better, and a respectable rate of performance. This
was all done during the same single `copy` command - I did not stop/restart
the copy when running the perl oneliner.

So it seems based on this that ZFS is keeping too much data cached and not
being smart about when to release 'old' cache entries in favour of new ones.

Any suggestions at this point? The only tuning I have done so far is to
disable prefetch, since my primary usage is streaming HD media, and prefetch
has been known to cause problems in that situation.