ZFS performance degradation over time

[Jonathan]

On 1/19/2010 12:01 PM, Jeremy Chadwick wrote:
> On Tue, Jan 19, 2010 at 11:40:50AM -0500, Garrett Moore wrote:
>> I've been watching my memory usage and I have no idea what is consuming
>> memory as 'Active'.
>>
>> Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB
>> buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was
>> again slow (<1MB/s). I tried killing rTorrent and no significant amount of
>> memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any
>> significant amount of memory, and I was definitely nowhere near 6500MB
>> usage.
>>
>> I tried running the perl oneliner again to hog a bunch of memory, and almost
>> all of the Active memory was IMMEDIATELY marked as Free, and my performance
>> was excellent again.

I'm having this same issue, although my performance does not go back to 
freshly booted levels it still goes from <1MBs to ~11MB/s after running 
that Perl one liner.  I'm running RELENG 8 as of about 2 days ago
FreeBSD 8.0-STABLE #2 r202777: Fri Jan 22 00:15:43 EST 2010 [...]  amd64

Just to be clear it also seems to be related to something rtorrent does 
while downloading torrents but it's not rtorrent itself using the memory 
because quitting rtorrent doesn't release more than 100MB of multiple GB 
of memory marked as active but running the Perl one liner does.

>> I'm not sure what in userland could be causing the issue. The only things
>> I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash,
>> Python, Perl, and SABNZBd. There is nothing that *should* be consuming any
>> serious amount of memory.
>
> I've two recommendations:
>
> 1) Have you considered "upgrading" to RELENG_8 (e.g. 8.0-STABLE) instead
> of sticking with 8.0-RELEASE?  There's been a recent MFC to RELENG_8
> which pertain to ARC drainage.  I'm referring to the commit labelled
> revision 1.22.2.2 (RELENG_8):

I definitely have that commit, see above.  I just checked my arc size 
and it's only using ~170 of a ~600MB limit.  This is after running the 
Perl script and without the strangely large amount of "active" memory, I 
forgot to run on before as well.

This machine is just a personal file server so I can restart it as 
needed for testing and I have serial console access to it if needed.

Thanks,
Jonathan