ZFS performance degradation over time

[Alexander Leidinger]

Quoting Jeremy Chadwick <freebsd at jdc.parodius.com> (from Tue, 19 Jan  
2010 09:01:01 -0800):

> On Tue, Jan 19, 2010 at 11:40:50AM -0500, Garrett Moore wrote:
>> I've been watching my memory usage and I have no idea what is consuming
>> memory as 'Active'.
>>
>> Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB
>> buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was
>> again slow (<1MB/s). I tried killing rTorrent and no significant amount of
>> memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any
>> significant amount of memory, and I was definitely nowhere near 6500MB
>> usage.
>>
>> I tried running the perl oneliner again to hog a bunch of memory, and almost
>> all of the Active memory was IMMEDIATELY marked as Free, and my performance
>> was excellent again.
>>
>> I'm not sure what in userland could be causing the issue. The only things
>> I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash,
>> Python, Perl, and SABNZBd. There is nothing that *should* be consuming any
>> serious amount of memory.
>
> I've two recommendations:
>
> 1) Have you considered "upgrading" to RELENG_8 (e.g. 8.0-STABLE) instead
> of sticking with 8.0-RELEASE?  There's been a recent MFC to RELENG_8
> which pertain to ARC drainage.  I'm referring to the commit labelled
> revision 1.22.2.2 (RELENG_8):
>
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

This patch can be merged stand-alone if necessary, no need to go to  
RELENG_8 if there are reservations.

> 2) Have you tried using vfs.zfs.arc_max in loader.conf to limit the ARC
> size?  I'd recommend picking something like 1GB as a cap (your machine

Or even less... to be determined by experimenting.

> has 8GB total at present, if I remember right).  I believe long ago
> someone said this isn't an explicit hard limit on the maximum size of
> the ARC, but I believe this was during the RELENG_7 days and the ARC
> "stuff" on FreeBSD has changed since then.  I wish the tunables were
> better documented, or at least explained in detail (hello Wiki!).

The commit you refer to above is just doing this: limiting the arc  
more to the arc_max than it was the case before.

This patch is in 7-stable too (in case someone is interested).

Bye,
Alexander.

-- 
Johnson's First Law:
	When any mechanical contrivance fails, it will do so at the
	most inconvenient possible time.

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137