ZFS performance degradation over time
Jeremy Chadwick
freebsd at jdc.parodius.com
Tue Jan 19 17:01:04 UTC 2010
On Tue, Jan 19, 2010 at 11:40:50AM -0500, Garrett Moore wrote:
> I've been watching my memory usage and I have no idea what is consuming
> memory as 'Active'.
>
> Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB
> buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was
> again slow (<1MB/s). I tried killing rTorrent and no significant amount of
> memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any
> significant amount of memory, and I was definitely nowhere near 6500MB
> usage.
>
> I tried running the perl oneliner again to hog a bunch of memory, and almost
> all of the Active memory was IMMEDIATELY marked as Free, and my performance
> was excellent again.
>
> I'm not sure what in userland could be causing the issue. The only things
> I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash,
> Python, Perl, and SABNZBd. There is nothing that *should* be consuming any
> serious amount of memory.
I've two recommendations:
1) Have you considered "upgrading" to RELENG_8 (e.g. 8.0-STABLE) instead
of sticking with 8.0-RELEASE? There's been a recent MFC to RELENG_8
which pertain to ARC drainage. I'm referring to the commit labelled
revision 1.22.2.2 (RELENG_8):
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
2) Have you tried using vfs.zfs.arc_max in loader.conf to limit the ARC
size? I'd recommend picking something like 1GB as a cap (your machine
has 8GB total at present, if I remember right). I believe long ago
someone said this isn't an explicit hard limit on the maximum size of
the ARC, but I believe this was during the RELENG_7 days and the ARC
"stuff" on FreeBSD has changed since then. I wish the tunables were
better documented, or at least explained in detail (hello Wiki!).
Finally:
Does anyone have reservations about me crossposting this thread to
freebsd-fs at freebsd.org, to get some additional eyes?
Reports like this often scare/worry those of us who run servers. :-)
> On Mon, Jan 18, 2010 at 11:29 AM, Norbert Papke <npapke at acm.org> wrote:
>
> > On January 17, 2010, Garrett Moore wrote:
> > > I upgraded my system to 8GB of ram to see if that would help. It hasn't
> > > made much of a difference. After having rTorrent running for a while, my
> > > performance again tanked. Around 6.5GB of memory was showing as 'Active'
> > > according to top.
> >
> > 6.5GB of "active" memory seems to imply that a user process is growing or a
> > large number of user processes are being created. I would expect ZFS's
> > cache
> > to increase the size of "wired" memory.
> >
> > Sorry, I have not followed this thread closely. Are you sure that the
> > degradation is ZFS related? Could it be caused by, for instance, a
> > userland
> > memory leak? What happens to active memory when you restart rtorrent?
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
More information about the freebsd-stable
mailing list