ZFS performance degradation over time

[Jeremy Chadwick]

On Fri, Jan 08, 2010 at 04:23:03PM -0500, Jonathan Noack wrote:
> On Fri, January 8, 2010 11:31, Garrett Moore wrote:
> > No, I haven't isolated the cause to only be uptime related. In my original
> > email I mentioned that "as suggested by someone in the thread, it's
> > probably
> > not directly related to system uptime, but instead related to usage - the
> > more usage, the worse the performance."
> >
> > I've been starting my system with different combinations of applications
> > running to see what access patterns cause the most slowdown. So far, I
> > don't
> > have enough data to give anything concrete.
> >
> > This weekend I'll try some tests such as the one you describe, and see
> > what
> > happens. I have a strong suspicion that rTorrent is to blame, since I
> > haven't seen major slowdowns in the last few days with rTorrent not
> > running.
> > rTorrent preallocates the space needed for the file download (and I'm
> > downloading large 4GB+ files using it), and then writes to them in an
> > unpredictable pattern, so maybe ZFS doesn't like being touched this way?
> 
> Probably unrelated, but this prefetch issue results in a slowdown:
> http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007481.html

With the introduction of 8.0-RC1, FreeBSD explicitly disables prefetch
if your system has less than 4GB usable[1] memory.  The code does print
a warning/FYI message when this condition is met[2].

You can disable the warning message by setting prefetch_disable="1" in
loader.conf (and this naturally means prefetch is disabled).  You can
also explicitly enable prefetch by setting the value to "1", and this
trumps the how-much-usable-RAM check.

The OP doesn't disclose if he's enabled prefetch, so I assume he hasn't,
which means he shouldn't be susceptible to the bug described above.

All that said -- I know what the OP is referring to, as I've seen it
myself (on RELENG_7, and possibly early releases of 8.0).  The only
way to relieve the pain, AFAIK, is to reboot.

I do see some MFC's done about 13 hours ago to RELENG_7 and RELENG_8
that talk about the ARC and "paging pressure", which to me means
decreased performance when it occurs... or maybe it helps with the kmem
exhaustion problem?  The brief description in the commit is simply not
enough to suffice; it's almost like we need a "FreeBSD ZFS Newsletter"
that documents what all the changes are that get committed, what they
fix, and what's being worked on/tested in HEAD (for potential MFC).

It would also be good to get some concise documentation with regards to
what all the kstat.zfs.misc.arcstats.* counters mean and represent;
admins like myself would love to track/graph these for helping correlate
performance or stability issues, but we don't even know what they
represent.  For example, just earlier this week I read a semi-recent
message talking about how a large kstat.zfs.misc.arcstats.evict_skip
counter indicates something bad, yet I have no idea what "evict skip"
means in reference to the ZFS core/model itself.


[1]: Because of #2 (see above/below), I had to analyse the code.  My
system was amd64 + 4GB RAM yet ZFS was telling me the system did not
have 4GB of RAM thus had disabled prefetch.  My analysis is near the
bottom of my blog post:

http://koitsu.wordpress.com/2009/10/12/testing-out-freebsd-8-0-rc1/

[2]: The grammar of the message printed has gone through numerous
revisions (I forget the exact number, but *at least* 2), and the most
recent one is still too ambiguous, resulting in reader confusion.  In my
above blog post, I propose what the kernel message should read (in my
opinion), which is significantly less ambiguous and hopefully won't
cause "but I do have 4GB!" confusion.

[3]: src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

http://freshbsd.org/2010/01/08/09/59/13 -- RELENG_8
http://freshbsd.org/2010/01/08/11/06/13 -- RELENG_7

It sure would be useful if these source files had $FreeBSD$ ID tags in
them, rather than having to grep /var/db/sup.  I swore there was some
command which did this, but I might be thinking of object files.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |