ZFS performance degradation over time

[Artem Belevich]

Keep an eye on ARC size and on active/inactive/cache/free memory lists:

sysctl kstat.zfs.misc.arcstats.size
sysctl vm.stats.vm.v_inactive_count
sysctl vm.stats.vm.v_active_count
sysctl vm.stats.vm.v_cache_count
sysctl vm.stats.vm.v_cache_count

ZFS performance does degrade a lot if ARC becomes too small. Writes
also get throttled if ZFS thinks the system is running low on memory.

One way to help the situation somewhat is to bump vfs.zfs.arc_min
tunable. It would make ZFS somewhal less eager to give up memory.
However, write throttling seems to rely on amount on memory on the
free list. FreeBSD appears to have somewhat different semantics for
'free" compared to solaris and that makes ZFS think that we're running
low on memory while there's plenty of it sitting on inactive/cached
lists and could be used.

One rather crude way to get ZFS back in shape in this situation is to
temporarily cause real memory shortage on the system. That would force
trimming of active/inactive lists (and ARC, too) but once it's done,
ARC would be free to grow and that may restore ZFS performance for a
while.

Following command will allocate about 8G of memory on my system --
enough to start swapout:
perl -e '$x="x"x3000000000'

--Artem


On Fri, Jan 8, 2010 at 8:31 AM, Garrett Moore <garrettmoore at gmail.com> wrote:
>
> No, I haven't isolated the cause to only be uptime related. In my original
> email I mentioned that "as suggested by someone in the thread, it's probably
> not directly related to system uptime, but instead related to usage - the
> more usage, the worse the performance."
>
> I've been starting my system with different combinations of applications
> running to see what access patterns cause the most slowdown. So far, I don't
> have enough data to give anything concrete.
>
> This weekend I'll try some tests such as the one you describe, and see what
> happens. I have a strong suspicion that rTorrent is to blame, since I
> haven't seen major slowdowns in the last few days with rTorrent not running.
> rTorrent preallocates the space needed for the file download (and I'm
> downloading large 4GB+ files using it), and then writes to them in an
> unpredictable pattern, so maybe ZFS doesn't like being touched this way?
>
>
>
> On Wed, Jan 6, 2010 at 2:21 PM, Ivan Voras <ivoras at freebsd.org> wrote:
>
> > On 3.1.2010 17:42, Garrett Moore wrote:
> >
> > > I'm having problems with ZFS performance. When my system comes up,
> > > read/write speeds are excellent (testing with dd if=/dev/zero
> > > of=/tank/bigfile and dd if=/tank/bigfile of=/dev/null); I get at least
> > > 100MB/s on both reads and writes, and I'm happy with that.
> > >
> > > The longer the system is up, the worse my performance gets. Currently my
> > > system has been up for 4 days, and read/write performance is down to
> > about
> > > 10MB/s at best.
> >
> > Are you sure you have isolated the cause to be only the uptime of the
> > machine? Is there no other change between the runs? E.g. did you stop
> > all other services and applications on the machine before doing the test
> > for the second time? Can you create a big file (2x memory size) when the
> > machine boots, measure the time to read it, then read it again after a
> > few days when you notice performance problems?
> >
> >
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> >
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"