zfs arc and amount of wired memory

Jeremy Chadwick freebsd at jdc.parodius.com
Thu Feb 9 00:28:39 UTC 2012

On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote:
> Andriy Gapon wrote:
> >on 08/02/2012 12:31 Eugene M. Zheganin said the following:
> >>Hi.
> >>
> >>On 08.02.2012 02:17, Andriy Gapon wrote:
> >>>[output snipped]
> >>>
> >>>Thank you.  I don't see anything suspicious/unusual there.
> >>>Just case, do you have ZFS dedup enabled by a chance?
> >>>
> >>>I think that examination of vmstat -m and vmstat -z outputs may provide some
> >>>clues as to what got all that memory wired.
> >>>
> >>Nope, I don't have deduplication feature enabled.
> >
> >OK.  So, did you have a chance to inspect vmstat -m and vmstat -z?
> >
> >>By the way, today, after eating another 100M of wired memory this server hanged
> >>out with multiple non-stopping messages
> >>
> >>swap_pager: indefinite wait buffer
> >>
> >>Since it's swapping on zvol, it looks to me like it could be the mentioned in
> >>another thread here ("Swap on zvol - recommendable?") resource starvation issue;
> >>may be it happens faster when the ARC isn't limited.
> >
> >It could be very well possible that swap on zvol doesn't work well when the
> >kernel itself is starved on memory.
> >
> >>So I want to ask - how to report it and what should I include in such pr ?
> >
> >I am leaving swap-on-zvol issue aside.  Your original problem doesn't seem to be
> >ZFS-related.  I suspect that you might be running into some kernel memory leak.
> >  If you manage to reproduce the high wired value again, then vmstat -m and
> >vmstat -z may provide some useful information.
> >
> >In this vein, do you use any out-of-tree kernel modules?
> >Also, can you try to monitor your system to see when wired count grows?
> I am seeing something similar on one of our machine. This is old 7.3
> with ZFS v13, that's why I did not reported it.
> The machine is used as storage for backups made by rsync. All is
> running fine for about 107 days. Then backups are slower and slower
> because of some strange memory situation.
> Mem: 15M Active, 17M Inact, 3620M Wired, 420K Cache, 48M Buf, 1166M Free
> ARC Size:
>          Current Size:             1769 MB (arcsize)
>          Target Size (Adaptive):   512 MB (c)
>          Min Size (Hard Limit):    512 MB (zfs_arc_min)
>          Max Size (Hard Limit):    3584 MB (zfs_arc_max)
> The target size is going down to the min size and after few more
> days, the system is so slow, that I must reboot the machine. Then it
> is running fine for about 107 days and then it all repeat again.
> You can see more on MRTG graphs
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
> You can see links to other useful informations on top of the page
> (arc_summary, top, dmesg, fs usage, loader.conf)
> There you can see nightly backups (higher CPU load started at
> 01:13), otherwise the machine is idle.
> It coresponds with ARC target size lowering in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
> And with ARC metadata cache overflowing the limit in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
> I don't know what's going on and I don't know if it is something
> know / fixed in newer releases. We are running a few more ZFS
> systems on 8.2 without this issue. But those systems are in
> different roles.

This sounds like the... damn, what is it called... some kind of internal
"counter" or "ticks" thing within the ZFS code that was discovered to
only begin happening after a certain period of time (which correlated to
some number of days, possibly 107).  I'm sorry that I can't be more
specific, but it's been discussed heavily on the lists in the past, and
fixes for all of that were committed to RELENG_8.  I wish I could
remember the name of the function or macro or variable name it pertained
to, something like LTHAW or TLOCK or something like that.  I would say
"I don't know why I can't remember", but I do know why I can't remember:
because I gave up trying to track all of these problems.

Does someone else remember this issue?  CC'ing Martin who might remember
for certain.

| Jeremy Chadwick                                 jdc at parodius.com |
| Parodius Networking                     http://www.parodius.com/ |
| UNIX Systems Administrator                 Mountain View, CA, US |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

More information about the freebsd-stable mailing list