Still getting kmem exhausted panic

[Jeremy Chadwick]

On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote:
> on 28/09/2010 14:50 Jeremy Chadwick said the following:
> > I believe the trick -- Andriy, please correct me if I'm wrong -- is the
> 
> Wouldn't hurt to CC me, so that I could do it :-)
> 
> > tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high
> > watermark".
> 
> Not sure what you mean here.
> What is hard limit, what is high watermark, what is the difference and when is
> "now"? :-)

There was some speculation on the part of users a while back which lead
to this understanding.  Folks were seeing actual ARC usage higher than
what vfs.zfs.arc_max was set to (automatically or administratively).  I
believe it started here:

http://www.mailinglistarchive.com/freebsd-current@freebsd.org/msg28884.html

With the "high-water mark" statements being here:

http://www.mailinglistarchive.com/freebsd-current@freebsd.org/msg28887.html
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2010-04/msg00129.html

The term implies that there is not an explicitly hard limit on the ARC
utilisation/growth.  As stated in the unix.derkeiler.com URL above, this
behaviour was in fact changed.  Why/when/how?  I had to go digging up
the commits -- this took me some time.  Here they are, labelled r197816,
for RELENG_8 and RELENG_7 respectively.  These were both committed on
2010/01/08 UTC:

http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.22.2.2
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.15.2.6

In HEAD/CURRENT (yet to be MFC'd), it looks like above code got removed
on 2010/09/17 UTC, citing they should be "enforced by actual
calculations of delta":

http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.46
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.45

So what's this "delta" code piece that's mentioned?  That appears to be
have been committed to RELENG_8 on 2010/05/24 UTC (thus, between the
above two dates):

http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.22.2.4

(Side note: the "delta stuff" was never committed to RELENG_7 -- and
that's fine.  I'm pointing this out not out of retaliation or insult,
but because people will almost certainly Google, find this post, and
wonder if their 7.x machines might be affected.)

This situation with the ARC, and all its changes over time, is one of
the reasons why I rant aggressively about the need for more
communication transparency (re: what the changes actually affect).  Most
SAs and users don't follow commits.

> I believe that "the trick" is to set vm.kmem_size high enough, eitehr using this
> tunable or vm.kmem_size_scale.

Thanks for the clarification.  I just wish I knew how vm.kmem_size_scale
fit into the picture (meaning what it does, etc.).  The sysctl
description isn't very helpful.  Again, my lack of VM knowledge...

> > However, I believe there have been occasional reports of exhaustion
> > panics despite both of these being set[1].  Those reports are being
> > investigated on an individual basis.
> 
> I don't believe that the report that you quote actually demonstrates what you say
> it does.
> Two quotes from it:
> "During these panics no tuning or /boot/loader.conf values where present."
> "Only after hitting this behaviour yesterday i created boot/loader.conf"
> 
> > [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059109.html

You're right -- the report I'm quoting is not the one I thought it was.
I'll see if I can dig up the correct mail/report.  It could be that I'm
thinking of something quite old (pre-ARC-changes (see above
paragraphs)).  I can barely keep track of all the changes going on.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |