Still getting kmem exhausted panic

[Ben Kelly]

On Sep 28, 2010, at 12:30 PM, Andriy Gapon wrote:

> on 28/09/2010 18:50 Ben Kelly said the following:
>> 
>> On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote:
>>> Well, no time for me to dig through all that history. arc_max should be a
>>> hard limit and it is now. If it ever wasn't then it was a bug.
>> 
>> I believe the size of the arc could exceed the limit if your working set was
>> larger than arc_max.  The arc can't (couldn't then, anyway) evict data that is
>> still referenced.
> 
> I think that you are correct and I was wrong.
> ARC would still allocate a new buffer even if it's at or above arc_max and can not
> re-use any exisiting buffer.
> But I think that this is more likely to happen with "tiny" ARC size.  I have hard
> time imagining a workload at which gigabytes of data would be simultaneously and
> continuously used (see below for definition of "used").
> 
>> A contributing factor at the time was that the page daemon did not take into
>> account back pressure from the arc when deciding which pages to move from
>> active to inactive, etc.  So data was more likely to be referenced and
>> therefore forced to remain in the arc.
> 
> I don't think that this is what happened and I don't think that pagedaemon has
> anything to do with the discussed issue.
> I think that ARC buffers exist independently of pagedaemon and page cache.
> I think that they are held only during time when I/O is happening to or from them.


Hmm.  My server is currently idle with no I/O happening:

  kstat.zfs.misc.arcstats.c: 25165824
  kstat.zfs.misc.arcstats.c_max: 46137344
  kstat.zfs.misc.arcstats.size: 91863156

If what you say is true, this shouldn't happen, should it?  This system is an i386 machine with kmem max at 800M and arc set to 40M.  This is running head from April 6, 2010, so it is a bit old, though.

At one point I had patches running on my system that triggered the pagedaemon based on arc load and it did allow me to keep my arc below the max.  Or at least I thought it did.

In any case, I've never really been able to wrap my head around the VFS layer and how it interacts with zfs.  So I'm more than willing to believe I'm confused.  Any insights are greatly appreciated.

Thanks!

- Ben