ZFS: unlimited arc cache growth?
ben at wanderview.com
Mon Apr 20 05:25:36 UTC 2009
On Apr 18, 2009, at 5:17 PM, Ben Kelly wrote:
> After the rsync completed my machine slowly evicts buffers until its
> back down to about twice arc_c. There was one case, however, where
> I saw it stop at about four times arc_c. In that case it was
> failing to evict buffers due to a missed lock. Its not clear yet if
> it was a buffer lock or hash lock. When this happens you'll see the
> arcstats.mutex_missed sysctl go up. I'm going to see if I can track
> down why this is occuring under idle conditions. That seems
> suspicious to me.
Sorry to reply to my own mail, but I found some more information I
thought I would share.
First, the missed mutex problem was an error on my part. I had
accidentally deleted a rather important line when I was instrumenting
the code earlier. Once this was replaced that missed mutex count
dropped back to a more reasonable level.
Next, the arcstats.size value is not strictly the amount of cached
data. It represents a combination of cached buffers, actively
referenced buffers, and "other" data. In this case "other" data is
things like dnode structures that are directly allocated using
kmem_cache_alloc() and simply tacked on to the ARC accounting variable
using arc_space_consume(). At this point I don't think the ARC has a
way of signaling these "other" data users of memory pressure.
The actual amount of memory the ARC has cached that can actually be
freed is limited to buffers it internally allocated that have zero
active references. This consists of the data and metadata lists for
the MRU and MFU caches.
On my server right now I have an arc_c_max of about 40MB. After
running a simple find(1) over /usr/src I ended up with the following
arcstats.size = 132MB
anonymous inflight buffers = 212KB
MRU referenced buffers = 80MB
MFU referenced buffers = 1KB
dbuf structure "other" data = 8MB
dnode structure "other" data = 25MB
unknown "other" data (probably dbuf related) ~= 18MB
evictable buffer data = 3KB
So right now the ARC has done the best it can to free up data. If you
define the cache as storing only inactive data, then basically the ARC
has emptied the cache completely. This just isn't visible from the
exported arcstats.size variable.
I guess there is some question as to whether data is being referenced
longer than it needs to be by outside consumers.
Anyway, just thought I would share what I found. At this point it
doesn't look like tweaking limits will really help. Also, my previous
idea that the inactive buffers were being prevented from eviction for
too long was incorrect.
If anyone is interested I can put together a patch that exports the
amount of evictable data in the cache.
More information about the freebsd-current