ZFS: unlimited arc cache growth?

[Ben Kelly]

On Apr 17, 2009, at 12:28 PM, Marius Nünnerich wrote:

> On Fri, Apr 17, 2009 at 16:18, Bernd Walter  
> <ticso at cicely7.cicely.de> wrote:
>> On Fri, Apr 17, 2009 at 02:50:24PM +0200, Alexander Leidinger wrote:
>>> Hi,
>>>
>>> to fs@, please CC me, as I'm not subscribed.
>>>
>>> I monitored (by hand) a while the sysctls  
>>> kstat.zfs.misc.arcstats.size
>>> and kstat.zfs.misc.arcstats.hdr_size. Both grow way higher (at some
>>> point I've seen more than 500M) than what I have configured in
>>> vfs.zfs.arc_max (40M).
>>
>> My understanding about this is the following:
>> vfs.zfs.arc_min/max are not used as min max values.
>> They are used as high/low watermarks.
>> If arc is more than max the arc a thread is triggered to reduce the
>> arc cache until min, but in the meantime other threads can still grow
>> arc so there is a race between them.
>
> Hmm, if this is true the ARC size should go down to arc_min once it
> did grow past arc_max and no new data is coming along but I do not
> observe such a thing here. It simply stays near but below arc_max here
> all the time. I have only /home on ZFS with moderate load.

It appears arc_reclaim_thread only shrinks from arc_max when the  
system vm_lowmem event is fired or more than 75% of max kmem is in use  
by the system.

If you want to make it try to shrink the arc all the time you could  
try the patch below.  This worked to reduce arc_c on my system, but it  
was unable to reduce arc_size to match due to an apparent mutex miss.   
I'm still trying to track that down.

Hope that helps.

- Ben

Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
===================================================================
--- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c         
(revision 205)
+++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c         
(working copy)
@@ -1963,7 +1963,7 @@
                 if (needfree ||
                     (2 * arc_c < arc_size +
                     arc_mru_ghost->arcs_size + arc_mfu_ghost- 
 >arcs_size))
-                       arc_adjust();
+                       arc_shrink();

                 if (arc_eviction_list != NULL)
                         arc_do_user_evicts();