Re: RFC: How ZFS handles arc memory use

Hi Rick,

On 22.10.2025 10:34, Rick Macklem wrote:
> A couple of people have reported problems with NFS servers,
> where essentially all of the system's memory gets exhausted.
> They see the problem on 14.n FreeBSD servers (which use the
> newer ZFS code) but not on 13.n servers.
> 
> I am trying to learn how ZFS handles arc memory use to try
> and figure out what can be done about this problem.
> 
> I know nothing about ZFS internals or UMA(9) internals,
> so I could be way off, but here is what I think is happening.
> (Please correct me on this.)
> 
> The L1ARC uses uma_zalloc_arg()/uma_zfree_arg() to allocate
> the arc memory. The zones are created using uma_zcreate(),
> so they are regular zones. This means the pages are coming
> from a slab in a keg, which are wired pages.
> 
> The only time the size of the slab/keg will be reduced by ZFS
> is when it calls uma_zone_reclaim(.., UMA_RECLAIM_DRAIN),
> which is called by arc_reap_cb(), triggered by arc_reap_cb_check().
> 
> arc_reap_cb_check() uses arc_available_memory() and triggers
> arc_reap_cb() when arc_available_memory() returns a negative
> value.
> 
> arc_available_memory() returns a negative value when
> zfs_arc_free_target (vfs.zfs.arc.free_target) is greater than freemem.
> (By default, zfs_arc_free_target is set to vm_cnt.v_free_taget.)
> 
> Does all of the above sound about right?

There are two mechanisms to reduce ARC size: either from ZFS side in the 
way you described, or from kernel side, when it calls ZFS low memory 
handler arc_lowmem().  It feels somewhat overkill, but it came this way 
from Solaris.

Once ARC size is reduced and evictions into UMA caches happened, it is 
up to UMA how to drain its caches.  ZFS might trigger that itself, or it 
can be done by kernel, or few years back I've added a mechanism for UMA 
caches to slowly shrink by themselves even without pressure.

> This leads me to...
> - zfs_arc_free_target (vfs.zfs.arc.free_target) needs to be larger

There is a very delicate balance between ZFS and kernel 
(zfs_arc_free_target = vm_cnt.v_free_target).  Imbalance there makes one 
of them suffer.

> or
> - Most of the wired pages in the slab are per-cpu,
>    so the uma_zone_reclaim() needs to UMA_RECLAIM_DRAIN_CPU
>    on some systems. (Not the small test systems I have, where I
>    cannot reproduce the problem.)

Per-CPU caches should be relatively small. IIRC in dozens or hundreds of 
allocations per CPU.  Their drain is expensive and should rarely be 
needed, unless you have too little RAM for the number of CPUs you have.

> or
> - uma_zone_reclaim() needs to be called under other
>    circumstances.
> or
> - ???
> 
> How can you tell if a keg/slab is per-cpu?
> (For my simple test system, I only see "UMA Slabs 0:" and
> "UMA Slabs 1:". It looks like UMA Slabs 0: is being used for
> ZFS arc allocation for this simple test system.)
> 
> Hopefully folk who understand ZFS arc allocation or UMA
> can jump in and help out, rick

Before you dive into UMA, have you checked whether ARC size really 
shrinks and eviction happens?  Considering you mention NFS, I wonder 
what is your number of open files?  Too many open files might in some 
cases restrict ZFS ability to evict metadata from ARC.  arc_summary may 
give some insights about ARC state.

-- 
Alexander Motin