Re: ZFS ARC tuning

On Sun, Sep 14, 2025 at 4:16 PM Karl Denninger <karl@denninger.net> wrote:
>
> On 9/13/2025 21:16, Garrett Wollman wrote:
>
> <<On Sat, 13 Sep 2025 04:02:01 +0100, Graham Perrin <grahamperrin@gmail.com> said:
>
> Also, might one of the following be a better alternative?
>
> vfs.zfs.arc_free_target
>
> vfs.zfs.arc.sys_free
I have looked and, for Linux systems with a reasonable amount of memory,
this is set to about 1Gbyte + allmem/32. You can look at arc_set_sys_free() in
sys/contrib/openzfs/module/os/linux/zfs/arc_os.c for the exact calculation.

FreeBSD doesn't set this at all (it appears to be 0 unless you set it),
see kstat.zfs.misc.arcstats.arc_sys_free.

It appears that the setting of this is how much memory the arc code
attempts to reserve to the rest of the system.

Garrett, have you tried playing with setting vfs.zfs.arc.sys_free?

rick



>
> The former *sounds* like it might be a reasonable lever, although I
> still feel like there's some sort of accounting issue here.  I note on
> my systems the latter is zero, but the former has a positive value
> (which seems to be related to memory size).
>
> -GAWollman
>
> Have you looked at vmstat -z?
>
> Specifically, the zfs-related ones.  Of particular interest is if the "Free" count for any of the sizes is unreasonably large.
>
> There used to be a loooong time ago a problem with the zfs code and vm system interaction that under certain workloads the VM would not release free allocations back.  This would lead to severe pathological behavior including serious stalls, swap-outs and OOM kills.
>
> I haven't had trouble with this in my workloads since the newer OpenZFS import occurred, but I'm curious if this is actually ARC not releasing memory when memory pressure occurs (that is, the space is actively allocated by ARC and is not being released back) OR if the space IS being released back but the VM system isn't freeing the space back.
>
> If its the former then while ZFS should release it under pressure its possible the event happens fast enough that it doesn't respond in time and perhaps the knob to limit it will resolve it even though you shouldn't have to.  But if its the latter than twisting that knob is unlikely to bring joy.
>
> --
> Karl Denninger
> karl@denninger.net
> The Market Ticker
> [S/MIME encrypted email preferred]