The pagedaemon evicts ARC before scanning the inactive page list

From: Alan Somers <asomers_at_freebsd.org>
Date: Tue, 18 May 2021 21:07:44 UTC
I'm using ZFS on servers with tons of RAM and running FreeBSD
12.2-RELEASE.  Sometimes they get into a pathological situation where most
of that RAM sits unused.  For example, right now one of them has:

2 GB   Active
529 GB Inactive
16 GB  Free
99 GB  ARC total
469 GB ARC max
86 GB  ARC target

When a server gets into this situation, it stays there for days, with the
ARC target barely budging.  All that inactive memory never gets reclaimed
and put to a good use.  Frequently the server never recovers until a reboot.

I have a theory for what's going on.  Ever since r334508^ the pagedaemon
sends the vm_lowmem event _before_ it scans the inactive page list.  If the
ARC frees enough memory, then vm_pageout_scan_inactive won't need to free
any.  Is that order really correct?  For reference, here's the relevant
code, from vm_pageout_worker:

shortage = pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);
if (shortage > 0) {
        ofree = vmd->vmd_free_count;
        if (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)
                shortage -= min(vmd->vmd_free_count - ofree,
                    (u_int)shortage);
        target_met = vm_pageout_scan_inactive(vmd, shortage,
            &addl_shortage);
} else
        addl_shortage = 0

Raising vfs.zfs.arc_min seems to workaround the problem.  But ideally that
wouldn't be necessary.

-Alan

^ https://svnweb.freebsd.org/base?view=revision&revision=334508