ZFS overgrowing arc_max -> OOM kills

[Peter Much]

Hi all,

I thought this matter was solved a long time ago! In fact, I am
using ZFS almost from the beginning, and I never observed it
seriousely overgrowing ArcMax, until now (this is 11.3-RELEASE-p3).

> System Memory:
>        9.38%   187.32  MiB Active,     0.37%   7.35    MiB Inact
>         78.60%  1.53    GiB Wired,      0.00%   0 Cache
>         1.11%   22.14   MiB Free,       10.54%  210.59  MiB Gap
>        Real Installed:                         2.00    GiB
>         Real Available:                 99.19%  1.98    GiB
>         Real Managed:                   98.33%  1.95    GiB
>        Logical Total:                          2.00    GiB
>         Logical Used:                   98.56%  1.97    GiB
>         Logical Free:                   1.44%   29.49   MiB
> Kernel Memory:                                  768.84  MiB
>         Data:                           98.21%  755.05  MiB
>         Text:                           1.79%   13.79   MiB
> Kernel Memory Map:                              1.38    GiB
>         Size:                           98.79%  1.36    GiB
>         Free:                           1.21%   17.10   MiB

See the KMEM! Obviousely, this was the last useful vital sign
   from the machine, then it stalled normal I/O and did only
network gatewaying:

> last pid: 51252;  load averages:  0.39,  3.09,  3.68    up 5+07:46:39   
> 03:09:25
> 1569 processes: 3 running, 1530 sleeping, 36 waiting
> CPU 0:  1.1% user,  0.0% nice,  5.0% system,  0.4% interrupt, 93.5% idle
> CPU 1:  0.0% user,  0.0% nice,  3.4% system,  1.9% interrupt, 94.6% idle
> Mem: 169M Active, 4K Inact, 204M Laundry, 1582M Wired, 85M Buf, 43M Free
> ARC: 1026M Total, 483M MFU, 160M MRU, 11M Anon, 188M Header, 184M Other
>      127M Compressed, 526M Uncompressed, 4.16:1 Ratio
> Swap: 8192M Total, 1965M Used, 6227M Free, 23% Inuse, 56K In

Now see the ARC:

> ARC Summary: (HEALTHY)
>         Memory Throttle Count:                  0
> ARC Misc:
>         Deleted:                                22.35m
>         Recycle Misses:                         0
>         Mutex Misses:                           22.11m
>         Evict Skips:                            733.63m
> ARC Size:                               127.01% 1016.06 MiB
>         Target Size: (Adaptive)         25.00%  200.00  MiB
>         Min Size (Hard Limit):          25.00%  200.00  MiB
>         Max Size (High Water):          4:1     800.00  MiB

This was steadily growing since 03:01 - likely due to the influx of
metadata from these daily/security/100/110 "find" jobs.

I did my calculations to cater for a 25% safety margin (which appear to
be quite accurate), but nevertheless, ZFS goes beyond that. :(

(Yes, this is i386 - it's a twenty-year-old machine. ;))

The problem started with OOM kills at 03:02 reporting "out of swap",
which is bogus (there is no swap shortage); so I found a way to switch
such annoying behaviour off:
> vm.pageout_oom_seq=512

Then, instead, I got massive "indefinite buffer wait" messages. And these
seem to come from what You can read above: out of kernel heap.

PMc