ZFS+find(1) wiring all RAM

[Peter Jeremy]

I've noticed that 11-stable/amd64 has been wiring seemingly excessive
amounts of RAM for some time (the problem goes back at least 6 months).
This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap
errors killing processes on a low-memory system.  I'm not sure when it
started by it seems to hawe gotten worse between r331535 and r334494.

I can see the "excessive wired memory" on my main home system with 32GB RAM
but haven't seen it completely run out of RAM.  After some gentle use and a
nightly run, there is 10GB more wired RAM than ARC.

My "low memory" system is a Google GCE f1-micro instance[*] (600MB RAM) with
about 723k inodes used and the following ZFS tuning:
vfs.zfs.arc_max="128M"
vfs.zfs.arc_meta_limit="50M"
vfs.zfs.arc_min="25M"

The following numbers were gatherer by looking at top(1).  Running r334494,
after booting, to multi-user, the system has about 187MB wired (94MB ARC).
If I then run /etc/periodic/security/100.chksetuid, wired RAM increases to
about 580MB, with 380MB ARC, dropping to 467MB and 217MB ARC when the script
exits (this is still nearly twice arc_max).  Free memory can drop to <10KB
whilst the find(1) is running.

I have several issues with this behaviour:
0) ARC usage can significantly exceed arc_max.  I understand that arc_max is
a soft limit but IMO, 3x is unreasonable - especially when the system is
under extreme memory pressuse.
1) Significant amounts of wired memory are in use but I can't find anything
in "vmstat -mz" that would explain where it's going.

Does anyone have any suggestions for digging into this?

[1] I get the same behaviour using a VBox instance with similar dimensioning
and the same tuning)

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20180607/5b1b6eb4/attachment.sig>