13-CURRENT: several GB swap being used despite plenty of free RAM

[Stefan Blachmann]

On 11/19/18, Wojciech Puchar <wojtek at puchar.net> wrote:
>> can go a long way. Having said that, it's been a while since I've had
>> to do this. The updates made to ZFS ARC and NUMA have allowed me to
>> rely on the algorithms baked into FreeBSD. My 8 GB systems have
>> performed rather well.
>
> 8GB is LOT LOT LOT of memory.
>

This might indeed be a cue.
Actually, I *never* had such annoying swap lag issues, until I
replaced my old computers (8GB) with newer ones (24-48GB).

With this increased memory, the ZFS ARC issue became apparent, causing
massive problems when ARC used more than 40 of the 48GB, starving
programs from memory, but this can be tuned easily.

And the "preemptive swapping" issue became apparent, too. Letting
programs idle for some time caused swapping out. This led to a very
bad user experience... often when one changes to another
desktop/virtual screen to another program, there is a delay of
swapping in, which in some cases could last for seconds when hundreds
of megabytes were swapped in again, in spite of some tens of gigabytes
of memory that never have been used since system boot.

> and my servers that have LOTS of httpd servers each for one webpage which are usually rarely visited.

I wondered why my test server had a delayed response time the first
few times I call a page from it after having it idled for some time.
To me it looks like that the threads apache keeps cached in memory for
fast response times got swapped out and the server response is slow
until all these have become "memory resident" again.

As the bad response times due to unnecessary preemptive swapping would
induce Google downranking for sites that aren't visited sufficiently
frequently enough to keep them from being swapped out, I consider this
a very very serious problem.


Disabling swap (either totally or for the affected
programs/users/jails using rctl) and taking care that memory never
runs out was the only way I was able to fix these issues.

And I find very interesting that this seems to occur *only* on
machines with *much* memory.
The complaints about this swapping behavior I saw on the forums
*always* regarded machines with at least 48GB of RAM. (I am not sure
whether I saw reports regarding 32GB machines, so I would suggest
investigating the issue on machines >=48GB)