lightly loaded system eats swap space

Shane Ambler FreeBSD at ShaneWare.Biz
Mon Jun 18 06:14:41 UTC 2018

On 18/06/2018 07:49, tech-lists wrote:
> Hello list,
> context is (server)
> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM,
> E5-2630 @2.3GHz, generic kernel.
> There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also
> freebsd-11-stable)
> There have been no special options for zfs configuration on the server,
> apart from several datasets having the compressed property set (lz4).
> The server runs nothing else really apart from sshd and it uses ntpd to
> sync local time.
> How come such a lightly loaded server with plenty of resources is eating
> up swap? If I run two bhyve instances, i.e. two of the same size as
> indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace
> errors in the daily logs:
> +swap_pager_getswapspace(24): failed
> +swap_pager_getswapspace(24): failed
> +swap_pager_getswapspace(24): failed

Out of swap messages can be misleading, you can also get out of swap
messages if too much ram gets wired - wired doesn't get swapped out.
Too much wired can also cause everything else to be paged out.

How do you start bhyve instances? A previous issue used one of the bhyve
helper ports that used -S to wire all guest memory as a default setting.

> Here's top, with one bhyve instance running:
> last pid: 49494;  load averages:  0.12,  0.13,  0.88
>                              up 29+11:36:06  22:52:45
> 54 processes:  1 running, 53 sleeping
> CPU:  0.4% user,  0.0% nice,  0.4% system,  0.3% interrupt, 98.9% idle
> Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf,

You have 116G wired of 128G, this would be the cause behind everything
else going to swap. 108G of that wired looks to be arc.

> ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other
>      103G Compressed, 104G Uncompressed, 1.00:1 Ratio
> Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse

There is `sysctl vm.max_wired` which limits the kernel wired memory, the
arc is then also wired but added on top of that setting, if your guests
are then wiring more ram, you can get too much wired.

There is a patch in review that attempts to improve arc releasing when
free ram is low. See if this helps -

Other than that check that your guest ram + max_wired + arc_max adds up
to less than your physical ram minus some host system ram. Note that
wax_wired is a page count so multiply it by hw.pagesize

FreeBSD - the place to B...Software Developing

Shane Ambler

More information about the freebsd-questions mailing list