RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"]

[John Kennedy]

On Sat, Aug 04, 2018 at 06:40:15AM -0700, John Kennedy wrote:
> For what it's worth, when I was first getting my rpi3 up and running, tuning
> swap and rebuilding world+kernel it really seemed to favor killing off ntpd
> and then usually a cc during the build (which resulted in a recoverable fail
> as I had to restart the build process with NO_CLEAN=yes.
> 
> This was using 12-current kernels.  Bumping /tmp (tmpfs) up to 75M (from 50)
> and having a proper freebsd-swap (vs swap-on-UFS-file) (and perhaps a V30 SD
> card) fixed all of that.  I still have to set TMPDIR=/var/tmp (disk, not
> tmpfs) during installworld beacuse it tries to strip at least one really big
> executable during the install and that'll break with <= 75M of /tmp.

Hmm.  Not perfect even now.  Hit during buildworld with all sources/objects
are local (vs NFS-mounted), intended to tweak out this experience for me.

	Aug  5 01:34:24 rpi3 kernel: pid 63223 (ld.lld), uid 0, was killed: out of swap space
	Aug  5 01:34:26 rpi3 kernel: pid 63360 (c++), uid 0, was killed: out of swap space
	Aug  5 01:34:26 rpi3 kernel: pid 846 (ntpd), uid 123, was killed: out of swap space

In theory, I have lots of swap (via top)

	Swap: 7174M Total, 25M Used, 7149M Free, 16K In

It varies a bit depending on the kernel, but I haven't adjusted for this
particular warning:

	rpi3 kernel: warning: total configured swap (1946624 pages) exceeds maximum recommended amount (918256 pages).
	rpi3 kernel: warning: increase kern.maxswzone or reduce amount of swap.

So far those have always been somewhere north of 3G:

	[kernel 12-CURRENT]
	r336739 ->	maximum recommended amount (918488 pages).
	r336795 ->	maximum recommended amount (918488 pages).
	r336962 ->	maximum recommended amount (918256 pages).

	# gpart show mmcsd0s2
	=>        0  124624896  mmcsd0s2  BSD  (59G)
	          0   83886080         1  freebsd-zfs  (40G)
	   83886080   25165824         2  freebsd-ufs  (12G)
	  109051904   15572992         4  freebsd-swap  (7.4G)

So 15572992 512-byte sectors -> 7604MiB -> ~7.42GiB, so that checks out.  For
swap, 15572992 512-byte sectors is 1946624 4K pages (presumably), so 918256 ->
~3.5GiB as a recommended swap value.  Not sure why top shows 7174M.

I know I can do a buildworld on my ~1GiB RAM/~2GiB swap amd64 droplet, so that
should be excessive swap for the arm64.

Of course, I'm not expecting any -j# buildworld to be exactly reproducible
where swap comes into play.