RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"]

[Mark Johnston]

On Wed, Aug 08, 2018 at 08:38:00AM -0700, bob prohaska wrote:
> The patched kernel ran longer than default but OOMA still halted buildworld around
> 13 MB. That's considerably farther than a default build world have run but less than
> observed when setting vm.pageout_oom_seq=120 alone. Log files are at
> http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/batchqueue/
> 
> Both changes are now in place and -j4 buildworld has been restarted. 

Looking through the gstat output, I'm seeing some pretty abysmal average
write latencies for da0, the flash drive.  I also realized that my
reference to r329882 lowering the pagedaemon sleep period was wrong -
things have been this way for much longer than that.  Moreover, as you
pointed out, bumping oom_seq to a much larger value wasn't quite
sufficient.

I'm curious as to what the worst case swap I/O latencies are in your
test, since the average latencies reported in your logs are high enough
to trigger OOM kills even with the increased oom_seq value.  When the
current test finishes, could you try repeating it with this patch
applied on top? https://people.freebsd.org/~markj/patches/slow_swap.diff
That is, keep the non-default oom_seq setting and modification to
VM_BATCHQUEUE_SIZE, and apply this patch on top.  It'll cause the kernel
to print messages to the console under certain conditions, so a log of
console output will be interesting.