RPI3 swap experiments

[John Kennedy]

On Tue, Jul 17, 2018 at 11:06:50PM -0700, bob prohaska wrote:
> It appears that some progress has been made in getting swap working reasonably
> on the RPI3. A -j4 buildworld attempt running r336356 to compile 336431 failed
> with "out of swap" but the worst read and write delays were less than 5 seconds,
> a marked improvement over previous examples.
> 
> In this case swap was split, 2 each 1GB  partitions on USB flash plus 1 GB on the 
> microSD card. Previous attempts using 3 each 1 GB partitions on USB flash have been
> repeatedly successful, while a single attempt using 3 each 1GB partitions on microSD
> failed.

  So I've got a RPI3B+.  YMMV.

  I've recompiled 12 ~7 times since June 27th and run into a handful of
problems but haven't had issues lately.  I think I started out with the
stock images (*-RPI3-20180618-r335317 and *-RPI3-20180709-r336134).

  I had OOM problems, so added swap (typical swap-on-ufs).  Could only get
~3.6G of swap without triggering the kern.maxswzone warning, and then had the
pauses (forget the exact text, but basically FreeBSD telling me that something
disk related had timed out after ~20 secs).  That stopped being a problem as
I took the stock images and created a real freebsd-swap partition (and maybe
a SDCARD upgrade).

  Just a gut feeling, but I figured that the kernel recompile was causing
enough buffering (and perhaps double-buffering with swap-on-ufs) that it was
getting itself soft-wedged (so timeouts, and OOM even though it didn't look
like I was getting anywhere near maxing out ~3.6G of swap).  OOM, kill ntpd,
then eventually kill part of the build and eventually the build would fail.
I didn't know if there might have been some memory spikes that blew it up that
I wasn't seeing with top.  I could restart and it would eventually finish,
but building everything takes too long as it is (very spoiled by beefy AMD64
box, but RPI3B+ destroys my old RPI1).

  So yes, I've (re)compiled everything on just an SDCARD.  I'll note that said
SDCARD is a SanDisk Extreme Plus 64G (UHS3 / V30, so ~30MB/s).  I think I was
using a SanDisk Ultra Plus (UHS1, no V-speed, so maybe ~10MB/s) initially
because it was laying around.  That was an educated guess on my part that
something that could sink 30MB/s of video would make a better sink that
something that averaged out to ~10MB/s data, and that spending more than that
was a waste based on iostat stats (I think topping out at ~6+MB/s with -w5).
I think I managed to get up to ~12+MB/s with some dd-bs=1M chunking.  I theory,
I think the RPI3 should basically have a ~60MB/s USB2 "backplane", so not sure
what the bottleneck is.

  At the moment, I have /usr/src and /usr/obj NFS-mounted, mostly to waste
a lot of time copying that work around while I try to figure out what is
wrong with my attempted ZFS setup.