Pi3 out of swap at < 50%

[Ian Lepore]

On Sat, 2018-02-17 at 08:27 -0800, bob prohaska wrote:
> Running make -j4 buildworld on a Pi3 at r329360 tends to result
> in  messages similar to:
> 
> pid 33492 (llvm-tblgen), uid 0, was killed: out of swap space
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 11764, size: 28672
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16080, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 15721, size: 20480
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28139, size: 65536
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 40544, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 58384, size: 65536
> pid 49735 (c++), uid 0, was killed: out of swap space
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28470, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 24889, size: 8192
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28736, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 29736, size: 4096
> 
> even though swap usage appears to be less than one GB out of two available.
> 
> So far it's been possible to use the -DNO_CLEAN option with make to pick
> up where things left off and I've backed down to -j3 to see if that helps.
> 
> There are two 1 GB swap partitions, one on USB flash and one on the microSD
> card and both appear to be in use according to swapinfo. There are no warnings
> in the boot messages and no explicit changes have been made to configuration
> files apart from adding the swap entries to /etc/fstab.
> 
> World and kernel are about two weeks out of sync, so top isn't perfectly
> up-to-date with this kernel; could that account for the mismatch between
> apparent swap usage and the "out of swap" messages?
> 
> Thanks for reading, and any ideas!
> 
> bob prohaska

I suspect your swap devices are too slow to keep up with instantaneous
demands.  An sdcard can easily have read and write latencies as big as
30 seconds; they make pretty poor swap devices (have gstat running in
another window during the build to see what I mean, look at the ms/r
and ms/w columns).  USB flash drives may not be much better.

-- Ian