Pi3 out of swap at < 50%

[bob prohaska]

On Sat, Feb 17, 2018 at 09:43:21AM -0700, Ian Lepore wrote:
> On Sat, 2018-02-17 at 08:27 -0800, bob prohaska wrote:
> > Running make -j4 buildworld on a Pi3 at r329360 tends to result
> > in??messages similar to:
> > 
> > pid 33492 (llvm-tblgen), uid 0, was killed: out of swap space
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 11764, size: 28672
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16080, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 15721, size: 20480
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28139, size: 65536
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 40544, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 58384, size: 65536
> > pid 49735 (c++), uid 0, was killed: out of swap space
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28470, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 24889, size: 8192
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 28736, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 29736, size: 4096
> > 
> > even though swap usage appears to be less than one GB out of two available.
> > 
> > So far it's been possible to use the -DNO_CLEAN option with make to pick
> > up where things left off and I've backed down to -j3 to see if that helps.
> > 
> > There are two 1 GB swap partitions, one on USB flash and one on the microSD
> > card and both appear to be in use according to swapinfo. There are no warnings
> > in the boot messages and no explicit changes have been made to configuration
> > files apart from adding the swap entries to /etc/fstab.
> > 
> > World and kernel are about two weeks out of sync, so top isn't perfectly
> > up-to-date with this kernel; could that account for the mismatch between
> > apparent swap usage and the "out of swap" messages?
> > 
> > Thanks for reading, and any ideas!
> > 
> > bob prohaska
> 
> I suspect your swap devices are too slow to keep up with instantaneous
> demands. ?An sdcard can easily have read and write latencies as big as
> 30 seconds; they make pretty poor swap devices (have gstat running in
> another window during the build to see what I mean, look at the ms/r
> and ms/w columns). ?USB flash drives may not be much better.
> 
I'm watching gstat now. With make -j3 running and three cc processes in
the top of top the usual value of ms/r and ms/w is less than 5 ms. An
eyeball average is probably less than 10 ms. Before cc got going (make
dominant in top) there were very brief excursions to 2000 ms, as you posit,
but they didn't last and didn't put any errors on the console. 

The experiment ended when make stopped with:

/usr/src/lib/libc/rpc/auth_time.c:115:10: error: unknown type name 'endpoint'
free_eps(endpoint eps[], int num)
         ^
/usr/src/lib/libc/rpc/auth_time.c:143:8: error: unknown type name 'nis_server'
static nis_server *
       ^
/usr/src/lib/libc/rpc/auth_time.c:144:49: error: unknown type name 'nis_server'
get_server(struct sockaddr_in *sin, char *host, nis_server *srv,
                                                ^
/usr/src/lib/libc/rpc/auth_time.c:145:5: error: unknown type name 'endpoint'
    endpoint eps[], int maxep) 

among others.

I'll update sources and restart buildworld with -j4, watching gstat more
carefully. 

Thanks very much for the hint!

bob prohaska