RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"]

[bob prohaska]

On Wed, Aug 08, 2018 at 03:55:52PM -0600, Warner Losh wrote:
> On Wed, Aug 8, 2018 at 3:42 PM, bob prohaska <fbsd at www.zefox.net> wrote:
> 
> > On Mon, Aug 06, 2018 at 11:58:37AM -0400, Mark Johnston wrote:
> > >
> > > If the above suggestion doesn't help, the next thing to try would be to
> > > revert the oom_seq value to the default, apply this patch, and see if
> > > the problem continues to occur.  If this doesn't help, please try
> > > applying both measures, i.e., set oom_seq to 120 _and_ apply the patch.
> > >
> >
> > Applying both measures resulted in a panic, not entirely unlike the effect
> > of deliberately running the machine out of swap on microSD. The first
> > obvious
> > sign of trouble was errors attributed to da0:
> >
> > (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 25 bc b8 00 00 80 00
> > (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
> > (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
> >
> 
> So we're pushing so hard that the writes are actively failing...
> 
> With the da driver there's some hope. Add options IOSCHED to the  kernel
> config file and reboot. This will give you some detailed statistics, as
> well as power-of-two bucketized latency histograms. It may even be a vector
> forward to slow the writes / trims down, though there's some issues when
> you slow writes down TOO much, it helps *A*LOT* keep the system responsive.
> We do that at work to make our consumer SSDs not suck for serving content
> (reading) while we're doing some writes to them... The thumb drives are
> like the consumer SSDs we buy, only crappier...
> 

I've added some sorted versions of the swapscript output keyed on ms/read and
ms/write to the directory at
http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/batchqueue/pageout120/
and randomly through the older testcases.

Some of the delays approach a minute, and that's been the case for as long as I've
been keeping track. However, the  big delays tend to be scattered through the gstat 
log and aren't necessarily associated with trouble, nor always with high levels of 
swap usage.

Mark Johnston's slow_swap.diff patch is compiling now, if that succeeds I'll
try it first. If it fails (I may have damaged /usr/obj in the last test run)
I'll add 
options IOSCHED 
to the GENERIC config file before trying again. 

Thanks for reading!

bob prohaska