Jails on ZFS yielding 100% load on gstat

[Marco Steinbach]

On Mon, 13 Aug 2018 10:12:26 -0400
Mike Tancsa <mike at sentex.net> wrote:

> On 8/12/2018 2:50 PM, Marco Steinbach wrote:
> > 
> > These loads lead to the system suffering from very much delayed
> > responses to even the basic task of echoing characters entered on
> > the console, consequently rendering the services offered unusable
> > to the users because of the delays.  
> 
> 
> Do you have a LOT of files and or metadata ? Have a look at the cache
> stats to see if you are perhaps grinding away on big directory
> lookups? Install sysutils/zfs-stats and post
> zfs-stats -a
> 
> Also, does
> top -mio -I
> shed any light as to whats taking up the disk io ?
> 
> 	---Mike

I've had these kinds of sudden load spikes on my raidz1 setups in the
past, which I can't remember seeing, before all machines were migrated
from UFS to ZFS.

Stats during these spikes, that is, if the machine in question would
still respond to commands, did show 100% load on gstat, with ZFS having 
low throughput (typically less than a megabyte per second).

Top display varied wildly, up to the point where each and any process
accessing storage would put loads > 80% on IO.  Which was kind of to be
expected, since something was clogging the queue.

In this particular case, as noted in another post, a customers php
script ran into a defective MySQL table, and instead of simply
erroring out kept hammering it. Leading to what I assume to be a myriad
of small IO operations -- probably amplified by the blocksize
clash between MySQL and ZFS, which otherwise never impacted performance
in my use-cases m|


I'll gather more detailed stats the next time I see ZFS storage
bogging down.

MfG CoCo