ZFS "stalls" -- and maybe we should be talking about defaults?

[Jeremy Chadwick]

On Tue, Mar 05, 2013 at 09:08:09PM -0800, Jeremy Chadwick wrote:
> > > * How long the stall is in duration (ex. if there's some way to
> > >   roughly calculate this using "date" in a shell script)
> > They're variable.  Some last fractions of a second and are not really
> > all that noticeable unless you happen to be paying CLOSE attention. 
> > Some last a few (5 or so) seconds.  The really bad ones last long enough
> > that the kernel throws the message "swap_pager: indefinite wait buffer".
> 
> The message "swap_pager: indefinite wait buffer" indicates that some
> part of the VM is trying to offload pages of memory to swap via standard
> I/O write requests, and those writes have not come back within kern.hz*20
> seconds.  That's a very, very long time.

Two clarification points:

1. The timeout value is passed to msleep(9) and is literally kern.hz*20.
Per sys/vm/swap_pager.c:

   1216                 if (msleep(mreq, VM_OBJECT_MTX(object), PSWP, "swread", hz*20)) {
   1217                         printf(
   1218 "swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n",
   1219                             bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount);

How that's interpreted is documented in msleep(9):

     The parameter timo specifies a timeout for the sleep.  If timo is
     not 0, then the thread will sleep for at most timo / hz seconds.
     If the timeout expires, then the sleep function will return
     EWOULDBLOCK.

2. The message appears to be for swap I/O *reads*, not writes; at least
that's what the "swread" STATE string (you know, what you see in
top(1)) implies.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |