top io mode

[Jeremy Chadwick]

On Thu, Nov 25, 2010 at 12:57:33PM -0800, Jeremy Chadwick wrote:
> On Thu, Nov 25, 2010 at 02:28:35PM -0600, Adam Vande More wrote:
> > On Thu, Nov 25, 2010 at 2:08 PM, Jeremy Chadwick
> > <freebsd at jdc.parodius.com>wrote:
> > 
> > > >  Well the top bug didn't seem to resolve my actual issue and yes it it a
> > > > complete ZFS system.  Problem is that the HD activity indicator light is
> > > > constantly flickering even though should be minimal activity.  top still
> > > > shows no activity around the blinks, and there's no swapping happening
> > > > although gstat does seem to roughly match the blinks.  I can't tell what
> > > > your patch does, would it enable me to see what's touching the disk?
> > >
> > > Please try using gstat(8) instead.
> > >
> > 
> > Like I said gstat shows activity, but I have no way find what process is
> > causing the activity unless there is some hidden gstat option I'm unaware
> > of.
> 
> Sorry, I missed that part of your explanation, my apologies.
> 
> > to be more clear, if I have top -m io in one terminal, I see no movement in
> > the READ/WRITE columns.
> > 
> > while polling gstat output I captured an IO "spike"
> > 
> > http://pastebin.com/f84nuzxt
> > 
> > the percent busy is at 0.00 for every entry with the exception of these
> > spikes.
> 
> I don't have an answer for you, other than "maybe top shows a different
> 'kind' of I/O than what gstat or iostat does".
> 
> Only ideas I have:
> 
> 1) Possibly acct(2) with sa(8) could provide some insight, but I have no
> idea how well it works.
> 
> 2) I see some ps(8) -O parameters labelled "inblk" and "oublk", but I'm
> not sure what "block" means given the man page context.  If these are
> "disk blocks", then possibly a shell script that repetitively calls ps
> with these arguments could narrow it down.
> 
> It's too bad, since sar(1M) on Solaris can be used to achieve exactly
> what you're looking for -- and the closest thing to that on FreeBSD that
> I know of is acct(2) and sa(8).

Bad form to follow up to my own Email of course, but some discussion
material:

"top -m io" doesn't show any I/O writes, while gstat(8) does, and to
numerous devices all which make up some form of ZFS pool.

If you do something like "dd if=/dev/urandom of=/pool/file bs=64k", does
"top -m io" show write I/O for the "dd" process?

If so, then possibly ZFS "internally" is doing something quietly/behind
the scenes (such as a disk flush or some sort of internal transaction
"thing"), thus the kernel would be responsible and not a process.

So that said, how about "top -S -m io -U root" instead, which includes
system processes, since we can (probably) assume a non-root process
isn't responsible?

HTH.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |