More IO identification problems

[Adam Vande More]

As a follow-up, after applying both patches presented in this thread(FreeBSD
8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010),

http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html

"top -m io" now displays much more info and is generally consistent with
gstat in that IO spikes occer around the same time.  However, gstat and "top
-m io" are still not displaying any stats for 90%+ of my hard drive
indicator light blinks.  Since the issue effects gstat as well, it doesn't
seem like it could be related to ZFS.  When the system is basically idling,
the only consistent IO related entries are these;

    6 root           15      0      1     12      0     13  20.00%
{l2arc_feed_threa}
    6 root           15      0      1     12      0     13  20.00%
{arc_reclaim_thre}
    6 root           15      0      1     12      0     13  20.00% {zvol
zoot/usr/ho}
    6 root           15      0      1     12      0     13  20.00%
{txg_thread_enter}
    6 root           15      0      1     12      0     13  20.00%
{txg_thread_enter}

Entries like these occur like clockwork on 30 sec intervals.  My theory here
is that since the ZVOL has UFS + SU, this is causing a sync?

What I'm trying to diagnose is a much more frequent hard drive access which
occurs on approximately 2 sec intervals.  I timed this by pinging localhost
and comparing the response to the blinks.  It's very consistent although not
completely so as once in awhile the blink occurs every second.  If I listen
carefully I can hear increased drive activity during many of these intervals
so the indicator light seems to be working correctly.

If anyone has ideas or tests they'd like me to run, it would be appreciated.

-- 
Adam Vande More