More IO identification problems

[jhell]

On 11/26/2010 08:41, Andriy Gapon wrote:
> on 26/11/2010 08:05 Adam Vande More said the following:
>> As a follow-up, after applying both patches presented in this thread(FreeBSD
>> 8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010),
>>
>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html
>>
>> "top -m io" now displays much more info and is generally consistent with
>> gstat in that IO spikes occer around the same time.  However, gstat and "top
>> -m io" are still not displaying any stats for 90%+ of my hard drive
>> indicator light blinks.  Since the issue effects gstat as well, it doesn't
>> seem like it could be related to ZFS.  When the system is basically idling,
>> the only consistent IO related entries are these;
>>
>>     6 root           15      0      1     12      0     13  20.00%
>> {l2arc_feed_threa}
>>     6 root           15      0      1     12      0     13  20.00%
>> {arc_reclaim_thre}
>>     6 root           15      0      1     12      0     13  20.00% {zvol
>> zoot/usr/ho}
>>     6 root           15      0      1     12      0     13  20.00%
>> {txg_thread_enter}
>>     6 root           15      0      1     12      0     13  20.00%
>> {txg_thread_enter}
>>
>> Entries like these occur like clockwork on 30 sec intervals.  My theory here
>> is that since the ZVOL has UFS + SU, this is causing a sync?
>>
>> What I'm trying to diagnose is a much more frequent hard drive access which
>> occurs on approximately 2 sec intervals.  I timed this by pinging localhost
>> and comparing the response to the blinks.  It's very consistent although not
>> completely so as once in awhile the blink occurs every second.  If I listen
>> carefully I can hear increased drive activity during many of these intervals
>> so the indicator light seems to be working correctly.
>>
>> If anyone has ideas or tests they'd like me to run, it would be appreciated.
>>
> 
> Perhaps it's some external component?
> E.g. hald is known to perform some disk/media checks every two seconds.
> 

As well syslod will also cause sync to happen prematurely when something
goes to log.

syslog.conf(5):
***
 To ensure that kernel messages are written to disk promptly,
syslog.conf calls fsync(2) after writing messages from the kernel.
Other messages are not synced explicitly. You may prefix a pathname
with the minus sign, ``-'', to forego syncing the specified file
after every kernel message. Note that you might lose information if
the system crashes immediately following a write attempt. Neverthe-
less, using the ``-'' option may improve performance, especially if
the kernel is logging many messages.
***

This is obviously the old way of ensuring logs are written to disk
before crash but now ZFS is handled differently. Check to see if adding
'-' to your syslog entries relieves that problem as ZFS will do the
right thing to ensure your data is written to disk.

-- 

 jhell,v