rrdtool / mtr causing stalling on 7.0

[Robert Watson]

On Sat, 8 Mar 2008, Steven Hartland wrote:

> We've been suffering on our stats box for some time now where by the machine 
> will just stall for several seconds preventing everything from tab 
> completion to vi newfile.txt.
>
> I was hoping an upgrade to 7.0 and ULE may help the situation but 
> unfortunately it hasn't.
>
> I've attached both dmesg and output from lock profiling during a 5 minute 
> period where I know the stall happened at least once.
>
> Any advice / pointers would be gratefully received.

It looks like the attachment got lost on the way through the mailing list.

I think the first starting point is: what sort of stall is this?  Is it, for 
example, all network communication stalling, all disk I/O stalling, or the 
entire kernel and all processes stalling?  The usual diagnostics are:

- Does the machine stop responding to pings while stalled, and/or possibly
   "catch up" all at once when it recovers?

- If you run the following loop on the machine without any network or console
   I/O, do you see gaps in time stamps:

 	while (1) {
 		sleep 1
 		date >> date.log
 	}

- If you write a short C program that looks a lot like the above loop, but
   logs time stamps into an in-memory buffer, and have it look for gaps in the
   sequence of >3 seconds, does it run across the stall?

Robert N M Watson
Computer Laboratory
University of Cambridge