Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others)

[Luigi Rizzo]

On Tue, Oct 13, 2009 at 11:13:31AM +0100, Kris Kennaway wrote:
> Luigi Rizzo wrote:
> >On Mon, Oct 12, 2009 at 09:48:42PM +0200, Thomas Backman wrote:
> >>I'm copying this over from the freebsd-performance list, as I'm  
> >>looking for a few more opinions - not on the problems *I* am having,  
> >>but rather to check whether the problem is universal or not, and if  
> >>not, find a possible common factor.
> >>In other words: I want to hear about your experiences, *good or bad*!
> >>
> >>Here's the original thread (not from the beginning, though): 
> >>http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003843.html
> >>
> >>Long story short, my version: when the disk is stressed hard enough,  
> >>console IO becomes COMPLETELY unbearable. 10+ seconds to switch  
> >>between windows in screen(1), running (or even typing) simple  
> >>commands, etc. This happens both via SSH and the serial console.
> >
> >hi,
> >this issue (not specific to FreeBSD, and not new -- it has been
> >like this forever) is discussed in some detail here
> >
> >	http://www.bsdcan.org/2009/schedule/events/122.en.html
> >
> >The following code (a bit outdated) can help
> >
> >http://lists.freebsd.org/pipermail/freebsd-stable/2009-March/048704.html
> 
> Are you certain?  The reported symptoms sound very unusual.  Can you 
> reproduce the problem with the provided instructions yourself?

sure -- with ATA/SATA disks, the test in the original post is
enough to trigger cwpart of the problem

	time file /etc/*	# this one is fast
	cat /dev/zero > /same_disk_as_etc/somedir/somefile &
	sleep enough_to_flush_disk_cache # 10-30 sec
	time file /etc/*	# this one takes forever

Now, getting sluggish behaviour on the console might need something
more such as dependencies between the program being run and disk
activity (logging, etc.) but the 'capture effect' on the disk
is completely reproducible. Perhaps SCSI and various RAID incarnations
may have a better behaviour or need a larger number of greedy
clients to trigger the problem.

cheers
luigi