Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others)

[Ivan Voras]

Ivan Voras wrote:
> 2009/10/13 Larry Rosenman <ler at lerctr.org>:
> 
>>>>> note huge packet loss. It looks like it's VM fault or something like it.
>>>> It sounds like the VM is failing to execute the guest during certain
>>>> types of I/O. A bit of scheduler tracing in the host OS probably wouldn't go
>>>> amiss to confirm that the VM really is suspending the guest
>>> It's VMWare ESXi underneath, which is *Officially Not Linux* though some
>>> ducks may disagree - anyway, I suspect tracing the host in this way is next
>>> to impossible without some kind of diamondium-level contract.
>>>
>> What information do you need?  I have a platinum VMWare contract.
>>
>> What version of ESXi?
> 
> Hi,
> 
> It is ESXi 3.5 - but if the problem is really in ESXi I presume anyone
> could reproduce it. My setup is nothing special - Xeon 5405, 8 GB RAM,
> SATA drives on ICH9.
> 
> As for what data is needed, it depends on what you can get - from this
> discussion thread it looks like it would be enough to verify that disk
> IO doesn't leave VM processes waiting (i.e. that disk IO doesn't
> interfere with CPU-bound or idle virtual machines). Though now when I
> think of it - doesn't Linux ATA driver poll IO in some funky way,
> expecting to get lower latency that way?

Another data point - the OS in the VM in question hanged today sometime 
after 5 AM in the following way:

	* console nonresponsive (also to ctrl-alt-del)
	* ssh login nonresponsive (timeout)
	* ping works (!)

Judging by the last seen timestamp, the machine should have been in the 
process of receiving rsync backups - so IO-bound.