Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

[Dieter BSD]

> Subsequent inspection suggested that it was happening during the
> periodic daily, though we never managed to get it to happen by manually
> forcing periodic daily, so that's only a theory.

Perhaps due to a bunch of VMs all running periodic daily at the same time?

> We had a perfectly functional, nearly zero-traffic VM, since Jabber
> traffic averages no more than a few messages per hour.  It was working
> for quite some time.
>
> We moved it from a local datastore to an iSCSI datastore that ended up
> getting periodically crushed by the load (in particular during the
> periodic daily load imposed by a bunch of VM's all running at once).
> At this point, this one VM started hanging on I/O.  We expected that
> this would clear up upon return to a host with a local datastore.  It
> did not.
>
> This ended up as a broken VM, one that would hang up overnite, maybe
> not every night, but several times a week at least.

...

> For the problem to "follow" the VM in this manner, and afflict *only*
> the one VM, strongly suggests that it is something that is contained
> within the VM files that constitute this VM.  That is consistent with
> the observation that the problem arose at a point where the VM is
> known to have had all those files moved from one location to a dodgy
> location.
>
> That's why I believe the evidence points to corruption of some sort.

Compare a backup of the VM before it broke to a backup of the same VM
after it broke.  Hopefully the haystack of insignificant differences
isn't too large, or the significant difference needle might be a
lot of "fun" to find.