saving job state over a power outage

[Norberto Meijome]

On Wed, 5 May 2010 13:49:31 -0300
Joey Mingrone <joey at mingrone.org> wrote:

> Hello,
> 
> Our lab has a cluster (Sun Fire X40z master node, Opteron 270 and
> generic 2.0 GHz Opteron based compute nodes) running 8.0-RELEASE.
> We've been informed that the power has to be turned off for the
> weekend, but one of the lab members has been running some jobs since
> December.  I don't know much about the jobs, i.e., if they can be
> restarted without losing all the computing work that's been done so
> far.
> 
> Can anyone suggest a way to save the state of the jobs so they can
> continue when the power comes back on? 

Sorry, no real answers :(

If your nodes were actually virtual machines, one per host, you could suspend
the VM to disk, then shut down your hosts safely. Of course, this assumes your
VMs could suspend to disk at all... 

> From experience, ACPI seems
> flaky if it will work at all because of problems with BIOS
> implementations.  Also, iirc ACPI has issues with SMP kernels.  Is
> there something similar to software suspend found in Linux?  Does
> anyone have any other suggestions to accomplish this?

I think you'd be needing suspend to disk.... unless you have enough UPS to keep
the machines with enough juice in suspend mode for the duration of your
outage...

good luck,
_________________________
{Beto|Norberto|Numard} Meijome

Unix is user friendly. However, it isn't idiot friendly.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.