Slashdot Mirror
UNIX Process Cryogenics?
shawarma asks: "Due to a recent
power outage, I've had to shut down a server running a process that had
been running for ages calculating something. The job it was doing would
have been done in a few days, I think, but I had to shut it down before the
UPS ran out of juice. This got me thinking: Why can't I freeze down the
process and thaw it back up at a later time? It ought to be possible to take
all the connected memory pages and save them in some way, preserve file
handles and pointers, and everything. Maybe net-connections would die,
but that's understandable. Has any work been done in this field? If not,
shouldn't there be? I'd like to contribute in some way, but I think it's a bit
over my head.." Laptops have been doing this in some form for years:
most laptops, when they run out of power, or when told by the user will
go into "suspend" mode which is similar to what the poster is describing,
however outside of laptops, I haven't seen this done. Sleeping processes
also do something similar, sending their memory pages into swap so other
running processes can use the memory. What, if anything, is preventing
someone from taking this a step further?
Windows XP has 2 features that are improved over Windows 2000 they are called suspend and hibernate. Suspend is a low power standby mode but it keeps all of your applications up and running when you come back. Hibernate actually saves everything to disk and shuts the computer all the way off. When you come back everything you are working on is there.
Good luck.
is not suspend, it is hibernate. Suspend will power down the computer except for the energy needed to keep the ram alive. hibernate will save all data to from memory to disk. I, personally, use neither on my laptop.
Remember, there were no nuclear weapons before women were allowed to vote.
http://www.cs.wisc.edu/condor/
Free-as-in-beer, on most major UNIX platforms. Check out our publications, we have several that give all the details you'd need to write it yourself.
Plenty of others, too - libckpt, there was a "Checkpointing Threaded Programs" paper at USENIX this past summer... there are some kernel patches that can do, most of them under the GPL.
It's called software suspend for linux. look for it on freshmeat.net
Do not look at laser with remaining good eye.
You can't just serialize and page out one process. Under every process are a slew of kernel objects and kernel crud including the virtual to physical mappings of your address space. It would be quite a challenge to isolate all of this and somehow persist it.
To make suspend work, you'd have to dump your entire memory image to disk. Then you swap in the entire image, kernel and user pages alike.
Someone you trust is one of us.
(I'm a Solaris user but I assume linux is the same). You can type Ctrl-\ to get a coredump of a runnining process and you can load a coredump with dbx. It seems like that's 90% or the infrastructure. You'd want it to run outside dbx and do it automatically. My guess is you'd have to just remap some addresses, recreate file pointers (assuming said files haven't been modified), reinstate the stack, and go.
This should be even easier to do in a JVM, even not relying on their serialization stuff.
I've used the Suspend/Resume feature on a sun box. IIRC, it mostly worked, but with a minor hitch that made me worry enough to never do it again. This suspend/resume is just like the laptop version -- save a copy of all memory to disk -- not the cryogenic per-process version you're talking about.
The per-process sounds neat, but usable only if you've got a simple critical task you're running. For a more complicated application, multiple processes may be working together, and you'd have to suspend all of them at the same time.
One big question I would have would be file handles... if you restore a process that thinks it owns file handle #5 and some other process is already using it, it would be awkward to get either process to use a different handle.
HIV Crosses Species Barrier... into Muppets
Vmware does this for the VM's it hosts. Works great.
Creed
All that is necessary for the triumph of good is that evil men do nothing.
For long-running processes, rather than shut down the process when the UPS kicks in, I've always found it easier to have the program snapshot its data tables periodically (say every half-hour) and build a "resume from disk" feature into the program. This lets you restart the program from its last check-point even in the event of uncontrolled program termination (e.g. kill -9 and the like).
-JS
Vanity of vanities, all is vanity...
--
"If you are an idealist it doesn't matter what you do or what goes on around you, because it isn't real anyway."-R.P.W.
If you are a grad student who has been waiting for a month for a job to finish...this could be very important. I was in an engineering department where jobs that ran for weeks were not uncommon (fortunately most of mine only took a day or two). A shutdown of a critical machine could set someone back months.
Linux software suspend may be of interest.
Once you've enabled it, you create a hibernation file on the C: drive. Hibernation should only take place when there is minimal disk activity (eg, don't hibernate while trying to save your Word document). The system saves the contents on RAM to the hard drive, and then shuts down. When the machine boots, a flag was set (I assume) indicating the system should resume from hibernation... so the hibernation file is read from disk and written to RAM and you're back up and running, in less time than it takes to boot. Plus it keeps your uptime from resetting back to zero.
Some things to note:
You will need WHQL certified drivers, or at least properly-written drivers. I have a SB Audigy and the first drivers I used (the ones on the included CD) caused a blue screen on resume from hibernation. When a updated driver was released, it fixed this issue.
Applications need to be properly-written as well, as there is some sort of Win32 suspend signal that is sent to apps just before the system hibernates, so the app must support this and the resume command when the system is restored.
Hibernation works great on my laptop and on my workstation, and I especially like the fact that I don't need to create a separate partition or install special drivers to make it work (you can even use it on an NTFS formatted drive).
But isn't it overkill for a data-crunching operation? As many other people have noted, it would seem you're much better off checkpointing your data to disk, rather than relying on low-level OS process wizardry.
Sig: What Happened To The Censorware Project (censorware.org)
You can't. The previous poster was making it sound too easy. Real checkpointing needs to save Kernel state as well -- file handles, device driver state, you name it. It isn't as simple as saving the in-memory image of the process.
the seti@home client uses its *.sah files to save the state of a calculation. of course, this is program dependent, not OS dependent. I guess if you have the source files for the program doing the counting.....
Tequila: It's not just for breakfast anymore!
STANDALONE CONDOR CHECKPOINTING:
..
...
Using the Condor checkpoint library without the remote system call functionality and outside of the Condor system is known as
"standalone" mode checkpointing.
To link in standalone mode, follow the instructions for linking Condor executables, but replace condor_syscall_lib.a with libckpt.a. If you
have installed Condor version 5.62 or above, you can easily link your program for standalone checkpointing using the condor_compile
utility with the little-known "-condor_standalone" option. For example:
condor_compile -condor_standalone [options/files....]
where is any of cc, f77, gcc, g++, ld, etc. Just enter "condor_compile" by itself to see a usage summary, and/or refer to
the condor_compile man page for additional information.
Once your program is relinked with the Condor standalone-checkpointing library (libckpt.a), your program will sport two new command
line arguments: "_condor_ckpt " and "_condor_restart ".
If the command line looks like:
exec_name -_condor_ckpt
then we set up to checkpoint to the given file name.
If the command line looks like:
exec_name -_condor_restart
then we effect a restart from the given file name.
Any Condor command line options are removed from the head of the command line before main() is called. If we aren't given
instructions on the command line, by default we assume we are an original invocation, and that we should write any checkpoints to the
name by which we were invoked with a "ckpt" extension.
To cause a program to checkpoint and exit, send it a SIGTSTP signal. For example, in C you would add the following line to your code:
kill( getpid(), SIGTSTP );
Note that most Unix shells are configured to send a TSTP signal to the foreground process when the user enters a Ctrl-Z. To cause a
program to write a periodic checkpoint (i.e., checkpoint and continue running), sent it a SIGUSR2:
kill( getpid(), SIGUSR2 );
In addition to the command-line parameters interface described above, a C interface is also provided for restarting a program from a
checkpoint file. The prototypes are:
void init_image_with_file_name( char *ckpt_name );
void init_image_with_file_descriptor( int fd );
void restart( );
The init_image_with_file_name() and init_image_with_file_descriptor() functions are used to specify the location of the checkpoint file.
Only one of the two must be used. The restart() function causes the process image from the specified file to be read and restored.
I think it was somewhere in the list of patches from the -mjc tree (see here) that there was a patch for the entire kernel for linux. Basically it let the system save it's state, and then restore it if it detects that it was shut down at that point. I'm not sure if this is what you want (and I couldn't get it working), but it's certainly a step in the right direction to what you're looking for.
Just found it here, it's the 'swsusp' patch.
Here's a mutation of FreeBSD that can do exactly that. I've put my laptop to sleep in the middle of installing software while running MacOS X and brought it back up several hours later to resume installation with no problems. The same function works on my G4 tower. Yes, it does drop network connections. However, it does use a trickle charge to power the LED's and presumably to keep the processor alive, and possibly some memory. Paging several hundred megabytes in a couple of seconds would be quite the task! One item of note is that all Apple machines have a special piece of hardware known as the PMU (Power Management Unit). In the desktops, it's parted out onto the mother board and into the power supply, but in the laptops it's a seperate card which controls both sleep and the charging of the battery. Perhaps other UNIX machines would need a similar device for this function to work properly.
Karma: Ran over your dogma.
A file descriptor is a per-process entity. Yes, there's a big table of file descriptors that exists for the entire sstem, but file descriptor 5 for process a is not file descriptor 5 for process b. Not even if they point to the same file/pipe. A case in point is FD 0, aka stdin. Every process starts out with a stdin on FD 0.
More important is how do you tell the kernel what file descriptor 5 pointed to? What if the file/pipe doesn't exist any more?
The condor project is still alive and well: http://www.cs.wisc.edu/condor/ and should do what this guy wants to accomplish (but not what he's asking).
ÕÕ
In the 1980s and maybe much earlier, the Cray supercomputers NLTSS operating system had this feature to allow stopping/restarting of applications and it was called checkpointing.
For Windows NT, Lucent had a group that developed Fault-Tolerance software which had a checkpointing feature. This was called SwiFT.
http://www.bell-labs.com/project/swift/
At the same place but under the support, there is some mention of a Unix version.
WhatMeWorry
You don't need hibernation for that.
Yeah, not really relevent to the main topic, but any modern PC's do have suspend support built into them, so the no-additional software thing is a pretty moot point.
Hibernation IS a software thing, and it just means that when the OS receives or generates a shudown-hibernate event, that the OS writes all available memory and state to disk and shutdown, setting a flag that the OS can know that it was hibernated to begin with.
Bye!
Eros has in its version 2 spec moved away from this idea implemented for all processes because of difficulties with retaining state for various knew intertfaces such as USB. But the core idea remains for some processes I beliver. Not sure what the current state of play is with eros as it seems to have a lower profile than it deserves. God forbid geeks more interested in coding than talking.
check out for more details.
hopefully Jonathan Shapiro will will add more details here if he sees this thread.
When the UPS daemon senses that it's time to shut down, it sends all processes a SIG to warn them. This gives each process a chance to clean up, save state, and exit. Your program just needs to respond in the appropriate way to the SIG your UPS daemon already posts, so it can resume where it left off next time it's started. Doing this on an OS-wide basis, I think, would be overkill.
Try it with vi sometime. Start editing some file. Then, from another window, kill -HUP it and watch it checkpoint everything. "vi -r" recovers.
The whole point of SIGHUP was to give your programs a chance to do something reasonable when your modem (or whatever) connection drops for whatever reason.
Python supports a concept that it calls 'pickling' (which is also known as Object Serialization).
It's extremely easy to save the state of any object along with the objects it references to disk with literally a couple of lines of code (like, 3). You cannot pickle whole processes, but it's effortless to write some skeleton code to resume the process from its last pickle. You can also define specific methods in each object that are called on pickle/unpickle for special cases (restoring network connections, for example).
The fact that it's an interpreted language shouldn't deter you. Python integrates easily with modules compiled from C, allowing you to accelerate time critical aspects of your code while rapidly developing the not so critical aspects.** Python was designed to solve the problems you're working on.
Oh, and if you're short on time, don't worry; Python is extremely easy to learn.
** As most programmers have found, about 90% of their program's execution is spent in 5% of their code.
EROS' predecessor, KeyKOS, made waves at USENIX when they did a demo of a UNIX system + Xwindows which would instantly restore the running state of all software when rebooted. It was basically a UNIX port to KeyKOS, and since everything in KeyKOS was persistant, so was everything in the UNIX.
One interesting caviat with this type of OS is that you really need to use ECC memory, because bit errors can get saved to disk and propagated forever!