Slashdot Mirror
UNIX Process Cryogenics?
shawarma asks: "Due to a recent
power outage, I've had to shut down a server running a process that had
been running for ages calculating something. The job it was doing would
have been done in a few days, I think, but I had to shut it down before the
UPS ran out of juice. This got me thinking: Why can't I freeze down the
process and thaw it back up at a later time? It ought to be possible to take
all the connected memory pages and save them in some way, preserve file
handles and pointers, and everything. Maybe net-connections would die,
but that's understandable. Has any work been done in this field? If not,
shouldn't there be? I'd like to contribute in some way, but I think it's a bit
over my head.." Laptops have been doing this in some form for years:
most laptops, when they run out of power, or when told by the user will
go into "suspend" mode which is similar to what the poster is describing,
however outside of laptops, I haven't seen this done. Sleeping processes
also do something similar, sending their memory pages into swap so other
running processes can use the memory. What, if anything, is preventing
someone from taking this a step further?
is not suspend, it is hibernate. Suspend will power down the computer except for the energy needed to keep the ram alive. hibernate will save all data to from memory to disk. I, personally, use neither on my laptop.
Remember, there were no nuclear weapons before women were allowed to vote.
Of course, you could write your application so that it saves state at regular intervals (aka checkpointing). Especially with calculations you should be able to store intermediate results.
MSN 8: Now Microsoft even has bugs in their ad campaigns.
External dependancies might include open files (what if you freeze, and then delete the file?), open TCP sockets to daemons elsewhere that wouldn't get frozen, sub processes, etc... These would probably have to be revived, but how?
http://www.cs.wisc.edu/condor/
Free-as-in-beer, on most major UNIX platforms. Check out our publications, we have several that give all the details you'd need to write it yourself.
Plenty of others, too - libckpt, there was a "Checkpointing Threaded Programs" paper at USENIX this past summer... there are some kernel patches that can do, most of them under the GPL.
for the "Classic" environment. It seems so stupid watching macos9 boot up in a window when you want to use a classic program; Apple ought to save the state of the classic environment in to a file that could be quickly reloaded into ram when classic is called for. As the blurb said, laptops have had the suspend feature for years; would it really be so hard to apply the same concept elsewhere?
___
The way to see by faith is to shut the eye of reason. --Ben Franklin
They do exist in other systems, or at least, they work on other systems.
My laptop has no problems suspending/hibernating linux.
The question here is about process hibernation, not the whole box.
I had Be installed for a while and I thought it would do that. I do know I never lost anything due to it crashing. Of course, it didn't crash much. I think using a journaled file system or at least soft-updates would be a good start. Frankly, I have no idea how to code something simlar to Win XP hibernate. Shouldn't be that hard though.
--- Think of it as evolution in action ---
What you want is known as "checkpointing."
There have been a number of projects that do this under Unix over the years. Many of them do it for the purpose of process migration. Others do it just for recovery.
One such project that I used in the early 90s was Condor.
The typical approach is to do something along the lines of forcing a core dump and then doing some magic to restart the process from the core file.
The difference is that suspending a laptop is done using hardware, but the suspend mode in WindowsXP is done in software, so desktop PCs can do it without additional functionality.
Ewan
VMware suspends to disk. You can go as far as suspending the Virtual Machine, not Virtual Memory. Then copy the "data" files to another machine and resume the same suspended virtual machine like nothing ever happened, as long as the same basic hardware exists on the host system (e.g. NIC, sound, serial ports, etc).
While this isn't quite what you are looking for, it spawn an idea of the level this can be taken to. Think of how neat it is for distributed applications. Of course, something like this has to exist somewhere. . .
Almost all of the stuff you need is already in a core dump. Perhaps the appropriate approach to this is to try to extend the core-dumping mechanism to also dump other pieces of state. Then you would just need a way to reconstruct process state from a core dump, which most runtime debuggers can almost do anyway.
I suspect that all the pieces of a solution are written and it's just a tricky pick-choose-and-integrate problem.
And damn but I'd love to have this ability.
--G
The job it was doing would have been done in a few days,
In that case, Arthur Dent should know the answer.
back in the day there was a post:
1 2&mode=thread
http://slashdot.org/article.pl?sid=99/10/28/01512
about an operating system with "journaled" processes of a sort, that would automatically back up images of it's processes.
It's called software suspend for linux. look for it on freshmeat.net
Do not look at laser with remaining good eye.
He's talking about on a process-level, as in freeze a lengthy game of Asteroids and restore it later. Hibernation is system-wide, not on a process-by-process basis. And Linux has that too ;-) Note: this comment is reused!
There has been a lot of work done on "process migration". That is moving processes from machine to machine.
Obviously those techniques would apply to what you are asking about.
google has lots of links about it
I once had an enourmous computer working out a very important question but it was destroyed by Volgons five minutes before it was finished. I feel your pain.
spacefem.com
I think the same solution would apply here: Find Arthur Dent.
Chris Kuivenhoven is a thief, beware
The answer is 42. :D
+5:offtopic,but anti-American
This sounds like common sense to me. You never know when the disk is going to poop, the power shut off, the network reset.
At my old job, we were required to record the status of all jobs that took longer than an hour (on a 6 cpu SGI). They never crashed on their own, but I would usually interrupt them if the requirements changed or whatever. If they ever did crash, then there was a record of exactly where they left off.
I've always wondered how hard it would be to resurrect a core file. One would think that there's enough info in a complete core to reopen all the open fd's, and possibly even reinitiate network connects. Everything else is there-- program counter, stack, heap, etc. As such, one could 'kill -ABRT' the process and revive it again later. Has anyone seen this done?
So long, and thanks for all the Phish
You can't just serialize and page out one process. Under every process are a slew of kernel objects and kernel crud including the virtual to physical mappings of your address space. It would be quite a challenge to isolate all of this and somehow persist it.
To make suspend work, you'd have to dump your entire memory image to disk. Then you swap in the entire image, kernel and user pages alike.
Someone you trust is one of us.
1) Produce the core dump of a process
/. posts this as a relevant question is very
2) Use the core and process image to restart it
(for example in the debugger such as gdb, if you
don't want to write specialized software).
To the best of my knowledge perl "compiler" uses
precisely this technique to produce perl "executables" - dumps them out as a core right
after compilation and reuses it later on.
You can do this to a kernel as well, if you
REALLY want to.
However, since indeed many things may be dependant
on state of kernel, files, network connections, devices etc. etc. doing this is not adviseable.
Good coding practice for long-running processes is
to actually spend some time on writing the state
saving functionality to support process restart.
Anyway, (call it a flame if ya will) but the fact
that
disquieting - level of technical knowledge here
gets reduced day after day.
I've used the Suspend/Resume feature on a sun box. IIRC, it mostly worked, but with a minor hitch that made me worry enough to never do it again. This suspend/resume is just like the laptop version -- save a copy of all memory to disk -- not the cryogenic per-process version you're talking about.
The per-process sounds neat, but usable only if you've got a simple critical task you're running. For a more complicated application, multiple processes may be working together, and you'd have to suspend all of them at the same time.
One big question I would have would be file handles... if you restore a process that thinks it owns file handle #5 and some other process is already using it, it would be awkward to get either process to use a different handle.
HIV Crosses Species Barrier... into Muppets
First, let me say that what the poster is suggesting sounds a little more sophisticated then a simple re-implementation of XP's hibernate function, although functionality like that under UNIX would certainly be invaluable. It sounds like the poster wants control over individual processes, something that I consider far more interesting.
What's said here is certainly very reasonable. But the extensions of whats being suggested are even more fantastic. Once a process is completely removed from memory, with file handles and storage and status all kept away safely, is there any reason that the process is really tied to that computer? Why wouldn't it be possible to take that 'frozen' process, transfer it to another machine with access to the same filesystem on some level (some translation of file handles would likely be neccesary), and thaw it there, allowing someone to move a running process to another machine? Need to replace your web server's only CPU, but don't want downtime? Move the process to a backup machine, replace the original's hardware, and move the process back.
I even thought I had heard that someone was working on just such a project, or at least thinking about the details of implementing it. (I'm just getting started in learning UNIX internals myself). Anybody have more references to information on this sort of thing?
"You know, Hobbes, some days even my lucky rocketship underpants don't help" -- Calvin
A different solution, which is very common for long running processes, is to use savepoints, i.e. save the state of the process regularly to a file at suitable points of the algorithm. Once your process dies or you killed it, you can restart from that savepoint. If your state information is very large, you can stretch the save interval to reasonable long times, e.g. several hours. Typically you don't mind to lose some hours of calculations due to an occasional power outage.
Of course this solution is not as general as the "process cryogenics" you describe, but it's also easier to implement because you have more information about the problem.
There's no reason why you can't do it either in an app by saving state or in the OS by saving memory to disk as on a laptop.
GEOS had the concept of state-saving in the OS circa 1990, so it's nothing new. The UI saves its state, what apps are running, what windows are open, etc. and restores it exactly as you left it when you restart. If an app has extra data to save, such as where it was in a lengthy computation, it can save it, too.
A slightly different approach than brute-force writing out all of used memory, but both work quite well with the speed of current hard drives.
This facility is called checkpoint/restart. It was a feature of OS/360 and other operating systems in the 1960s. In some very early versions of Unix, core files were restartable. Usually it's pretty easy for programs to save enough state to be restartable on a case by case basis, except when it's just about impossible (like when networks reconfigure) so it's not a popular system feature these days (hard to implement in a general way, doesn't do a very good job in the cases that can be handled easily.)
A friend of mine (Hugh Redelmeier) ran a very long (~400 day) computation on a PDP-11 in the mid-1970s. The program ran stand-alone, and part of the test plan involved flipping the power switch on and off a few times -- very amusing to watch the program keep on running right through power failures. (Main memory on the machine in question was magnetic cores, which are non-volatile.)
-Tom Duff
Vmware does this for the VM's it hosts. Works great.
Creed
All that is necessary for the triumph of good is that evil men do nothing.
Any program that you intend to run for more than a day or two you should checkpoint its intermediate results to disk, even if this adds 100% to the run time.
--Blair
P.S. Alternatively, you could write a program to have the rebooted computer pull scrabble tiles from a bag structure and print them to the screen. You might at least get some clue as to whether it was asking the right question.
This next one would complicate things a bit: the user should also be able to wake up the process the same way, i.e. kill -WAK $PID. This means that an index of hibernated processes also needs to be kept synchronized between the kernel process tables and a file on disk, to be preserved between reboots.
Maybe I'll write another kernel patch...
For long-running processes, rather than shut down the process when the UPS kicks in, I've always found it easier to have the program snapshot its data tables periodically (say every half-hour) and build a "resume from disk" feature into the program. This lets you restart the program from its last check-point even in the event of uncontrolled program termination (e.g. kill -9 and the like).
-JS
Vanity of vanities, all is vanity...
The main reason this "suspend" feature works relatively well for a laptop is because the hardware is a "given". The laptop has to have a certain video card and motherboard chipset, specific type of hard drive, floppy, CD-ROM and sound device. (In fact, when laptops fail to come back up properly from a suspend, it's almost always the one "add-on" card people have in laptops, the PCMCIA network adapter, that causes the problem.)
3Com PCMCIA cards are about the only ones I've used that allow the laptop to power them down and back up again, and resume network activity without a complete machine reboot.
The comments to the effect of "it's called hibernation, and has done it for years" are missing the point. That hibernation is a BIOS supported dump to disk. It's a feature on most laptops and works with just about any OS -- it's worked on my Linux laptop for years.
/var/longoperation.pid`
I think the feature to be discussed is Operating System (not BIOS) level support of the hibernation of a single process. It'd be nice if I could do a:
kill -HIBERNATE `cat
and have that program get frozen to disk. Then if I could resurrect just that process later it'd be a handy feature for the long running program that you want to postpone until after you've done whatever you needed to do in single user mode.
There are big problems with such an approach, and mainly with device usage. Basically they are all the problems that you would have with process migration add a few because of temporal discontinuity.
If you are using a scanner, or a mouse, or whatever, that device may not be there or may not be available when the process is brought back. Furthermore you may have a file descriptor opened on a local (or network shared) file which no longer exists or has changed drastically.
There are further non-device-dependent problems with shared memory, opened-but-unlinked files, parent PID, IPC resources.
Having said all of the above... I suppose that for the very rare case that your program is completely memory and CPU dependent you could retire and recover a task.
my $0.02
-- bartman
The idea was that when you put your computer to sleep, instead of keeping the SDRAM (or whatever the laptop had) powered to preserve the memory contents, it would write it all to a special sector on the hard drive that the firmware knew to read from when starting from sleep. This allowed sleep to be even more low-power than it already is, since a hard drive does not require power to retain data.
--
"If you are an idealist it doesn't matter what you do or what goes on around you, because it isn't real anyway."-R.P.W.
If you could sleep processes you could run some intensive job at a high priority when your not logged into your workstation and then sleep the processes when you log in. This way you could run some job that takes weeks or months but not bog down a workstation that you need for doing daily work on.
Yeah, you could "nice" down the process so that it doesn't slow things down while your logged in... but then system processes at higher priorities might slow down your number crunching when you're not logged in... It'd be best to be able to run it at high priority at night only.... ya know, use those unused cycles.
There are 10 types of people in this world, those who can count in binary and those who can't.
One fairly simple alternative is to simply have the application save it's own state to a "checkpoint" file periodically. This approach has been used in other applications for a long time in the form of auto-save files (ie: emacs) and would be easily adapted to a long running program like the one you describe.
Just because the OS doesn't support it automagically it doesn't mean that you can't solve it for yourself with a little bit of extra work and planning.
If you are a grad student who has been waiting for a month for a job to finish...this could be very important. I was in an engineering department where jobs that ran for weeks were not uncommon (fortunately most of mine only took a day or two). A shutdown of a critical machine could set someone back months.
Linux software suspend may be of interest.
So, you mean that the next time my app segfaults and dumps core, I can say it was a feature designed to allow it to be restarted...? Cool. Seriously though, how can you restart a core (obviously not one from a segfault) using gdb?
The bastard children of Vogons and Vorlons?
Best Slashdot Co
Long ago and far away (about 15 years ago) I recall that TeX was frequently built in a fashion that required running the binary on some "initialization" information. That process took some nontrivial amount of time back in those days (I'm sure now it would be an eyeblink), and the program could be made to \dump its state in some way.
Then, when you ran TeX in everyday circumstances, the digested initialization file was read in by the application as part of the usual startup process.
I'm probably botching the explanation of how this really worked, but I guess my point is that the "resume" function had to be coded into the specific application.
"Provided by the management for your protection."
Once you've enabled it, you create a hibernation file on the C: drive. Hibernation should only take place when there is minimal disk activity (eg, don't hibernate while trying to save your Word document). The system saves the contents on RAM to the hard drive, and then shuts down. When the machine boots, a flag was set (I assume) indicating the system should resume from hibernation... so the hibernation file is read from disk and written to RAM and you're back up and running, in less time than it takes to boot. Plus it keeps your uptime from resetting back to zero.
Some things to note:
You will need WHQL certified drivers, or at least properly-written drivers. I have a SB Audigy and the first drivers I used (the ones on the included CD) caused a blue screen on resume from hibernation. When a updated driver was released, it fixed this issue.
Applications need to be properly-written as well, as there is some sort of Win32 suspend signal that is sent to apps just before the system hibernates, so the app must support this and the resume command when the system is restored.
Hibernation works great on my laptop and on my workstation, and I especially like the fact that I don't need to create a separate partition or install special drivers to make it work (you can even use it on an NTFS formatted drive).
But isn't it overkill for a data-crunching operation? As many other people have noted, it would seem you're much better off checkpointing your data to disk, rather than relying on low-level OS process wizardry.
Sig: What Happened To The Censorware Project (censorware.org)
No, I wouldn't design a totally new memory dump system, I'd keep logs. Have the app keep track of where it is so that should the system restart it can pick back up again. That could be done without new BIOS and memory systems.... And you could do it TODAY with your existing hardware setup.
There is a kernel patch to do this. It's called Software Suspend. It is also part of the FOLK project (Functionality Overloaded Linux Kernel, a project to merge the largest possible amount of patches into the kernel).
The filesystem is the package manager
Surely if this process takes so long to execute the person who wrote it should have made it save its state every once in a while. Problems like these can have been avoided! Setiathome to name but one does exactly this.
James
My Intel processor puts it somewhere around 41.99999999967
This comment is far from (Score:4, Informative) ... it's not even relevant. We're not talking about the whole OS hibernating, we're talking about saving the execution state of an executing process so that it can be resurrected later and continued (ie. if a reboot is necessary).
----- rL
You can't. The previous poster was making it sound too easy. Real checkpointing needs to save Kernel state as well -- file handles, device driver state, you name it. It isn't as simple as saving the in-memory image of the process.
I think that this might also be a really good bug fix/hacking tool. I can also remember something like this for the Apple II in years gone by. You could press a button and take a snapshot of all memory in the system. Then you could write the executable part to disk and pick up where you left off. Good for freezing a copy of a game or whatever.
This would also be good for tracking down bugs using the "before and after" technique.
Such a program could be tied into the UPS monitor in such a way as to save everything that couldn't be stopped.
As usual, this is ancient. Back at FSU, we had a CDC Cyber 205, a vector pipeline supercomputer, back in 1985. Any process could be crashed for a shutdown, and it produced a file that worked exactly like an executable and resumed computation from the time it was crashed.
I was thinking about this and here was my dirty hacky idea. You need kexec, lobos, or something similar (actually a fairly modified version of it) you'll need on the order of 8MB of disk space and some kernel mods, which might not be that extensive.
I was thinking we develop some driver or process that consumes all of the memory and CPU in a system. It forces all of the processes to swap out, it would probably need to be a driver of sorts on current linux systems. Then it could dump the kcore out to a file somewhere, sync it, and hibernate. Then when the kernel boots up, if the right arg is passed in it could either load this image back in to ram in place of the kernel and then jump into it (easier said than done) early in the boot (page tables are made long before you have access to the drives and such so the logistics of this would need to be figured out) or it could boot up and use a different swapper partition and then have some kind of tool like kexec to load that image back in to ram and start it up. Or something, some how you should be able to recover the state of the system. File handles and everything would be there.
The harder part would be hardware and network transparency. You'd need to modify all of your drivers to make sure that the hardware could be reset and they could deal with it. I think it's a little easier for the network side because it would be similar to simply unplugging the network cable, you have open sockets that are talking to nothing and some software can deal with that pretty well. There is also some kind of system integrity or robustness piece that is needed, if the system some how changes when you bring your old image back it could break things, munge files, etc..
the seti@home client uses its *.sah files to save the state of a calculation. of course, this is program dependent, not OS dependent. I guess if you have the source files for the program doing the counting.....
Tequila: It's not just for breakfast anymore!
STANDALONE CONDOR CHECKPOINTING:
..
...
Using the Condor checkpoint library without the remote system call functionality and outside of the Condor system is known as
"standalone" mode checkpointing.
To link in standalone mode, follow the instructions for linking Condor executables, but replace condor_syscall_lib.a with libckpt.a. If you
have installed Condor version 5.62 or above, you can easily link your program for standalone checkpointing using the condor_compile
utility with the little-known "-condor_standalone" option. For example:
condor_compile -condor_standalone [options/files....]
where is any of cc, f77, gcc, g++, ld, etc. Just enter "condor_compile" by itself to see a usage summary, and/or refer to
the condor_compile man page for additional information.
Once your program is relinked with the Condor standalone-checkpointing library (libckpt.a), your program will sport two new command
line arguments: "_condor_ckpt " and "_condor_restart ".
If the command line looks like:
exec_name -_condor_ckpt
then we set up to checkpoint to the given file name.
If the command line looks like:
exec_name -_condor_restart
then we effect a restart from the given file name.
Any Condor command line options are removed from the head of the command line before main() is called. If we aren't given
instructions on the command line, by default we assume we are an original invocation, and that we should write any checkpoints to the
name by which we were invoked with a "ckpt" extension.
To cause a program to checkpoint and exit, send it a SIGTSTP signal. For example, in C you would add the following line to your code:
kill( getpid(), SIGTSTP );
Note that most Unix shells are configured to send a TSTP signal to the foreground process when the user enters a Ctrl-Z. To cause a
program to write a periodic checkpoint (i.e., checkpoint and continue running), sent it a SIGUSR2:
kill( getpid(), SIGUSR2 );
In addition to the command-line parameters interface described above, a C interface is also provided for restarting a program from a
checkpoint file. The prototypes are:
void init_image_with_file_name( char *ckpt_name );
void init_image_with_file_descriptor( int fd );
void restart( );
The init_image_with_file_name() and init_image_with_file_descriptor() functions are used to specify the location of the checkpoint file.
Only one of the two must be used. The restart() function causes the process image from the specified file to be read and restored.
I think it was somewhere in the list of patches from the -mjc tree (see here) that there was a patch for the entire kernel for linux. Basically it let the system save it's state, and then restore it if it detects that it was shut down at that point. I'm not sure if this is what you want (and I couldn't get it working), but it's certainly a step in the right direction to what you're looking for.
Just found it here, it's the 'swsusp' patch.
If you utilize the java.io.serialization stuff right, you can create a lightweight persistence and should be able to freeze and resume processes on the same application if you handle threading right with it.
The answer would have been 42 once the processing was complete. So who cares? Get a bigger UPS :-)
Here's a mutation of FreeBSD that can do exactly that. I've put my laptop to sleep in the middle of installing software while running MacOS X and brought it back up several hours later to resume installation with no problems. The same function works on my G4 tower. Yes, it does drop network connections. However, it does use a trickle charge to power the LED's and presumably to keep the processor alive, and possibly some memory. Paging several hundred megabytes in a couple of seconds would be quite the task! One item of note is that all Apple machines have a special piece of hardware known as the PMU (Power Management Unit). In the desktops, it's parted out onto the mother board and into the power supply, but in the laptops it's a seperate card which controls both sleep and the charging of the battery. Perhaps other UNIX machines would need a similar device for this function to work properly.
Karma: Ran over your dogma.
Easier said than done. If this wasn't part of the application's design or if it's relatively sophisticated, making these changes can be non-trival. And (shock/horror) if you don't have the source code, it's impossible without OS assistance.
I think this problem is more easily solved in hardware than in software. With recent advances in solid-state memory, hopefully a standard can be worked out so that solid-state memory can replace or complement volatile memory (i.e., RAM as we know it.) Solid-state memory could would survive a power outage, and you could pick up where you left off.
The disadvantages are speed (solid-state memory is getting faster all the time, but it is still slower than volatile RAM), cost, and lack of current standardized implementations (I'm not even sure there are any working implementations.)
For some background research in solid-state memory, check out this site (it's a bit old, but still interesting.
What if the process has forked off a bunch of children? Are you going to archive all the children at the same time? What if the process has a whole bunch of files in /tmp, are you going to roll them up into the freeze state as well? What if your using pthreads? Are you going to keep the state for each thread? How about file pointers?
I think the better solution is to write a new signal called "SIGFREEZE" and have programs just write code that could handle such an event. Let the program figure out how to save their own stuff.
A good example would be a program that was calculating pi. The programmer would have to implient a signal handler that would when it recieved a SIGFREEZE would stop its computating and write what its currently working on out to file. The other thing the programmer should be doing is periodically writing their data out to a file anyway. Then the programmer should have implement a command line option that would facilitate reloading from a saved state.
Thats my take on it...
If you see any problems with it... bring it on.
Yes Francis, the world has gone crazy.
This could be done without doing anything to your BIOS; youc could just dump all the memory allocated to a certain program to disk and put that process in a list of hibernating processes. What's so hard about that?
It only is worth it if you expect to have to halt the program more than once. Assuming only one halt and restart, VMware is still slower.
Don't label something "offtopic" unless you know the topic well enough to tell what's on topic.
A file descriptor is a per-process entity. Yes, there's a big table of file descriptors that exists for the entire sstem, but file descriptor 5 for process a is not file descriptor 5 for process b. Not even if they point to the same file/pipe. A case in point is FD 0, aka stdin. Every process starts out with a stdin on FD 0.
More important is how do you tell the kernel what file descriptor 5 pointed to? What if the file/pipe doesn't exist any more?
Of course, if your running some job which could take a month to finish... you code it so that it can pick up where it left off, or at least where it will only have lost a couple of hours-worth of work at the most.
Or is that too sensible?
(and if it's a proprietary package and it can't pick up from where it left off, find a different one).
My concern with that is this: Let's say something buggy is making the system crash. Then if the persistant OS does it's job with perfect accuracy, it's just going to end up re-creating the conditions that caused the crash, and Boom - crash again. The only way to avoid this is to NOT succeed at the goal of re-creating the conditions before the crash.
Don't label something "offtopic" unless you know the topic well enough to tell what's on topic.
While I love VMWare, it does consume a substantial amount of CPU/memory. The problem is a job like what the original poster described is usually CPU or IO bound, and VMWare just starves the process from what it needs even more.
:)
Granted, it is a solution, but your job that ran in 3 days just got pushed out to a week. It's just a tradeoff.
What the poster really needs is to rewrite the program to drop intermediate data along the way. If you have hourly checkpoints you can minimize the amount of data lost. How to implement checkpoints is left as an exercise to the reader
If memory serves me (hey, it is Friday after all and both brain cells are pretty tired) we looked into something like what the poster was asking about years ago. In those days, we were running some simulations on a PDP-11/70 that took 7-10 days to complete. In the event of a general power failure we wouldn't have been able to run on backup power for very long. DEC's RSX had a feature whereby a task could be checkpointed to disk. Then, presumably, it could be reloaded and resumed at the same state it was in at the time of the checkpoint. We never did implement it since it would have introduced too much delay into the project schedule (adding it to the simulation, testing, etc.) but it sounds like the sort of thing that could be useful in current day OSs. Anyone know of any general purpose operating systems today that have this feature? I haven't heard of any and wonder (not too seriously, mind you) if anyone sells core memory for a PC architecture computer. Of course, it wouldn't be very fast but you'd worry a lot less about power failures that are longer than the UPS's ability to provide power.
CUR ALLOC 20195.....5804M
This is why VMware suspend works the way it does. It provides a consistent virtualized hardware interface, regardless of the details of the real hardware. The original question referred to individual process saving, and VMware suspend is similar to the whole OS suspend feature in laptops. Nevertheless, if you consider VMware to be a wrapper for individual processes that you want to be able to checkpoint, it turns out to be quite a nice solution to the original problem with zero programming required, and just a little pocket money to implement.
bb
you can always dump core of the process
(e.g., kill -SIGSEGV), then load the core file
it into gdb (gdb program corefile) and
issue 'cont'.
The OS state would be gone though (so, no
files besides stdin/stdout), but for purely
computational process that might work as a
one-time shot. At least you could save main
arrays from gdb and read them in into a modified
program.
...and found esky, a purely userspace checkpoint/resume implementation.
I remember an option in Solaris 7 that lets you dump memory to swap, shut down the computer and when you restart it reads swap and drops you back into the exact same state as you were in before.
Pretty cool because you could restore to a full X-session with all the programs and documents you were working on before undisturbed.
I don't know if this is what you were looking for. . .
I think laptops work by getting the applications and OS into a safe and simpler state and then saving that state. I suspect they cannot save any arbitrary application you could write - just the applications they routinely run.
If you've ever used a laptop with this feature you'd realize what you just said is totally wrong.. the hibernate function of these laptops is managed by hardware not software and so is os and program agnostic. When you close the lid or hit the sleep button, it dumps the entire state of the ram in to a special partition and turns off.. when you revive it, you are back exactly where you left off, regardless if you are running windows or linux or if you are playing quake3 or cracking rc5 stuff.
By managing this stuff in hardware, its actually less complicated and works 100% of the time as opposed to the windows software solution that often refuses to 'wake' after being put in sleep mode and is dependant on the power supply being on and supporting the feature.
If someone were to add a feature like this to a large multiuser mainframe type system, it would definately make more sense to go with a hardware based solution that dumped the system state to a disk or multiple disks to ensure that it always worked and not just some of the time for some of the apps.
At first this seems like a nice idea. It would be elegant to be able to halt processes and resume them later without them consuming resources in the interim.
Before going forward ask yourself what the practical application of this work could be. If you have to reboot systems with long running computational work going on you may need more reliable hardware or better management of the system to increase uptime. Furthermore, adding "suspend/resume" functionality to a single process within it's own code would probably be far better as needed.
Secondly, think of the concerns you face in implementing this as a generalized solution for user processes. Here are the problems with this concept that I can see.
First, file handles, file system pointers, network connections may not exist when the process is restarted. Let's say that there is processing of NFS data being done and when the process is resumed that mount is no longer accessable. You get an error from NFS like ERRIO or the like and the process dies.
Secondly, the hardware may no longer be available. What if the process what using a PCMCIA card which has been removed. The process dies. In a more simple case, a process could have a tty open for I/O and that tty may no longer be owned by the user when the process is restarted.
This requires saving a lot of system state and does little to guarantee that the process can be restarted successfully and safely. Furthermore, the dependancies for a single process (being fairly complex) would require a good knowledge of the process by the user to determine the feasability of suspending and resuming the process.
It seems that this would not accessible by average users of the system if it were possible to create in a generic sense.
It does stand as a good question to start someone thinking about unix internals though.
Something many people not familiar with J2EE (Java 2 Enterprise Edition) know is that when you have an application running in a Java container, it, and the state of all its processes get automatically saved and restored whenever the container, the OS, or the machine crashes. True, in practice some diligence is required from the programmer (for example, when you need to set obejcts to specific state upon re-instantiation), but the functionality is there, is OS-independent, and it's been proven and used daily in heavy-duty environments for a few years now.
Yeah, not really relevent to the main topic, but any modern PC's do have suspend support built into them, so the no-additional software thing is a pretty moot point.
Hibernation IS a software thing, and it just means that when the OS receives or generates a shudown-hibernate event, that the OS writes all available memory and state to disk and shutdown, setting a flag that the OS can know that it was hibernated to begin with.
Bye!
...why not just boot up classic at startup? My brother set his computer to do this, you can too if you don't want to wait.
Basically that was one of the ideas behind the research on micro-kernels. If the state of the system gets small and centralized enough one could not only make a single process persistant but the full system persistant.
KeykOs was a very promising system offering this at the time. One could not checkpoint the connections outside of the machine, but their demo was a BSD machine with X11, whose powerplug was violently removed. When replugged the state of all processes saved at the last checkpoint was resumed and the system would continue ... Including X-Windows !!!!
Now wait for the Patent to expire, put it in Linux and watch the world of computing change.
It was very promising at the time I was doing my PhD 10 years ago, I don't know why this never "made it"
Daniel
It's not possible to hibernate a single process.
Wow, so the fact that its been done here is just a red herring?
Does Virtual Memory mean anything to you?
I demand a million helicopters and a DOLLAR!
Check out http://www.eros-os.org.
EROS processes persist until you take them down. They persist across power loss, system upgrades, etc, etc.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Sun already implements a system suspend/unsuspend in Solaris that works on all boxes but the Blade 100's.
:-)
10 years ago I worked on a Unisys Unix box that did it automatically, meaning you could pull the power out of the wall without any warning and then plug it back in later. When the system rebooted, it would say "there's been a power failure, recovering" and then put all the processes back to the way their before. Even with an open vi session where I was actively typing, I wouldn't lose more than a character or two.
I found out the machine had it quite by accident because my loser boss turned the box off one evening without doing a proper shutdown... Once I saw what it did, this required further testing
Still, what would be even better is if it could be done on a per process basis. I can think of many reason why you might want to suspend a process for a few days and bring it back later (say something you only wanted to run outside of work hours), but had no intention of shutting the whole box down. And this should be implemented in the kernel, not hacking each program to provide this functionality.
What if the process has forked off a bunch of children? Are you going to archive all the children at the same time? What if the process has a whole bunch of files in /tmp, are you going to roll them up into the freeze state as well? What if your using pthreads? Are you going to keep the state for each thread? How about file pointers?
Back in the 80s, Cray UNICOS had a cadillac checkpoint package. It could track child procs, save /tmp files, save threads, save pipe data, and pass down SIGCKPT for user-controlled checkpoint.
Of course at $1000/hour you want to damn sure be able to save your work :-)
On a palm you can shut if off and when you turn it on it is where you left the device at. I think it would be neat too if this could be the way operating systems worked. Ideally one would be able to turn off the computer in the middle of an app and it would turn on at the same place it was left at.
Of course the palm does not do multitasking, multiprocessing or anything like that and when you close an app it is usually sent back to its initial state.
Maybe the way to do what this user wants is to take journaling to a next step, and rather than have a journeleing file system have a database file system where stuff is done in commits like a jfs. Then one could do rollbacks as well. This would require the whole system to be rethough out though.
Only 'flamers' flame!
what if the OS had a hook in it to like
`kill -FREEZE <pid>`
No new hardware, only done once, will work on all processes.
And as described previously, the FREEZE signal would cause the process to dump execution code, memory pages, FD's etc. etc. to a dump file.
reboot the system.
Then find some way to execute that dump file which will in turn load FD's, pages, execution code, and resume with the IP (instruction pointer, not IP Addr. for those not arch inclined) in the same spot?
/me isnt much of a kernel hacker so I dont know the details of how to do, but that's my high level solution.
There are more than power problems to worry about with a long running process. There are other hardware failures, scheduled downtime, and system crashes to contend with. Just becuase in this instance it was a power failure that made him wish he had this ability doesn't mean it wouldn't be useful in other circumstances.
Acer Travel mates (well my 312T) do the same thing in SUSE linux. If you shut the case it goes into suspend mode (function key +f3) to hybernate fully i needed to leave a fat32 40MB partition to dump the ram to as the bios didn;t seem to want to dump to the linux parition.
Obviously i dont use the laptop for large numerical simulations but i just tested it with a fortan numbercruncher program running and it woke up fine
Why are software techniques shit today compared to yesterday?
Because we're hopeless caught up in trying to reinvent a somewhat limited computing paradigm (unix). No one, except for some CompSci projects that never really go anywhere, have any real interest in making a new operating system that builds on the lessons of all the previous operating systems and includes reasonable features like process checkpointing/suspension.
I'd bet there are patent considertions as well -- maybe many of the good OS features are not reproducable due to existing patents.
Step 1: Clean, fresh install of XP Pro corporate.
Step 2: The requisite reboots until everything works.
Step 3: Leave the office, set computer to hibernate for fun.
I. Results
A. Blue screen of death upon return to office.
B. Reboot yielded '/windows/config file is missing or corrupt'.
C. Much cursing and a swearing off of anything Microsoft.
XP isn't as wonderful as people would have you believe. A short trip to google inquiring about repairing this mess will result in endless posts.
And if those mice were so smart how come they didn't think about it? Even I know that hardware fails.
make Linux, not Microsoft. sin(beast) = -0.809016994374947424102293417182819
This hibernation mode snapshot can be duplicated or even put on other machines in the event of a system failure. The virtual machine will then come back online like nothing ever happened, with hardware devices effectively still attached and processes still running.
It works really slick, you can perform other tasks and come back to your virtual machine later without slow boot times. This will also work on Linux, Solaris, and Windows platforms. I'd highly recommend VMware for on-demand OS access.
-Pat
Because that's not what he asked. He asked, and I quote:
This has different implications. Let's say that you have to turn off your system to replace a noisy fan, but you have a process going that could take a few days (a render farm or cluster is place this might happen in). You'd like to pause it and then resume it once the computer is back on. In order to do that, you'd have to save EVERY piece of information associated with the running process like memory used, files, etc. THIS is what the guy is talking about, not hibernating the whole computer (which, if the computer is running many processes could be an extremely bad use of hard disk space, not to mention time consuming - time is something you don't have when running off a UPS).
Cliff's only made the situation worse by saying "Laptops have been doing this in some form for years", but really "in some form" is a generalized stretch. It seems to me that its likely that its much more complicated to save and restore one specific process than it is to save and restore all of them in one big dump back into memory when the system recovers.
----- rL
Python supports a concept that it calls 'pickling' (which is also known as Object Serialization).
It's extremely easy to save the state of any object along with the objects it references to disk with literally a couple of lines of code (like, 3). You cannot pickle whole processes, but it's effortless to write some skeleton code to resume the process from its last pickle. You can also define specific methods in each object that are called on pickle/unpickle for special cases (restoring network connections, for example).
The fact that it's an interpreted language shouldn't deter you. Python integrates easily with modules compiled from C, allowing you to accelerate time critical aspects of your code while rapidly developing the not so critical aspects.** Python was designed to solve the problems you're working on.
Oh, and if you're short on time, don't worry; Python is extremely easy to learn.
** As most programmers have found, about 90% of their program's execution is spent in 5% of their code.
I just hibernate and system state is written to disk.
GNU Emacs basically does this to reduce initialization times.
I heard about this. But, my dear boy, I do believe that VI does this better and with more cryptic keyboard commands.
I'm thinking an internal UPS that operates sort-of like the "Shead" switch in small aircraft. When you have to use it to conserve power, it kills all power to all instruments on the copilot side, some lights, and other stuff, leaving only the radios and nav instruments operating - enough to fly but not enough to do much else.
The question is - how would Windows react to suddenly having it's, say...CD-ROM and floppy drives just cease to exist while the OS were running? I've accidently pulled power cables on drives that weren't in use, had no active handles on them, and hadn't even had media inserted into them that session, and they still caused massive problems in Win2K.
I don't think most OSes would react well to having the power to everything except the processor and hdd0 shead from under it, even to conserve power while a savestate took place.
EROS' predecessor, KeyKOS, made waves at USENIX when they did a demo of a UNIX system + Xwindows which would instantly restore the running state of all software when rebooted. It was basically a UNIX port to KeyKOS, and since everything in KeyKOS was persistant, so was everything in the UNIX.
One interesting caviat with this type of OS is that you really need to use ECC memory, because bit errors can get saved to disk and propagated forever!
As you can see, freezing and thawing UNIX processes could get quite nightmarish if you account for all of the possibilities. (Most processes don't use SysV IPC, for instance.) Even the most (seemingly) trivial of syscalls would need to be modified (all socket functions, for instance).
Note that it's a lot easier to freeze and thaw a virtual machine, because it's so much more self-contained -- all you need to save then is:
The only way the typical /.er can pick up a chick is with a forklift. -- AC
MacOnLinux has the same feature, for those of us not in Intel land.
Suse on my Dell Inspiron 7.5K used to work with the suspend key, but no longer (X just hangs).
But ancient software is involved.
That said, rather than hibernation I'd prefer a software-UPS or time-rollback widget. How viable would it be to keep a very high frequency incremental save of state (even just the contents of a limited number of folders would be useful)?
It would be useful to be able to send your machine backwards in time without requiring everything to be in a database or versioning system that requires explicit saves. I'd like to be able to remove the effects of every command in the history of all shells in reverse, in the right order, and have high-granularity access to previous states of a filesystem.
If I could do that for all the relevant accounts on various machines it would be like never having to worry. I could leave the desk when I want to, kick the power cord or make meatheadded mistakes, and could keep a less paranoid number of full backups. I'd be worried about the life of my hard disk though. Already exists?
What I'd like would really be one step further in the chain. Something like my palm or the old Cannon Cat. Turn it off, come back a week, month or year later and voila. You are right back at the same point you left, as if you never turned it off. The basics as I see it would be that ram gets written to swap as an image, (which is what the Cannon Cat did.) Then when your restart the box by tuning it on, ram gets re-initialized from the swap file back to the state it was in before power off. The other option would mean adding a small battery pack to a desktop. If you hit the power button on a box or pull the tail from the wall ram is maintained by the battery until you re-power the box. (or the battery finally goes south.) As I see it there shouldn't be any reason why a box once run through startup shouldn't be able to maintain it's running state almost indefinitly. In fact if you could get Linux to do this one thing..... it would be on desktops so fast you wouldn't believe it. Unless you change hardware what is the diffence that occurs that requires the full init sequence anyway? The Green Peacers would love it because people wouldn't mind turning off there comp since it's "instantly on". The only down side would be that you wouldn't want to stay logged in, but then what's the diff between being logged in with the monitor off and being logged in with an instant on feature? Course it would mean uptimes in years instead of days.....
I'm sorry, I'm to tired to be witty at the moment so this message will have to do.
And what do you do when the program is so obscure that you can't just "find a different one"? Write a new one from scratch?
Well, if it just lost you a months worth of work, then your in exactly the right frame of mood to go out and do so!
And of course, if it was open source you wouldn't have to write it from scratch...
But seriously, if a program is so mission critical (or deadline critical) that it is important not to lose a months worth of work, and the software has no safeguards to prevent this from happening, and if you can't add any yourself AND you go ahead and use the software anyway... well your a fool and deserve everything you get.
Or at the least, learn a nice important lesson. And then go and rewrite the software.
And the same deal works no matter what the timescale. If the software isn't up to scratch, then get or make some that is.
Check out the software suspend patch for Linux. It allows the system to be suspended by SysRq-D (or shutdown -z) into swap space and resumed (or not) at the next reboot.
- Michael T. Babcock (Yes, I blog)