Slashdot Mirror
Complete Filesystem Checkpointing?
polymath69 asks: "Living on the edge of Debian unstable means that
updates sometimes break stuff, occasionally to an extent that is
difficult to recover from. This got me thinking about treating the
entire set of mounted filesystems as a transactional database.
Mark state, try something which might be dangerous, test, and approve (commit) or panic (rollback). Obviously some filesystem support would be required, but with ext3 and reiserfs available, maybe the potential is already there. And such a system would need lots of disk space, but these days that's a demand easily granted. There's lots out there on process-level checkpointing, and even some stuff about system-level checkpointing, but all I've found on that was in the context of saving and restoring processes for a system freeze and restore. But I couldn't find anything on Google or SourceForge about doing this sort of temporary branching in the filesystem. Is this idea feasible? Is anyone working on it?"
Well, looks like this guy Chuq is working on it. He seems to be a kernal hacker that works for VERITAS.
You can also find interesting filesystem info here
There's also work being done on TRAM (Transactional RAM).
I'm against picketing, but I don't know how to show it.
the default increment is daily
one can roll back the filesystem on a PER PROCESS basis with the yesterday command.
In this way you can narrow down what's broken by for instance using yesterday's c library, or last week's , or last years!
Also take a look at Venti
From: Sean Quinlan To: 9fans Mailing list
For those of you interested in the direction we are heading
with respect to plan 9's file system, you might want to
checkout our paper on Venti that will appear in the
USENIX fast conference.
http://www.cs.bell-labs.com/~seanq/pub.html#venti
Venti is a block level storage server that replaces the optical
juke box for a plan 9 file system. Some of the benefits include:
coalescing of duplicate blocks
compression
no block fragmentation
Also, we have switched from optical to magnetic disks as the storage
technology. I know many of you already use magnetic disks to
"fake" a worm, but for those of us using a optical juke box,
the performance improvement is rather substantial!!
seanq
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
It's a bit time consuming, but tar has been doing this job for me for a while. I finally (took me a long time, ok) created a working bootable linux CD that mounts my filesystems RO and tarballs the entire contents to a remote server. It takes about 1h20m to do a full backup.
After it finishes, I test software on the existing system. If it breaks I restore, if it doesn't I play-test it for a while and if it keeps behaving I commit it to the next full "state preservation".
The biggest drawback, of course, is that this sceme requires close to 1.5 hours of downtime for the backup and more if you need to restore. But for noncritical systems it works great.
I'm against picketing, but I don't know how to show it.
> Keeping a separate package database in one huge
> (ASCII!) file is advantageous. By being separate
> from the individual packages, the dependencies
> are kept nicely so they can be easily referenced
This is in no way an advantage of the implimentation format being one huge file. It could just as easily be one file per package, or one directory per package and multiple files referencing things like packing list, uninstall commands etc, plus a static, throwaway index file. It makes updates faster and a lot less disaster prone.
> As an ASCII file, any sort of little error
> which might cause a problem is easily
> correctable, as opposed to other packaging
> systems.
That's great until your filesystem decides to truncate it to 0 bytes, and it's lots of fun looking for parse errors 40,000 lines into a file and guessing what might have been in the garbage that you find.
> Regardless, this is not what the article
> submitter was talking about at all.
It is one of the dangers of Debian; upgrade to the latest unstable apt -> nuked package db, upgrade to the latest kernel and get fs corruption -> nuked package db. Find your ATA driver/chipset is buggy on handling large files -> nuked package db. Install a borked package -> nuked package db.
LVM provides the ability to create snapshots of your volumes, so you can easily roll back if anything icky happens. Mind you that write perfomance when using the snapshot feature goes down: instead of one write operation, every write becomes a read/write/write operation slowing things down. And this happens for every active snapshot, so you really can't have too much active snapshots
Then again, if it's just for checkpointing (create snapshot), installing experimental stuff and then committing (delete snapshot) or rollback (restore from snapshot, delete snapshot), it should do the trick wonderfully.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
I love, love, love VMWare for this. It's ctrl-Z for sysadmins.
Lately I wanted to experiment with the various kernel-level security packages like LOMAC, LIDS, and SELinux. It was great to be able to build a default linux install on a virtual disk and then copy it three or four times to install the weird security stuff.
It's even better for non-Unix OSes. A friend wanted help installing his Java web app on NT. I built a variety of virtual machines for testing, all using the VMWare "Undoable disk" choice. So when some weird registry key got screwed up by an Oracle installer, I just picked "Undo" and tried again!
If you have to use crappy OS or packages that are inclined to break things and put crap everywhere, VMWare is a delight!
(Yep, I'm just a happy customer.)
AFS will do something like that, although not to the extent that I hear NetApp Filers can. Off the top of my head, there are two ways to do this with AFS. Both these methods require superuser access to your AFS cell, unless backups or replication releases are being done automatically.
/afs/mycell/some/path
..
/afs/.mycell/some/volume
/afs/mycell/some/volume/my/file my/file
/afs/cellname and read-write on /afs/.cellname.
(CodaFS should be able to do this too. I haven't played with CodaFS enough to know if it offers any other way to accomplish checkpointing.)
Method 1: backup volumes
$ cd
$ kinit me/admin
Password for me/admin@MYCELL:
$ aklog
$ vos backup some.path.avol
$ kinit me
Password for me@MYCELL:
$ aklog
$ cd avol
do stuff with the filesystem...
Oops! I need files that I modified or deleted!
$ cd
$ fs mkm avol.backup some.path.avol.backup
$ cp avol.backup/little-lost-file avol/
$ fs rmm avol.backup
Many sites run 'vos backupsys' (generally before 'vos dump'ing volumes) every night to automatically back up all their volumes, and leave users' backup home volumes mounted under their home volumes, to provide easy access to yesterday's files without an administrator's help.
Method 2: for replicated volumes
$ cd
do stuff - uh-oh, I need a file back that I changed!
$ cp
ok, finished with the changes. Commit them!
$ kinit me/admin
Password for me/admin@MYCELL:
$ aklog
$ vos release some.volume
Released volume some.volume successfully
$ kinit me
Password for me@MYCELL:
$ aklog
Volume (for volume, read filesystem) backups work by saving the state of a volume at the time the backup command was issued. When changes are made to the volume, the original state is copied to the backup volume. The backup volume only takes as much space as the changes made since the last backup. Replication works by making read-only copies of a volume in one or more locations, as specified by 'vos addsite' commands. The copies are only updated when changes are 'released' from the read-write copy to the read-only copies. By convention, cell root volumes are mounted read-only on
I think that newer versions of Solaris will do checkpointing on UFS. I haven't adminned Solaris since 2.3 (the slooow SS20 with 2.8 under my bed dosen't count until I play with it some more), so I'm not familiar with the details.
In fact, we've had that since we first shipped our machines. There's a paper on our Web site that discuss how this works, File System Design for an NFS File Server Appliance.
However, although snapshot directories let you dredge up copies of files from snapshots in case you (or a program) screws up and trashes them, that's not a convenient way to roll back the state of the entire file system.
We did implement that later (atop the same mechanism); see SnapMirror and SnapRestore: Advances in Snapshot Technology - SnapRestore(TM)(R)(LSMFT) is the "roll back an entire file system to a snapshot" feature. (At times, all this SnapStuff makes me want to SnapTheNeckOfMarketing, but so it goes....) That paper doesn't discuss technical details to the extent that the other paper does, but it should be possible from the earlier paper to figure out at least some of how you'd do it.