Slashdot Mirror
Backing Up is Hard to Do?
Joe Barr writes "NewsForge is running a story this morning on a personal hardware/software backup solution for your Linux desktop (NewsForge is owned by Slashdot's parent OSTG). The solution doesn't require a SCSI controller, or tape drive, or the ability to grok a scripting language or archiving tool to work, either. It's based on point-and-click free software. Plus it includes a dead-parrot joke by Linus Torvalds."
Backing up isn't painful, restoring is.
I use rsnapshot to automate my backups to another host. Works like a dream, providing multiple virtual point in time copies (just like similar functionality from Network Appliance, etc.).
Dead Parrot Sketch
The cast:
MR. PRALINE John Cleese
SHOP OWNER Michael Palin
The sketch:
A customer enters a pet shop.
Mr. Praline: 'Ello, I wish to register a complaint.
(The owner does not respond.)
Mr. Praline: 'Ello, Miss?
Owner: What do you mean "miss"?
Mr. Praline: I'm sorry, I have a cold. I wish to make a complaint!
Owner: We're closin' for lunch.
Mr. Praline: Never mind that, my lad. I wish to complain about this parrot what I purchased not half an hour ago from this very boutique.
Owner: Oh yes, the, uh, the Norwegian Blue...What's,uh...What's wrong with it?
Mr. Praline: I'll tell you what's wrong with it, my lad. 'E's dead, that's what's wrong with it!
Owner: No, no, 'e's uh,...he's resting.
Mr. Praline: Look, matey, I know a dead parrot when I see one, and I'm looking at one right now.
Owner: No no he's not dead, he's, he's restin'! Remarkable bird, the Norwegian Blue, idn'it, ay? Beautiful plumage!
Mr. Praline: The plumage don't enter into it. It's stone dead.
Owner: Nononono, no, no! 'E's resting!
Mr. Praline: All right then, if he's restin', I'll wake him up! (shouting at the cage) 'Ello, Mister Polly Parrot! I've got a lovely fresh cuttle fish for you if you
show...
(owner hits the cage)
Owner: There, he moved!
Mr. Praline: No, he didn't, that was you hitting the cage!
Owner: I never!!
Mr. Praline: Yes, you did!
Owner: I never, never did anything...
Mr. Praline: (yelling and hitting the cage repeatedly) 'ELLO POLLY!!!!! Testing! Testing! Testing! Testing! This is your nine o'clock alarm call!
(Takes parrot out of the cage and thumps its head on the counter. Throws it up in the air and watches it plummet to the floor.)
Mr. Praline: Now that's what I call a dead parrot.
Owner: No, no.....No, 'e's stunned!
Mr. Praline: STUNNED?!?
Owner: Yeah! You stunned him, just as he was wakin' up! Norwegian Blues stun easily, major.
Mr. Praline: Um...now look...now look, mate, I've definitely 'ad enough of this. That parrot is definitely deceased, and when I purchased it not 'alf an hour
ago, you assured me that its total lack of movement was due to it bein' tired and shagged out following a prolonged squawk.
Owner: Well, he's...he's, ah...probably pining for the fjords.
Mr. Praline: PININ' for the FJORDS?!?!?!? What kind of talk is that?, look, why did he fall flat on his back the moment I got 'im home?
Owner: The Norwegian Blue prefers keepin' on it's back! Remarkable bird, id'nit, squire? Lovely plumage!
Mr. Praline: Look, I took the liberty of examining that parrot when I got it home, and I discovered the only reason that it had been sitting on its perch in the
first place was that it had been NAILED there.
(pause)
Owner: Well, o'course it was nailed there! If I hadn't nailed that bird down, it would have nuzzled up to those bars, bent 'em apart with its beak, and
VOOM! Feeweeweewee!
Mr. Praline: "VOOM"?!? Mate, this bird wouldn't "voom" if you put four million volts through it! 'E's bleedin' demised!
Owner: No no! 'E's pining!
Mr. Praline: 'E's not pinin'! 'E's passed on! This parrot is no more! He has ceased to be! 'E's expired and gone to meet 'is maker! 'E's a stiff! Bereft of life, 'e
rests in peace! If you hadn't nailed 'im to the perch 'e'd be pushing up the daisies! 'Is metabolic processes are now 'istory! 'E's off the twig! 'E's kicked the
bucket, 'e's shuffled off 'is mortal coil, run down the curtain and joined the bleedin' choir invisibile!! THIS IS AN EX-PARROT!!
(pause)
The best way to create differential backups under Unix is with hardlinked snapshots. Easy Automated Snapshot-Style Backups with Rsync has a good explanation of how to do this. The best part is that restoring is as simple as copying a file. Each snapshot is a folder hierarchy on disk, and you can browse through any snapshot and find files you want.
/tmp using mkzftree if I need to restore something.
One small improvement over rsync (IMO) is to use mkzftree from the zisofs-tools package. It's designed to create compressed ISO filesystems which will be transparently uncompressed when mounted under Linux (and other supporting operating systems; it's a documented ISO extension). mkzftree supports an option for creating hardlinked forest (like cp -al and rsync), with the advantage that the files are compressed, thus saving space. ISO isn't quite as flexible as ext2 for things like hardlinks, so what I do is have DVD-sized disk images formatted as ext2 to store the snapshots. I burn the disk images directly to DVD; each one can hold ten or twenty compressed snapshots (of my data anyway). The disadvantage is that I can't read the files directly (because they're compressed, and the transparent decompression only works with ISO) but it's easy to decompress a file or folder to
It shouldn't be hard to make the transparent decompression code work with other filesystems than ISO, as long as they're mounted read-only. The files are just gzipped with a header block indicating they are compressed.
1. Reach over and plug in USB 120 gig drive.
2. Become root, and go to /root.
3. Type "./backup.sh".
That is a script that goes to all the directories I care about (/root, /etc, /srv/www, /usr/local/share, and my home directory), and basically does this for each drive.
cd $DIR rsync -avz --progress --delete . $MNT/$DIR
where $MNT is where the USB drive mounts.
4. Unmount the drive and unplug it.
This is quick (a few minutes) and easy, and since rsync reads the files from the last backup to figure out what needs to be copied, it should catch it if I develop a bad sector on the USB drive.
I left it out in the above, but the backup script also, before doing the rsyncs, lists my crontab into a file, so that gets backed up.
Someone taught me a cool trick to backup up all files with the highest possible compression ratio and speed: mv * /dev/null
A hard drive crash over the holidays left me scrambling to get back to a productive desktop as quickly as possible. Luckily, I had my /home partition on a separate drive, so I didn't lose precious email, stories, research, and pictures. But it did get me thinking about my lack of preparedness. Where was the back-up system I've talked about for years, but never acquired? This is the tale of how I rectified that glaring omission, and built myself a personal back-up system using inexpensive parts and free software.
/home directory on hdd. Backing up directly to CD would be too slow and too cumbersome for me, so the first thing I needed was some new hardware.
The hardware
My desktop machine includes three IDE drives and an ATAPI CD-ROM drive. I have Debian installed on hda, SUSE on hdc, and my
In the past I've researched tape drives and found that for a decent drive, I would also have to add a SCSI controller. Those two items can be pretty pricey. I opted for a less expensive configuration.
I decided to go with a removable IDE drive, connected via USB. I bought a 3.5-inch hard disk enclosure with USB 2.0 connectivity on eBay. It cost roughly $45, including shipping. With three drives to backup, I needed a large-capacity IDE drive to hold all the data. It turns out I already had one, just waiting for me to use. I raided the stash of goodies I've been hoarding to build a killer MythTV box and found a 250GB Hitachi DeskStar -- just what the doctor ordered. I got it on sale at Fry's Electronics a couple of months ago for $189.
I have the mechanical skills of a three-toed sloth, but I still managed to cobble together the drive and the enclosure, neither of which came with directions. Four screws hold the faceplate on the enclosure, and four more hold the drive in place inside. Even I was able to puzzle it out.
The most difficult part was the stiffness of the IDE cable running between the faceplate and the drive. In hindsight, I recommend connecting the power and data cables from the faceplate to the drive before screwing the drive in place inside the enclosure. I also recommend not forgetting to slide the top of the enclosure back in place before reattaching the faceplate.
I connected the USB cable to the enclosure and the PC and powered on. Using the SUSE partitioning tool, I created an ext3 filesystem and formatted it on the Hitachi drive, using the default maximum start and stop cylinders. That worked, but there was a problem. My great big 250GB drive yielded only 32GB.
One of my OSTG cohorts asked if had clipped the drive for 32GB max, but I had done no such thing. All I did was check to see how the drive was strapped out of the box. It was set to Cable Select, which was fine with me, so I left it like that. His question worried me, though, because I had never heard of a 32GB clip thingie before.
I called Hitachi support to find out what was up with that. Their tech support answered quickly. When I explained what was going on, he agreed that it sounded like it was clipped to limit its capacity. This functionality allows these big honkers to be used on old systems which simply cannot see that much space. Without it, the drive would be completely unusable on those machines.
I asked why in the world they would ship 250GB drives configured for a max of 32GB by default, and he denied that they had. He asked where I got the drive, then suggested that Fry's had "clipped" it for some reason. There are jumper settings to limit the capacity, but my drive had not been jumpered that way. Perhaps Fry's sold me a returned drive that a customer had "clipped", then returned the jumpers to their original position. We'll never know.
The tech told me how it should be jumpered for Cable Select without reducing capacity. I opened the USB enclosure, pulled out the drive, and found it was already jumpered as he described. Undaunted, I pressed on.
On the Hitachi support page for the drive, I found a downloadable tool wh
He plugs in a USB drive, runs KDar to fill it with stuff.
/bin /usr, etc, which I then burn onto a couple of DVD9-Rs. I can run this to recreate my system.
/home.
Now, when his system borks, how does he restore? Or did he think that far ahead?
I skimmed the article, and nothing about restoring. Your backup is useless if you can't restore it.
Does he have to install and configure linux, X, and KDE just to be able to access KDar?
Forget all this jibberjabber, and emerge or apt-get or type whatever command you use to get Mondo/Mindi. Just perfect for home boxes, and most other use.
Burn yourself a bootable CD that can recreate your box, just like Norton Ghost for Linux. I have it write out the iso files and boot disk for
I run a seperate job to backup
Whats important, is to seperate system from user data when it comes to backups. This also forms my "archiving" system, since old "/home" backups stick around, so if I want to take a look at the version of foo.c I was writing 6 months ago, it's easy enough to find.
As much as I love Mondo/Mindi, it's not the be-all and end-all. AMANDA is a better choice for a corporate (more elaborate) environment. It's a PITA and not worth getting involved with for a simple user box.
I don't need no instructions to know how to rock!!!!
Just do an NFS or SMB mount:
/mnt/backup /mnt/drive /mnt/backup .
mount -t smbfs -o username:password \\10.0.1.111\backup
cd
tar -cvjf
(If I recall the commands correctly.) I use this all the time to make quick snapshots of my Gentoo installation before emerging some bleeding edge package.
I don't know what kind of crack I was on, but I suspect it was decaf.
i don;t know about everyone else, but isn't that one of those things that should have come up pretty early in BETA testing?
"Great backup program.. too bad it can't restore"
Restore is also straightforward - it can be done in place, or by downloading a zip/tar file.
The Raven
Is that some kind of sense that allows you to pick out other KDE users in a crowd?
Copying data to a single IDE drive and calling it "backup" is just pathetic.
He should read the Tao of Backup http://www.taobackup.com/ and be enlightend.
With just one or two boot floppies, I can back up and restore my Linux drives to either: other internal IDE drives, other parititons on the same drive, external USB1 and USB2 drives, burnable CDs, or burnable DVDs.
Heck, it is so fast and reliable, I've been known to backup the drive just before even *trying out* new software or options, and if I don't like it, I just Ghost it back to how it was.
Now, I know it isn't free, or even Linux based, but it is hard to argue with cheap, reliable, and fast backup procedures that just work all the time...
I've read the whole article. My! You'd better be a geek to have to cope with all the little worries..
Getting cheap AND working hardware on E-Bay. My mom will not do it for the sake of her computer.
32GB limitation by jumpers. Not obvious for an end-user.
Booting up *nixes from various drives in order to access the limited drive, then fiddle with partitions. I still don't dare touching my configs for more than OS at a time. Let alone various OSes on various drives.
Compiling KDart?! Compiling what? What do I have to do? "Comp..??" You have to admit, it's not for the dummy kind.
Definitely not "Backup made easy" but "Made not so expensive" since the price tag still reaches 300$ (drive + box from e-bay + screws + shots of valium to calm you down when your machine refuses to boot after all the offence you just did to it).
I bought Linux Hacks. This, Webmin and a remote machine accessible using Samba or sftp does the daily backup just fine.
Actually we use rsync for incremental backups and it works quite well. Its a simple modification or scripting of rsync commands and can be all scripted away pretty easily..
b-loo
In case you have a seperate computer or a seperate drive one can use rsync to relativly easily create backups, its just a few lines of Shell:
/your_directory_to_backup \
/your_directory_to_backup to user@other_host:/backup/current/ and in addition to that keeps all the changes you did to that directory in a seperate 'dated' directory like /backup/2005-01-15, so you can also recover files that you deleted some days ago. Some other posters seem to have missed the '--backup' option, which is why I repost the rsync trick.
rsync -e ssh \
--delete \
--relative \
--archive \
--verbose \
--compress \
--recursive \
--exclude-from list_of_files_you_don't_wanna_backup \
--backup \
--backup-dir=/backup/`date -I` \
user@other_host:/backup/current/
This command mirrors everything in
Disadvantage is that you can't easily restore an exact old state of the directory which you backuped, however you can retrieve all the files very easily.
There are also floating some shell scripts around which add to the above rsync line some vodoo to hardlink the different dated directories, so that you have a normal browsable copy of each and every day while only wasting the space for the changes.
And there are also tools which optimize this whole thing a bit more, by compressing the changes you did to files, like http://www.nongnu.org/rdiff-backup/
However overall I found the plain rsync solution the most pratical, since it doesn't require special tools to access the repo and 'just works' the way I need it.
The best solution for your backup problems is to learn to prioritize. No, you don't need to save your pr0n collection. No, you don't need to save every .jpg anyone's ever sent to you. No, you don't need to save every bad joke e-mail you've ever received. No, you don't need to save... you don't need to save... don't save... don't need.
When I was young (early 20s) I saved everything. Then I had an HD crash. I started over and, several years later, my new HD inherited an unrecoverable problem. I started over and then went through a run of about 4 Western Digital Caviar drives which each lasted about one year.
So... what do I save now? As little as possible. Believe me, when you're going through your tree and say to yourself,"well... I might want to save that just in case..." do yourself a favor and hit DELETE. Anytime the word "I might" or "just in case" comes to mind, hit DELETE.
After you've reprogrammed yourself to have a sane set of backup priorities, you'll find that rsync and tar are more than adequate.
For corporate solutions... well, that's a whole different story. Then you're getting paid to feed someone else's legal paranoia. Just buy more drives.
fast as fast can be. you'll never catch me.
I make daily differential backups (via AMANDA) to a rotating set of 12 tapes. If I accidentally delete /etc/shadow or some other important file, I have nearly two weeks to discover the problem and restore a previous version from tape. Your idea gives you, oh, until about the time that rsync discovers the missing file and dutifully nukes it from your "backup" drive.
What you're doing is certainly better than nothing, but it's not a backup solution by any definition of the term beyond "keeps zero or one copy of the file somewhere else".
Far, far better would be for your script to use dump or tar to create incremental backup files on your USB drive and to rotate them out on a regular basis.
Dewey, what part of this looks like authorities should be involved?
One flaw in any hard drive backup system: what happens if your system is cracked?
If someone gets into your system, they do an rm -r *, is your backup drive mounted?
What if they're clever and do a mount all, or find your backup.sh first?
I've seen some people take the first and last step of "inserting the USB cable" and "removing the USB cable". Is there any kind of automated system that would ease this, or is it the Hard drive equivelant of "Remove tape, insert new tape".
USB drives also suffer from problems with catastrophic failure, like a fire in your home.
I wonder if there exist any online backup systems that let you do offsite daily differential backups of your system (or critical files) that would let you download or mail you an image of your harddrive (on DVD-R) along with restore software in case anything went wrong. You could charge directly by bandwidth used. Hmm, interesting idea.
You can read from
I have a 20GB iPod, but only about 12GB is used. My $HOME is about 2GB, including a bunch of digital photos, but also a bunch of documents, my email, and other stuff I'd rather not lose.
My solution is simple:
This is a very simple shell script that deletes the backup file already on the iPod, then does a 'tar czf - $HOME' and pipes it into gpg using circular encryption (that is, a passphrase.) The encrypted, compressed tarball (about 1.7GB) is written directly to the iPod. Takes about 20 minutes.
I've used this backup copy to do restores, and it's really as simple as plugging in the iPod, using gpg to descrypt the file, piping that into 'tar xvzf -' to re-create my $HOME. I can move all my stuff back to where it needs to be after that.
(For those who wonder: I always make an encrypted backup file in case my iPod is ever lost or stolen. Sure, the bad guy can probably run something to brute force the passphrase, if that's something he's interested in doing, but it's a tough passphrase. I don't worry about it so much, and it's "only" email and family photos.)
mkcdrec is a really neat program that packs up your whole system and makes a recovery disk. Its something any sysop should take a look at.
See the homepage here.
tar -lcvf /backup/$HOSTNAME-$DATE.tar /
/). The -l switch tells tar to stay on the same FS, as /backup is NFS mounted to a RAID array. Thus, I just backup the local machine, without having to specify which directories to backup, and which to skip.
/backup/whatever.tar
(I'm the type that creates one big ass partition for
Restoration, I do the lazy way:
mkdir test
cd test
tar -xvf
and then I grab the files (the RAID array usually has plenty of space).
Bored? Why not join a decent mess
backup2l does a great job at figuring out which files are new or have been modified for incremental backups. Easy to configure a very lightweight.
Because RAID-1 is an exact mirror, I get a complete, bootable backup copy of my system at the time of the shutdown. Downtime is limited to the few minutes it takes to shut down and swap drives. The lengthy process of mirror rebuilding takes place while the system runs normally. And of course, RAID also protects me against random (single) hard drive failures.
This solves the full image backup problem, leaving only the more frequent partial backups you should also be doing. For this, rsync is your friend. The stuff that changes most often on my system are my IMAP folders, which I periodically (several times per day) rsync to my laptop. Besides backing up my mail server, this gives me copies I can carry around and read when I'm offline.
Tape is obsolete. It's just too slow, expensive, unreliable and small. Hard drives are so cheap, fast and capacious that there's little excuse to not run RAID on any machine that can physically hold more than one hard drive. Unfortunately, this leaves out most laptops.
So, first, we have this "Copy" system that's being called a "Backup" system. PHOOEY.
m 0204c.htm
Next, he admits that the "Backup" system can't restore files or directories. OMG!!
Anybody who adopts this system of backups better be praying to the hard drive gods and be making regular and appropriate homage.
For a REAL backup method that can stand the test of time, try this:
http://www.samag.com/documents/s=7033/sam0204c/sa
It's a system that compresses each file individually, writes them out to a temp directory, creates an iso and then writes the iso to a CD. This way, single bit errors in the compressed archive don't kill the entire thing, just a single file. This becomes more important as your backups begin to age because perfect playback of bits becomes more difficult as the media they are stored on ages.
Any system that creates a data stream of all files and then compresses it is prone to total loss of data beyond any significant error in the playback.
The order is everything.
Stream all files through the compress algorithm... Very Risky.
Compress each file individually and stream it to the archive... Very Safe.
Write everything to cheap read only media... Very smart.
Depending on the data, you can get several Gig's onto a single CD. High quality CD's should be readable for decades.
We should start referring to processes which run in the background by their correct technical name... paenguins.