Planet Labs' approach to orbital debris mitigation is discussed here:
https://www.planet.com/pulse/k...
tl;dr the satellites re-enter and burn up very quickly, most within a year or so.
The intent is not to substitute for tapes, but to supplement them. This technique makes the most common types of restore operations very easy (so that users can do it without admin assistance), and means you might not have to make backup tapes as often. When you do make backup tapes, you can pull them from a weekly snapshot, so that you're guaranteed the filesystem doesn't change while the tape is being written!
I wonder how he discovered this? I can't find it in the man page.
Rsync source code, then a lot of testing!:)
Mike
ps: You're right, if there is any change in the file, the original is unlinked first, then the new one is written over top of it. So it does work as advertised! Thanks for your help answering questions btw.
You can exclude any part of the filesystem from the backups, or particular types of files, or files that match a particular pattern; see the "exclude" section in the rsync man page.
I'm not sure I agree that applications should handle their own backups! Don't forget that applications are run as their owners, so if they are broken or hacked, they can destroy the backups too. Far better, I think, to have the backups removed where user-level processes can't touch them. And probably a lot simpler too!
These are not snapshots in the sense of LVM or NetApp; they do not freeze the whole filesystem at a particular point in time between atomic transactions. This technique is a hack for something like a small-office file server. It helps deal with accidental deletions or overwrites, which seem to account for the majority of restore jobs. Think of it as an easier and more intuitive replacement for tar-to-tape. If you're running a database where every transaction counts, you'll need to spend the money and buy a more reliable system!
As several have pointed out, LFS can freeze a volume, though I haven't played with it. Note that normally the reason we're worried about data changing out from under us is that with a conventional backup strategy (such as dump), the filesystem can be rendered inconsistent by such changes. Here, the worst that happens is a few files are messed up--you don't lose the whole backup. So maybe it isn't as important?
If you have a serious application and can't afford to lose a single transaction, you should consider something more solid (and expensive!). This is just a hack for office-type use.:)
Enjoy!
Mike
Re:rdiff-backup is easier and more efficient
on
Linux Backups Made Easy
·
· Score: 2, Informative
Thanks for mentioning this!
Rdiff-backup is an excellent utility, and Ben Escoto (its author) and I link to each other. You must realize, though, that the purposes are different. Rdiff-backup is more space efficient for things like text, email, and so on. My rotating snapshot trick is less space-efficient, but much simpler for the average user to understand ("just go into your snapshot directory and copy the old file back into reality"). It works on all kinds of files, and barely touches the CPU (since it isn't doing diffs). I would use rdiff-backup for administrative backups of email, code, and that sort of thing, where text is involved and user restore is not an issue.
Thanks for the suggestion! This should work fine. My only worry would be that if someone compromises the original machine, (s)he would presumably have access to whatever information it uses to authenticate itself to the backup server. If you take this approach, it would seem to make sense to lock down sshd on the backup server to only one command (namely, the one needed to initiate a snapshot). Even then, though, the attacker could wipe the original machine and force backups until the original ones are pushed off the end of the queue! More safeguards, such as a time-delay on the server, might be necessary.
I like the idea of having the backup server ssh into the client, but then the client has to
accept inbound-ssh from root, which has security implications as well.
Since MacOS X has a BSD core, this trick should work there too. All it needs are hard links (present in almost all UNIX-like OS's), rsync, and either GNU cp or cpio. Please let me know if you get it working!
Actually, I didn't post this; the slashdotting came as a complete surprise. I would have cleaned up the page if I knew all this traffic was on its way! My roommate is pretty annoyed that he can't play online games though.:)
Interestingly enough, the server has plenty of extra capacity, it's the cable modem connection that seems to have been saturated.
Thanks in advance for being courteous and using a mirror or the google cache!
I use tar to maintain critical daily backups [...] does this essentially do the same thing?
Not exactly. This is not a substitute for tar; it can work in conjunction with tar to reduce your backup work and make simple, day-to-day restores easier. Check out the article and let me know what you think!
I was originally using the --backup-dir trick, and you're right, it allows you to back up the same data. The advantage to doing it as described in the article is that you get what appear to be full backups at each increment. This makes it simpler for your users, who can now think of the backup directories as full backups.
Not to be rude, but I don't think you read the article. This is a very different trick. I also use the technique to back up a Win2K machine, via Samba.
This slashdotting comes as a bit of a surprise; many readers have sent me improved scripts that I haven't quite gotten around to posting yet. I'll try to put them up later this weekend when the slashdotting dies down.
The site was never down; it's just that my roommate, a windows user, noticed the connection was slow and reset the cable modem. He's quite upset about being unable to play Warcraft III.:)
I've never had a slashdot nick before, so I just created this one today. I'll try to go through some of the comments and provide useful feedback.
Slashdot Mirror
Planet Labs' approach to orbital debris mitigation is discussed here: https://www.planet.com/pulse/k... tl;dr the satellites re-enter and burn up very quickly, most within a year or so.
The intent is not to substitute for tapes, but to supplement them. This technique makes the most common types of restore operations very easy (so that users can do it without admin assistance), and means you might not have to make backup tapes as often. When you do make backup tapes, you can pull them from a weekly snapshot, so that you're guaranteed the filesystem doesn't change while the tape is being written!
Mike
I use this very technique to back up a Win2K machine. Mount the relevant shares with Samba, and have my linux box keep the snapshots.
Mike
Rsync source code, then a lot of testing! :)
Mike
ps: You're right, if there is any change in the file, the original is unlinked first, then the new one is written over top of it. So it does work as advertised! Thanks for your help answering questions btw.
You can exclude any part of the filesystem from the backups, or particular types of files, or files that match a particular pattern; see the "exclude" section in the rsync man page.
I'm not sure I agree that applications should handle their own backups! Don't forget that applications are run as their owners, so if they are broken or hacked, they can destroy the backups too. Far better, I think, to have the backups removed where user-level processes can't touch them. And probably a lot simpler too!
Mike
Mike
As several have pointed out, LFS can freeze a volume, though I haven't played with it. Note that normally the reason we're worried about data changing out from under us is that with a conventional backup strategy (such as dump), the filesystem can be rendered inconsistent by such changes. Here, the worst that happens is a few files are messed up--you don't lose the whole backup. So maybe it isn't as important?
If you have a serious application and can't afford to lose a single transaction, you should consider something more solid (and expensive!). This is just a hack for office-type use. :)
Enjoy!
Mike
Rdiff-backup is an excellent utility, and Ben Escoto (its author) and I link to each other. You must realize, though, that the purposes are different. Rdiff-backup is more space efficient for things like text, email, and so on. My rotating snapshot trick is less space-efficient, but much simpler for the average user to understand ("just go into your snapshot directory and copy the old file back into reality"). It works on all kinds of files, and barely touches the CPU (since it isn't doing diffs). I would use rdiff-backup for administrative backups of email, code, and that sort of thing, where text is involved and user restore is not an issue.
Different tools for different jobs!
Mike
Thanks for the suggestion! This should work fine. My only worry would be that if someone compromises the original machine, (s)he would presumably have access to whatever information it uses to authenticate itself to the backup server. If you take this approach, it would seem to make sense to lock down sshd on the backup server to only one command (namely, the one needed to initiate a snapshot). Even then, though, the attacker could wipe the original machine and force backups until the original ones are pushed off the end of the queue! More safeguards, such as a time-delay on the server, might be necessary.
I like the idea of having the backup server ssh into the client, but then the client has to accept inbound-ssh from root, which has security implications as well.
Mike
Mike
Mike
Mike
Interestingly enough, the server has plenty of extra capacity, it's the cable modem connection that seems to have been saturated.
Thanks in advance for being courteous and using a mirror or the google cache!
Mike
I use tar to maintain critical daily backups [...] does this essentially do the same thing?
Not exactly. This is not a substitute for tar; it can work in conjunction with tar to reduce your backup work and make simple, day-to-day restores easier. Check out the article and let me know what you think!
Mike
I was originally using the --backup-dir trick, and you're right, it allows you to back up the same data. The advantage to doing it as described in the article is that you get what appear to be full backups at each increment. This makes it simpler for your users, who can now think of the backup directories as full backups.
Hope that helps--
Mike
Mike
Thanks for mirroring!
Mike
The site was never down; it's just that my roommate, a windows user, noticed the connection was slow and reset the cable modem. He's quite upset about being unable to play Warcraft III. :)
I've never had a slashdot nick before, so I just created this one today. I'll try to go through some of the comments and provide useful feedback.
Thanks for your interest everyone!
Mike