Backing up a Linux (or Other *nix) System

bigsmoke writes "My buddy Halfgaar finally got sick of all the helpful users on forums and mailing lists who keep suggesting backup methods and strategies to others which simply don't, won't and can't work. According to him, this indicates that most of the backups made by *nix users simply won't help you recover, while you'd think that disaster recovery is the whole point of doing backups. So, now he explains to the world once and for all what's involved in backing up *nix systems."

Comments

Amanda

[Neil Blender]

http://www.amanda.org/

Does the trick for my organization.

Re:Backups

[WuphonsReach]

The problem with suggesting backup solutions is that everyone's tolerance of risk differs. Plus, different backup solutions solve different problems.

For bare metal restore, there's not much that beats a compressed dd copy of the boot sector, the boot partition and the root partition. Assuming that you have a logical partition scheme for the base OS, a bootable CD of some sort and a place to pull the compressed dd images from, you can get a server back up and running in a basic state pretty quickly. You can also get fancier by using a tar snapshot of the root partition instead of a low-level dd image.

Then there are the fancier methods of bare metal restore that use programs like Bacula, Amanda, tar, tape drives.

After that, you get into preservation of OS configuration. For which I prefer to use things like version control systems, incremental hard-link snapshots to another partition and incremental snapshots to a central backup server. I typically snapshot the entire OS, not just configuration files and the hardlinked backups using ssh/rsync keep things manageable.

Finally we get into data, and there's two goals here. Disaster recovery and archival. Archive backups can be less frequent then disaster recovery backups since the goal is to be able to pull a file from 2 years ago. Disaster recovery backup frequency depends more on your tolerance for risk. How many days / hours are you willing to lose if the building burns down (or if someone deletes a file).

You can even mitigate some data loss scenarios by putting versioning and snapshots into place to handle day-to-day accidential mistakes.

Or there's simpler ideas, like having backup operating systems installed on the partition (a bootable root with an old, clean copy) that can be booted in an emergency, run no services other then SSH, but have the tools to let you repair the primary OS volumes. Or going virtual with Xen where your servers are just files on the hard drive of the hypervisor domain and you can dump them to tape.

Re:Dump

[Retardican]

If you are going to talk about dump, you can't leave out why dump is the best. From the FreeBSD Handbook:

17.12.7 Which Backup Program Is Best?

dump(8) Period. Elizabeth D. Zwicky torture tested all the backup programs discussed here. The clear choice for preserving all your data and all the peculiarities of UNIX file systems is dump. Elizabeth created file systems containing a large variety of unusual conditions (and some not so unusual ones) and tested each program by doing a backup and restore of those file systems. The peculiarities included: files with holes, files with holes and a block of nulls, files with funny characters in their names, unreadable and unwritable files, devices, files that change size during the backup, files that are created/deleted during the backup and more. She presented the results at LISA V in Oct. 1991. See torture-testing Backup and Archive Programs.

I find dump to be the best backup tool for unix systems. One disadvantage is that it deals with whole file systems, which means things have to be partitioned intelligently before hand. I think that's actually a Good Thing (TM).

Re:Dump

[arivanov]

I find dump to be the best backup tool for unix systems.

First, looking at this statement it seems that you have never had to run backups in a sufficiently diverse environment. Dump "proper" has a well known problem - it supports only a limited list of filesystems. It originally supported UFS and was ported to support EXT?FS. It does not support JFS, XFS, ReiserFS, UDF and so on (last time I looked each used to have its own different dump-like utility). In the past I have also ran into some entertaining problems with it when dealing with posix ACLs (and other bells-n-whistles) on ext3fs. IMHO, it is also not very good at producing a viable back up of heavily used filesystems.

Second, planning dumps is not a rocket science any more. Nowdays, dumps can be planned in advance in an intelligent manner without user intervention. This is trivial. Dump is one of the supported backup mechanisms in Amanda and it works reasonably well for cases where it fits the bill. Amanda will schedule dumps at the correct levels without user attendance (once configured). If you are backing to disk or tape library you can leave it completely unattended. If you are backing to other media you will need only to change cartridges once it is set-up. Personally, I prefer to use the tar mechanism in Amanda. While less effective it supports more filesystems and is better behaved in a large environment (my backup runs at work are in the many-TB range and they have been working fine for 5+ years now).

Now back to the overall topic, the original ASK Slashdot is a classic example of "Ask Backup Question" on slashdot. Vague question with loads of answers which I would rather not qualify. As usually what is missing is what are you protecting against. When planning a backup strategy it is important to decide what are you protecting against: cockup, minor disaster, major disaster or compliance.

• Cockup - user deleted a file. It must be retrieved fast and there is no real problem if the backups go south once in a while. Backup to disk is possibly the best solution here. Backup to tape does not do the job. It may take up to 6 hours to get a set of files of a large tape. By the end you will have users taking matters in their own hands.
• Minor disaster - server has died taking fs-es with it. Taking a few hours to recover it will not get you killed in most SMBs and home offices. Backup to disk on another machine is possibly the best solution here. In most cases this can be combined with the "cockup" recovery backup.
• Major disaster - flood, fire, four horsemen and the like. For this you need offsite backup or a highly rated fire safe and backup to suitable removable media. Tape and high speed disk-like cartridges (Iomega REV) are possibly the best solution for putting in a safe. This cannot be combined with the "cockup/minor disaster" backups because the requirements contradict. You cannot optimise for speed and reliability/security of storage at the same time. Tapes are slow, network backup to remote sites is even slower.
• Compliance - that is definitely not an Ask Slashdot topic.

As far as with what to backup on unix IMO the answer is amanda, amanda or amanda:

• It plugs into supported and well known OS utilities so if worst comes to worst you can extract the dump/tar from tape and use dump or tar to process it by hand. Also, if you change something on the underlying OS the backups no longer stop working. For example while ago, I had that problem with Veritas which kept going south on anything but old stock RedHat kernels (wihtout updates). So at one point I said enough is enough, moved all of the Unix systems to amanda and never looked back since (that was 5+ years ago)
• It is fairly reliable and network backup is well supported (including firewall support on linux).
• It is not easy to tune (unix is userfriendly...), but can be tuned to do backup jobs where many high end commercial backup programs fail.
• It supports tape backup (including libraries), disk backup and various weird media (like REV)
• It works (TM).