What is Your Backup Policy?

higuita asks: "A few days ago, I was asked to check our backups policy, how they are being applied and to try to make it safer and more useful. Being new to the company, I started to check what is being done right now and found several problems. Since I don't have much experience with enterprise backups, what are the most used backup policies, software and global ideas about this issue? We have less than 1000 workstations (Windows and Macs), about 20 Oracle and Exchange servers (split between Windows, Solaris, and Linux), and it all needs to be backed up. Right now, we use the HP data protector with several tapes, where most things have a weekly full backup and daily incremental backups, and that most full backups are archived permanently in a safe we have for this purpose. We also have off-site storage for backups, as well. What practices and policies do Slashdot users implement for backups they perform at their office (home backups practices I am not interested in)?" "I've investigated Veritas NetBackup and other solutions, and I'm also curious if Amanda could be better or at approximate the features offered by HP Data Protector. What backup software have you used that you found enjoyable with the least bit of hassle?

I've thought about using Dirvish to backup the user's homes to a cheap server with several HDs, and only backup to tapes once every 15 days or even once a month. They will lose their Windows permissions, but I don't think that matters much, since this is just for safekeeping the users' work. I thought about making full backups of the servers every 15 days with daily incremental backups. This way I will free up tape drives' time and gain more flexibility with the backup schedule.

I would love it if users worked off of file servers, but right now this just isn't possible. It's a planned addition that we still don't have the time to make."

Comments

Use an enterprise commercial solution

[georgewilliamherbert]

For that many systems, use a professional, enterprise grade, commercial solution. The open source stuff doesn't supply the same manageability.

AND FOR GOD'S SAKE, REGULARLY VERIFY THAT YOU CAN READ THE TAPES BACK... More sites have been screwed by backup tapes that weren't readable than any other failure mode. Verifying every tape is best. Second best is every weekly. Random samples, but covering every single drive's tape output at least once a month, are poor third place.

The two obvious software suggestions are Veritas/Symantec NetBackup and Legato Networker.

Weekly fulls and daily incrementals are good. Your offsite schedule should be checked to ensure that you have a relatively recent restore point both onsite (in case of data loss) and offsite (in case of building loss).

In terms of offsites, having a prepared plan for where and how to restore (Disaster Recovery and Business Continuity) is also important. But those all start with "Go get the tapes...".

Focus on the systems.

[khasim]

This will take a LOT of research on your part.

You'll need to identify each application that is being used, where its data is being stored and what type of "backup" is needed for it.

Don't forget to include "backups" of the system software. There's nothing more annoying than having to rebuild a system, and you have a backup of the data, but you cannot find the install CD.

Older *nix systems were far easier than the "modern" PC-based servers. I could backup my old Sequent box to a bootable tape. If anything went wrong, I could boot the tape and re-write the system. This is somewhat supported now on some of the PC-based servers.

Anyway, back to the "backups". Once you have the systems identified, then you'll need to look at what scenarios you'll need to plan for.

#1. Server crash.
The data on the disk is destroyed. The OS is destroyed. But the hardware is okay.

#2. The building burns down.
All of your servers are now smoking heaps of plastic. So's your desk. And all the CD's you had.

#3. 5 years from now someone wants a critical policy that was deleted 3 years ago.

I spend most of my time kicking co-workers to get them to NOT just dump data any where that has free space and to NOT just throw up a new web server without telling me.

Paper

[NetDanzr]

My backup copy is paper. Granted, it gets a little awkward when I move, as I currently have six large file boxes of that stuff, but I know that as long as I keep it reasonably safe from humidity/mice it'll outlive all my computer media and file format changes.

At work we do the same, only to a larger extent. We've got an on-site and off-site storage, and each piece of information is printed in two copies to be stored at each. All that in addition to your usual Veritas tape and CD-RW backups, which we do for convenience of restoring lost data, but which we don't trust enough to eliminate paper copies.

Don't just look in the rear-view mirrors...

[tverbeek]

...actually turn your upper body around, so you can look in the direction you're driving.

Think of the children!

Re:Open Source can be enterprise grade

[georgewilliamherbert]

I use plenty of stuff for which I have the source code. Going back to the 4.2mumble BSDs, through SunOS, Linux, Solaris, the various x86 BSDs, and plenty of applications (this is Mozilla I'm /.ing with, and before that a long line of other open source browsers). I have no problem with installing large Linux farms, using Apache for an enterprise web deployment, using MySQL for moderate sized databases (or PostgreSQL, though I haven't deployed it personally).

Tape backup... NBU wins. Legato's a close second. Sorry, charlie. Open source as a category does not suck. The open source backup stuff doesn't suck, for small to medium sized sites. It's not enterprise class, though, and most of the trick to succeeding in IT is knowing when the tools you use aren't applicable anymore and how to figure out what are.

NBU can't RAIT, but it can stream across multiple tapes, and can write duplicate tapes if you want redundancy. And you can extract the files off tape with tar if you have to.

Amanda certainly doesn't suck, but it's not NBU.

Re:just what ever you do make sure offsite IS OFFS

[a9db0]

i would suggest minimum different zip codes different time zones would be best

Sounds funny but very true. Backups across town aren't terriby useful if across town is flat too. Sound farfetched? Ask a sysadmin in Miami how far off he ships his backups. If he was there when Andrew visited, I'll bet they're in New Mexico.

This may seem a tad offtopic, but it is relevant:

You have to think through both distance from and access to your backups as a part of disaster recovery planning. Backup isn't just recovering the CEO's email, though that is a (hopefully) far more frequent occurance than recovering from a hurricane/fire/mudslide/blizzard. Easy access to the backup media is important for daily operations. Recovery from disaster is quite a bit more complex. Your backup solution needs to be able to cover the full spectrum - from yestarday's lost spreadsheet to the area flattened by mother nature.

Personally, I keep two backups - one here locally, one 1000 miles away in another state. Backup to CD here, online rsync in NC.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - Variously attributed, frequently to Andrew Tanenbaum