What is Your Backup Policy?

higuita asks: "A few days ago, I was asked to check our backups policy, how they are being applied and to try to make it safer and more useful. Being new to the company, I started to check what is being done right now and found several problems. Since I don't have much experience with enterprise backups, what are the most used backup policies, software and global ideas about this issue? We have less than 1000 workstations (Windows and Macs), about 20 Oracle and Exchange servers (split between Windows, Solaris, and Linux), and it all needs to be backed up. Right now, we use the HP data protector with several tapes, where most things have a weekly full backup and daily incremental backups, and that most full backups are archived permanently in a safe we have for this purpose. We also have off-site storage for backups, as well. What practices and policies do Slashdot users implement for backups they perform at their office (home backups practices I am not interested in)?" "I've investigated Veritas NetBackup and other solutions, and I'm also curious if Amanda could be better or at approximate the features offered by HP Data Protector. What backup software have you used that you found enjoyable with the least bit of hassle?

I've thought about using Dirvish to backup the user's homes to a cheap server with several HDs, and only backup to tapes once every 15 days or even once a month. They will lose their Windows permissions, but I don't think that matters much, since this is just for safekeeping the users' work. I thought about making full backups of the servers every 15 days with daily incremental backups. This way I will free up tape drives' time and gain more flexibility with the backup schedule.

I would love it if users worked off of file servers, but right now this just isn't possible. It's a planned addition that we still don't have the time to make."

Comments

Use an enterprise commercial solution

[georgewilliamherbert]

For that many systems, use a professional, enterprise grade, commercial solution. The open source stuff doesn't supply the same manageability.

AND FOR GOD'S SAKE, REGULARLY VERIFY THAT YOU CAN READ THE TAPES BACK... More sites have been screwed by backup tapes that weren't readable than any other failure mode. Verifying every tape is best. Second best is every weekly. Random samples, but covering every single drive's tape output at least once a month, are poor third place.

The two obvious software suggestions are Veritas/Symantec NetBackup and Legato Networker.

Weekly fulls and daily incrementals are good. Your offsite schedule should be checked to ensure that you have a relatively recent restore point both onsite (in case of data loss) and offsite (in case of building loss).

In terms of offsites, having a prepared plan for where and how to restore (Disaster Recovery and Business Continuity) is also important. But those all start with "Go get the tapes...".

Re:Use an enterprise commercial solution

[Anonymous Coward]

"Verifying every tape is best. Second best is every weekly. Random samples, but covering every single drive's tape output at least once a month, are poor third place."

Can you explain to me how you accomplish this task? I've run our tape backup system for the past few years and we've used Legato NetWorker and Veritas NetBackup, and believe me, there just isn't time to verify everything. When an LTO2 tape can hold 200-400GB of data (I've seen >700GB of data compress onto these tapes sometimes) and you're backing up say from 6pm till 9am the next morning.... you see my point about time.

We test on a regular basis, but there's no way we can do every tape. The test is basically to ensure that something hasn't gone horribly wrong and mangled everything.

Re:Use an enterprise commercial solution

[Anonymous Coward]

I'm extremely less than impressed with 'enterprise backup solutions', regardless the endless touting they receive in the vendor-supported trade rags and by vendor-supported 'industry analysts'. The 'enterprise backup solutions' I've seen, have been so clunky and hard to use, setting up and maintaining backups is a full time job. When you think about ACTUALLY DOING A DR using these 'products' they often come up short.

Also, sorry, but reinstalling the OS, and then restoring files from tape is NOT acceptable DR (you can thank the proliferation of bill gates OSes in all datacenters for the idea that this could be even close to OK). Datacenters with lots of pc systems have as their saving hope - vmware. Now they can actually restore from bare metal (bare metal in this case being a standard virtual machine). What is needed in this case is a simple solid backup/restore program and a good integrated tape librarian program. But brainwashed IT managers want these overpriced, overhyped, overwrought, 'enterprise backup solutions'. Oh well!

Re:Use an enterprise commercial solution

[georgewilliamherbert]

What are your tape drives doing from 9am until 6pm?

That's not enough time for a full check, but that's enough time for checking half the tapes...

Re:Use an enterprise commercial solution

[georgewilliamherbert]

BMR has been standard for years.

I've seen attempts to build large enterprise backup environments with "simple open" software. They melt down somewhat short of the size that the original questioner is asking about, typically.

I've built environments with NBU and used Legato, at large sites (much larger than the original questioner). They just work. Configuring them initially can be non-trivial if you have no prior experience with them, but once set up right they just work.

Throwing a bunch of open source tech at the wall and seeing if it sticks will kill you here. I've been at places which were big enough to use 40, 60, 180, multi-hundred, 5000 tape changers. They use professional grade stuff. It works. If you can't make it work, don't go to work for large sites. Don't wire your whole five hundred cube building up with daisy-chained 8 port 10/100 switches, and don't use toy backup equipment if you're an enterprise class environment. Data backups aren't a tech game: they're how you survive the statistically likely disk outages and statistically unlikely building fires/floods/earthquakes/tornados/etc. This is important, and half-assed solutions shouldn't apply.

Re:Use an enterprise commercial solution

[wenchmagnet]

I am surprised you did not mention Tivoli Storage Manager. TSM just about rocks for these requirements and is ultra scalable. You can also set it up in such a way that the backups will simultaneously go to multiple tapes for the most critical data and that data can be periodically audited for readability etc.

http://www-306.ibm.com/software/tivoli/sw-atoz/ind exS.html

Re:Use an enterprise commercial solution

[bugpit]

Another product that is big in Europe and gaining acceptance in the US is Atempo Time Navigator. It has broad x-platform support on both the server and client sides, and if you have a substantial OS X install base that you need to backup, it's one of the few products that we've identified that can scale to handle hundreds or thousands of Macs. It can also provide "opportunistic" backups for laptops, which unlike desktops don't fit well into a predefined backup schedule.

- Gregg

Re:Use an enterprise commercial solution

[nbvb]

Hear hear!!

TSM is by far the best backup product I've ever used .... ever.

I just don't worry about getting my data back -- I know it's safe. It's NEVER a concern.

And even if the onsite tape(s) are damaged, TSM is smart enough to call out for the offsite copy so it can rebuild a new onsite copy. Slick. Really, really slick. :)

I wouldn't even -look- at other products if you're a large enterprise.

Re:Use an enterprise commercial solution

[nbvb]

Try TSM. DR is one of its strongest suits!

It's really pretty darned incredible. One command, and your TSM environment is rebuilt. We use the DR capabilities multiple times per year. Works great.

just what ever you do make sure offsite IS OFFSITE

[RobertLTux]

don't make the mistake that one guy did
the office was in the North Tower --- The "offsite backup" was in the South Tower

oops
i would suggest minimum different zip codes different time zones would be best
other than that Grand father > Father >Son GF gets sent offsite

Focus on the systems.

[khasim]

This will take a LOT of research on your part.

You'll need to identify each application that is being used, where its data is being stored and what type of "backup" is needed for it.

Don't forget to include "backups" of the system software. There's nothing more annoying than having to rebuild a system, and you have a backup of the data, but you cannot find the install CD.

Older *nix systems were far easier than the "modern" PC-based servers. I could backup my old Sequent box to a bootable tape. If anything went wrong, I could boot the tape and re-write the system. This is somewhat supported now on some of the PC-based servers.

Anyway, back to the "backups". Once you have the systems identified, then you'll need to look at what scenarios you'll need to plan for.

#1. Server crash.
The data on the disk is destroyed. The OS is destroyed. But the hardware is okay.

#2. The building burns down.
All of your servers are now smoking heaps of plastic. So's your desk. And all the CD's you had.

#3. 5 years from now someone wants a critical policy that was deleted 3 years ago.

I spend most of my time kicking co-workers to get them to NOT just dump data any where that has free space and to NOT just throw up a new web server without telling me.

Re:Focus on the systems.

[StillAnonymous]

"You'll need to identify each application that is being used, where its data is being stored and what type of "backup" is needed for it."

I second this. Nothing's worse than someone telling you "back up this system, full once a week, incrementals every other day, all local drives, blah blah" and then not telling you they've got some database on it (you can't back up a live database by just copying the files.) Of course, when failure hits, guess what needs to be restored and isn't usable?

Re:Focus on the systems.

[kahanamoku]

Option #3 brings in a whole new discussion on Retention Policy.

when you're backing up a few TB of data as we do in the company I work for, and with many cost contraints imposed, you have to look at what is most likely to be requested of the backups.

Options 1 & 2 are both Disaster Recovery scenarios. The only difference being the scale of the disaster.

Option 3 is "an" end-user stupidity scenario, which goes along with the "oh crap, I accidentally hit shift+delete and not shift+end to highlight files"

we have a 6 week incremental "Daily Point-In-Time" retention policy, where the files a user can accidentally delete, or change without making their own backups, can be restored to the user and keep them happy. if they request a file from longer than 6 weeks ago, they are only going to get weekly-level granularity with the restore back (weekly differential backups), and any longer than the 3 month retention policy on the differentials will see their granularity of point-in-time extend to the Monthly Fulls, which are permanent.

As for the disaster recovery based servers, where the data is only managed by the IT team, and controlled under strict change management procedures. this data is backued up with a shorter retention period, as there isn't many cases for these servers to require a point in time restore.

Re:Focus on the systems.

[sumdumass]

Or how about the database the backup software uses. I have seen other peoples solutions go down and after rebuilding the backup machine there wasn't a record of what was on what tape to be restored. I had to re-index each tape and restore from there. Then try and check the files for the newest ones.(incrementals spanning different tapes as well as recycled tapes so at best you have the changes to a few file but not the orignial to restore to.) It took weeks instead of hours or even days to get it cloe enough to use.

It is a good idea to look at whatever system you use and make sure you can backup the information that will let you restore from the tapes to somethign that will be usable when you need it. I cannpt repeat that enough!

Re:Focus on the systems.

[Mateito]

You missed a few:

#4: User deletes a file deemed by somebody important to be critical and you have to get it back.

Its amazing how much money is spent planning for the once-in-a-lifetime Twin-Towers disaster event, and how little is spent on the daily occurance of user-error. Unfortunately "User is an idiot" doesn't wash when its the company's financial records or the birthday party shots of the CEO's kid.

- Don't permit users to save things to their local disks. Ensure all files go onto a share that can be centrally backed-up. Important people (CEO, COO, etc) need to be treated as exceptions and have their Personal PCs, PDAs and even phone memories backed up somehow.

#5: Your CFO is found to be embezzeling money from the company, and you have to show compliance with whichever standards.

This is actually not a backup issue, but an archiving issue, but should be addressed at the same time as backup to ensure you have no holes in your solution.

You've opened a huge can of worms, but one that rightly should be opened. My advice to you is to call in EMC, Sun StorageTek and Symantec and get their presales engineers to do as much legwork as possible before they try to turn it into a chargable engagement. Having all three in there means th
(DONT call HP - Omniback is a dog). This will give you enough information to present to your manager or whoever is appropriate and get some idea of what budget they will give you. That's going to be your single biggest constraint. Backup/DR/BC is something that will easily absorb all the cash that you throw at it.

Just remember - No matter what EMC say, Tape's not dead - not even close - though it is no longer necessarily the best solution for quick restores of recently changed/deleted information.

There are experts in this stuff - trust me I am one - and we get paid a shitload. Trick is, we don't really know that much more than you, we just do it everyday. Exploit vendors' presales engineers. That's what they are there for.

Why do you need to backup the desktops?

[MBCook]

This may just be a wording issue, but it looks like you want to back up the desktops. Is that true?

I can't think of any good reason to do that. All the important data should be on the server. If the user wants to save a picture on the local disk to use as a background or something that's one thing (although I wouldn't allow that myself) but nothing important should be on those disks.

Past that, I don't have the experience to help you. All I can do is reiterate what another poster has already put up. Check the backups. I can't tell you how many stories I've heard about backups that "went fine" until someone needed data. Stories where the tapes were so old they almost shredded themselves in the drives. Stories of "backing up" for at least 6 months onto a cleaning tape (I bet the drive was in good condition though!). Stories of the backup data being garbage because of a faulty cable or something. The backup is worthless if you can't get the data back off it successfully.

Re:Why do you need to backup the desktops?

[Anonymous Coward]

Mod up the parent:

Don't put data on desktops, and don't build your desktops by hand. Use automated tools or system imaging to make rebuilds a non-issue. When a desktop drive fails you replace the drive, rebuild from the system image, and you're done.

Oh, and do back up the server where you data lives.

Re:Why do you need to backup the desktops?

[joe90]

I can't think of any good reason to do that. All the important data should be on the server. If the user wants to save a picture on the local disk to use as a background or something that's one thing (although I wouldn't allow that myself) but nothing important should be on those disks.

Parent is correct - to an extent. There is still probably a requirement to bring a failed desktop up and running quickly if there is a problem that requires a desktop restoration.

If centrally storing data is the way to take care of user data that's all fine and good, but there is still the requirement to restore the desktop environment quickly - and while a central backup solution might be one way to achieve this, it's probably not the best way.

Having a desktop re-imaging solution can enable an end-user (perhaps with a little help from your helpdesk (you do have a helpdesk right?)) to restore a desktop using something like a PXE enabled NIC to boot to either a RIS based imaging system (if you like Microsoft tech) or Linux + scripts to download and install a replacement image. While I'm not sure how well your Mac environment would work with a Linux solution (it won't work with RIS), it could at least take care of your Intel desktops.

As an aside, "enterprise" backup solutions require proper planning, both to design, implement, maintain and operate. Depending on your industry and location, you may be legally required to restore data going back decades, and if not well planned, you may have tapes that cannot be read, either because you've lost the means to restore (the tape drive or software required to restore a particular tape is no longer available) or the tape itself may not survive. Make sure that you have a mechanism to ensure that backed up data is still able to be restored. Some organisations transfer data from their legacy backup solution to their new one prior to each time their then legacy solution is replaced.

Re:Why do you need to backup the desktops?

[MBCook]

I agree. I assumed that the image of the computer(s) would be included in the backup. Having those images will save you a ton of time, even if each image is only for 50 computers.

That said, there is a big difference between backing up the images and backing up each individual desktop in the company.

My backup?

[helioquake]

I dump stuff on undergrads. They've got to be good for something.

/heh, just Kidding. I just mirror my scsi disks with a big ultra-ATA device weekly and daily.

Re:My backup?

[jrockway]

> I just mirror my scsi disks with a big ultra-ATA device weekly and daily.

You might like my backup software, Chroniton. It will happily run from cron and make incremental backups (and allow you to easily restore from one). It also stores everthing to the filesystem, so even if my software crashes and burns (which it won't; it's heavily tested in practice and with unit tests :), your data will still be just fine. All of your file's metadata is safely versioned and archived, as well. Take a look, it's perfect for backups to external hard drives (that's what it's designed for anyway).

Version 0.03, due out tomorrow, will support backup of extended filesystem attribures and will default to the builtin tar on systems without GNU tar. :)

Re:My backup?

[helioquake]

Thanks, but I have a couple of cron jobs running with my own bourne shell scripts for backup. Restore is easy since I just rsync when backing up.

Re:My backup?

[Millenniumman]

Version .03? Is the developer still only on the first line of code?

Re:My backup?

[jrockway]

> Version .03? Is the developer still only on the first line of code?

Perl programs traditionally start at 0.01 and move up by "hundredths" from there. Development releases contain an underscore, to prevent confusion. For example, the first test release of Chroniton was 0.01_1.

If it makes you feel better, just mentally multiply the version number by one million... then my software is at version 30000!

My backup strategy

[Millenniumman]

My backup strategy consists of hoping that my hard drive doesn't fail before I get a new computer/hard drive. It's worked so far, even with a laptop.

Re:My backup strategy

[bigmouth_strikes]

It's funny because it's true...

Re: Make sure offsite IS OFFSITE

[Alien54]

If you live in Southern California, there are four seasons:

Fire, Flood, Mud, and Earthquake

In which case, the best case for off site backup is out of state, like Las Vegas or something. This also gives you an excellent excuse for monthly road trips to "check out the quality of the backups"

That said, for simple off site backups, solutions like MOZY.com do just fine for a small small business. Otherwise, something like LiveVault.com is recommended. There are plenty of vendors out there.

Another thing is the insurance for replacements for each of your software media. Things like MS can be done in bulk via several MSDN subscriptions, a bargain even if you never develop anything. (300 bucks get you copies of everything MS is currently shipping, along with extra CDKeys for many items). In fact insurance for the media and other details is a very good idea.

It's very nice if the backup facility is also located at the bottom of a retired ICBM missile silo, or something similar.

best way to backup

[novastar123]

the best way ive found yet is to back everything up to /dev/null its incredibally fast and saves you on storage space for backup tapes.

Re:best way to backup

[schnits0r]

Thank you for the suggestion. I am doing that now, and my boss has given me a raise for that innovative idea.

Re:best way to backup

[kahanamoku]

LMAO

The restore process is even quicker, and works twice as well!

Re:just what ever you do make sure offsite IS OFFS

[Anonymous Coward]

In my opinion a good backup policy should account for anything up to and including a direct nuclear strike, if I manage to get superpowers and live, I don't want to have still lost my data.

What Policy?

[MissingIntellect]

Cross my fingers and hope the hard drive gods have pity on my pathetic self.

My backup policy:

[AEton]

I pray.

rsnapshot

[alfs boner]

I use and recommend rsnapshot for taking disk-to-disk backups of unix based servers and PCs. It has a *really* slick directory structure where each daily/weekly/monthly backup directory is a *full* snapshot - but by using hard links, it only saves the changed files multiple times. Also, because it uses rsync, it only copies changed files across the network, and can use ssh no problem.

It's downsides: it's basically just a wrapper for rsync. It requires a lot of babysitting (if your backups fail for some reason, it'll try to do full backups the next day possibly with disasterous consequences as it tries to jam hundreds of gig down your T1). Also, it has to log in as root on all of your boxes, so there are some very careful sercurity considerations.

But a box with a bunch of disks in it, put it off site, and whamo you have a complete backup solution.

For the windows users, I like backuppc. I have never actually used it, but it allows windows users to choose when their backups are taken, and allows them to recover files themselves through a web interface. It's big downside is the cryptic way it stores files internally, making it really hard to extract files without using the web interface.

My backup

[Anonymous Coward]

I have 10 chinese kids in my basement memorizing 0's and 1's.

my backup policy

[mctk]

1 click down, yell "Clear" and hit the gas.

Paper

[NetDanzr]

My backup copy is paper. Granted, it gets a little awkward when I move, as I currently have six large file boxes of that stuff, but I know that as long as I keep it reasonably safe from humidity/mice it'll outlive all my computer media and file format changes.

At work we do the same, only to a larger extent. We've got an on-site and off-site storage, and each piece of information is printed in two copies to be stored at each. All that in addition to your usual Veritas tape and CD-RW backups, which we do for convenience of restoring lost data, but which we don't trust enough to eliminate paper copies.

Re:Paper

[students]

If you are making backups every week, you only need them to last one week. Paper makes sense for an archive if you plan on needing the data long after you have stopped creating new data, but while you are working a short-term, cheap, space efficient and environmentally friendly solution is better.

Re:Paper

[Leibel]

When has the paper backup proved useful? I can't imagine running even a small/medium business and being able to retrieve useful information after a disaster. And then what do you do? Rekey it into a brand new system?

I think paper based backups would be fine if you had a paper-based business, but if you use databases to make it easier to getting to stuff, a paper-based recovery seems crazy.

You'd be far better off IMHO to get your tape backups to a state where they are reliable. Even if that means running a full Disaster Recovery test each week. If you don't trust the tapes, don't use them. They are useless. Get something better.

The main goal is to recover to a point that the business deems acceptable (ASAP, of course) for as little money as possible (of course), assuming everything in the building is destroyed (even the paper!). The costs and overheads obviously increase the closer you want to recover to the time of disaster.

Re:Paper

[NetDanzr]

When has the paper backup proved useful? I can't imagine running even a small/medium business and being able to retrieve useful information after a disaster. And then what do you do? Rekey it into a brand new system?

That's why I mentioned that we also keep electronic copies, for convenience, but ultimatelly paper copies are the primary backup. It works very well even in database-driven environment, as long as you don't update fields in a database, but add new rows. And that's exactly what we're doing at work, and what I'm doing at home: in order to document past actions, you don't change them; instead you create an adjustment. As such, all you have to do is to print out the new rows.

I personally do so at home primarily with Quicken, and with various articles I've written or e-mails I've gotten. Keeping track of them is fairly simple; I have a sheet taped to each box that lists its contents, and where I can write whatever I added to the boxes. Libraries have been doing something similar, but on a much greater scale for ages, and it worked just fine even without a computer.

Re:Paper

[pogle]

It cant be *that* hard to restore backups from paper....right?

http://ars.userfriendly.org/cartoons/?id=19971127

Business Continuity Planning?

[John the Kiwi]

I think you're jumping the gun a little here.

The first question you need to ask is:

What is the time frame for your servers to be restored in should servers and such completely fail?

If you don't know that answer to that question then how does your company know how much money to budget? Are you bound by HIPAA or Sarbanes-Oxley? You should know how much is your company's data worth prior to assigning a bidget.

Are some of your database servers supposed to be up 24x7? Maybe you should look at distributed transactions across databases located at different sites so if one server fails you still have everything live? Have you timed how long it takes to rebuild your servers to confirm your allotted time in your disaster recovery plan? Has your company considered imaging servers/ Is it possible to?

Have you consulted your disaster recovery plan? Have you checked with suppliers to see how long replacement parts will take to order? I can't tell you how many administrators get caught out by buying an expensive tape drive only to have it fail along woith the server and nothing can be restored until a new one can be sourced.

Without requirements, a disaster recovery time frame you will never be in control in the event of a disaster.

Your companies board of directors/owners will need this information. It's called operating under conditions of "due care and diligence".

If something goes wrong and you can't tell your boss exactly what is required and how long it will take to recover then you're working in the wrong job - a big part of being a network administrator is planning for ANY event.

Oh, most of the time my customers are happy with Robocopy. I hate paying for expensive hardware and backup software solutions when I can write something much simpler and document it properly rather than depending on someone else's buggy software. Of course this depends on the industry and their requirements.

Make sure that your boss completely understands these questions and issues. Ask him to see the current Business Continuity plan and Disaster Recovery documentation before you touch anything on those servers - can't stress that enough.

Hope that helps, sorry it's brief but if you're in charge of backups it's your job to be ANAL and PEDANTIC.

backup?

[apathy maybe]

Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies.

Re:backup?

[m_TheRedHead]

Wow, you can quote Linus and you don't even give him credit.

Re:backup?

[Anonymous Coward]

People should already know that was a Linus quote. If they don't they shouldn't be reading Slashdot.

Encryption?

[CFrankBernard]

What's a good inexpensive backup package for Windows that saves data encrypted to tape?

The Help in Backup Exec mentions that the password (if specified) will be required when accessing the files from within any Backup Exec program. I assume that means the data on the tape is not encrypted? I searched Symantec's Backup Exec 10d's online PDF manual but "encrypt" appears to be available only for DLO (Desktop/Laptop Option).

Maybe NovaBACKUP http://www.novastor.com/pcbackup/backup/n_backup.h tml ?

Re:Encryption?

[esper]

It's not something I've ever looked at, but I kind of doubt that encrypted backups are likely to be popular enough at the "serious backup" level for the simple reason that tape manufacturers advertise the (average) compressed capacity of the tape, with compression being done in hardware by the drive. This has generally been about twice the actual raw, uncompressed capacity of the tape. (It may be higher now; it's been a couple years since I went to virtual tapes.) Well-encrypted data is uncompressible, or very close to it (as in "about as likely to get bigger when compressed as it is to get smaller"). So how long do you think a publisher of encrypted backup software is going to last when corporate execs start to notice that, when they use the encrypted backup software, they can only get 200G onto their 400G tapes?

Re:Encryption?

[CFrankBernard]

In an era of larger/cheaper tapes/drives and news stories about the tapes being stolen (or unauthorized reading/copying), I should hope encryption is at least an option in the (several-hundred dollar) tape backup packages. Besides, there are plenty of encryption programs for Windows and USB sticks. I want this available in the scheduled backup tape job definitions : )

Re:Encryption?

[Anonymous Coward]

Encryption is available with NetBackup, but apparently not quite yet available for BackupExec.

Someone else mentioned that you can't effectively compress encrypted data. True. If you want/need compression, you need to turn on client-side compression, which will happen before the encryption. Note that this will suck up CPU on the client and make your backups slow.

Re:Encryption?

[fimbulvetr]

AMANDA does encryption. It even does RAIT, tape changes, tape spanning, client compression, and so on. I've used it for 8 years and have yet to be disappointed.

Re:Encryption?

[Noksagt]

While it is true that AMANDA can do client or server side data encryption and/or transport encryption, I'd not suggest using it for win32 servers (as the grand parent asks). If you're able to put in a cheap *nix backup server, running AMANDA under cygwin certainly works. But I don't know if client-side encryption works (client-side compression hasn't worked for cygwin clients) & don't know how well using AMANDA with kerb/ssh on cygwin works.

Re:Encryption?

[Anonymous Coward]

Software based encryption is SLOW! Expect your throughput to drop by 50% if you do encryption or ~60% if you do a compression and encryotion. HW compression doesn't work since encrypted data isn't compressible.

VMs

[OiBoy]

We moved all of our servers to VMware virtual machines. Now we back them all up every night, some of them we even back up multiple times a day. We tried esxRanger first, but it took too long (back up of all of the VMs took 4 days) and used too much space. Then we moved to esXpress, which does differential backups of VMs, so it is MUCH faster and uses MUCH less space. We keep 30 days worth of backups online, but once a week we cut tapes of the monthly full and that week's differentials and ship it off-site.

The beauty of VM backups is that it is a backup of the entire system. You restore it and power on and you are running. You don't have to install an OS first and then the agent, and then restore a tape over the fresh OS install and hope it works.

Restores take exactly as much time as it takes you to read the data off of tape/disk. Once it is done reading/writing you just turn it on. Talk about your easy DR. However, for workstations this won't fly, but you really shouldn't be keeping anything important on workstations, that's why you have servers. We treat workstations as an interchangeable part. You log into any workstation with your credentials and you see your desktop/files/settings/etc, since they are all actually on the server. Our Windows workstations achieve this with roaming profiles and our Linux/UNIX workstations do it with NFS mounted home directories.

I ignore the sign...

[Ignominious Cow Herd]

...that says "Severe Tire Damage!"

Backups ???

[eclectro]

Who bothers with backups? I've personally never wasted any time backing

A fatal exeeption 0E has occurred at 0137:BFFA21C9. The current application will be terminated.

  * Press any key to terminate the current application
  * Press CTRL+ALT+DEL again to restart your computer. You will lose any unsaved information in all applications.

                  Press any key to continue _

Oh shit! Oh shit! What do we do now?!

[SlappyBastard]

Please God... please say someone took the project home on CD, or we're fucked!

Re: Make sure offsite IS OFFSITE

[techno-vampire]

Fire, Flood, Mud, and Earthquake

Close, but no cigar. The four seasons in Southern California are Fire, Flood, Earthquake and Riot. I should know; I'm the one who posted that to rec.humor.funny about fourteen years ago. Besides, Mud is just a subsidiary of Flood.

get a real file server

[SQLz]

get a real file server,a small tape robot and veritas.

It's not about backups, it's about restores.

[mlheur]

I don't give two hoots for a backup policy. What you need is a data recovery policy. When will I need to recover data, and how will it that be attained.

I've been working with Symantec (formerly Veritas) Netbackup in my workplace for the past 6 years. About 6 months ago I became one of the backup admins, and the biggest barrier I have to break with our clients is the backup mentality - I must backup everything all the time...

Generally your data recovery will happen from two triggers:

1. A user broke his own stuff and needs a file restored.
2. Disaster Recovery.

Each has different requirements, user wants the backup copy to be onsite, DR wants it to be offsite.
User PC's typically can be rebuilt/imaged in a disaster, you're not going to have a hot-site contract for PC's. If your DR plan is to install an OS, install a backup/restore client software then recover databases/applications, then why fret about backing up the OS?

Our policy is as follows

NT/Unix OS and flat files
Monthly full backups retained for 13 months
Weekly full backups retained for 6 weeks
Daily cumulative incremental (everything changed since the last full) retained for 15 days

Oracle Datafiles
Weekly cold for 13 months
Daily hot for 6 weeks
1-6 hour archive logs for 15 days

Exchange Datastores
Daily full for 6 weeks
Weekly full for 13 months

Every day any full backups that are more than 10 days old (not copy 0) are sent offsite.

Any customer that has a DRP contract (banks etc) with a 4 hour recovery policy (we have 3 days to get the system back to how it was 4 hours before the disaster) we either run inline tape copies, one for onsite and one for offsite, or else we backup overnight and duplicate during the day.

Your most important backup (for Netbackup) is your catalog. If you go to DR and all you have is a box of tapes, good luck. You need to know what data is on what tape, and the only thing that knows that is the Netbackup catalog.
I don't know much about other backup products (HP OpenView and BackupExec are the only others I've touched), but I'm sure they'll have something similar.

I've got lots more to say, but I don't have time to put it all down now. Send me a /. message if you want more info.

Re: Make sure offsite IS OFFSITE

[Anonymous Coward]

what about a'lot?

Don't just look in the rear-view mirrors...

[tverbeek]

...actually turn your upper body around, so you can look in the direction you're driving.

Think of the children!

We use rsync

[dtfinch]

Rsync is very good at keeping two servers in sync with minimal bandwidth and disk activity, and can be configured so that you never lose a past revision. I have it set up so we have the latest copy, two weeks of revisions, and one previous revision for each file on every file share.

Some special consideration is needed for Windows servers. Some files get locked so they can't be read by rsync. We're not backing up anything that we'd run into that problem with, and we back up during a period of inactivity, but otherwise I suppose the right thing to do would be to create a volume shadow copy (look for vshadow.exe in the VSS SDK) so that rsync can back up from a consistent, non-locked copy of your data.

I'm still fighting to get our offsite backup hosting approved. It's no use to have all your servers back up to another server if it's in the same room and a fire takes them all out at once. Backup hosting advertised as backup hosting can get very expensive, but you can get dedicated linux hosting for probably 1/5th the monthly rate and configure it yourself.

There's a lot of backup software out there based on rsync. Rdiff-backup, duplicity, and BackupPC come to mind.

One question nobody's asked yet.

[techno-vampire]

You write that you're archiving your old backups. This is good, of course, for several reasons. You need multiple copies in case the newest one isn't usable, and you may need to acess old data. However, how far back do you plan to go in saving old data? If you just keep all backups from now on, you'll have an endlessly rising storage fee because they'll just take up more and more room, and the chances you'll need the older data will get smaller and smaller. Part of creating a good backup policy is deciding how much to keep, and what to do with obsolete tapes. In some cases, such as tax records, you may be bound by legal requirements, but even so, there's no need to keep them once they're old enough the government can't request them any longer.

My back up policy is ...

[the real darkskye]

Remember to change the tapes!

the cron scripts don't work otherwise!

#!/bin/sh

# Daily backup script

rm -rf /var/db/mysql_tmp
mkdir /var/db/mysql_tmp /usr/local/etc/rc.d/000.mysql-server.sh stop
cp -R /var/db/mysql/./ /var/db/mysql_tmp/ /usr/local/etc/rc.d/000.mysql-server.sh start
find /home/*/public_html /home/*/Mail /var/mail /usr/local/www/ /etc -newer /root/backup/last_backup -and \( -type f -or -type l \) > /root/backup/daily_increment
find /var/db/mysql_tmp \( -type f -or -type l \) >> /root/backup/daily_increment
touch /root/backup/last_backup
tar -cT /root/backup/daily_increment
mt rewoffl
cp /root/backup/daily_increment /var/log/backup
mv /root/backup/daily_increment /root/backup/daily_increment.`date "+%Y%m%d"`

and

#!/bin/sh

# Weekly backup script

rm -rf /var/db/mysql_tmp
mkdir /var/db/mysql_tmp /usr/local/etc/rc.d/000.mysql-server.sh stop
cp -R /var/db/mysql/./ /var/db/mysql_tmp/ /usr/local/etc/rc.d/000.mysql-server.sh start
touch /root/backup/last_backup
tar -c /home/*/public_html /home/*/Mail /var/mail /usr/local/www/ /var/db/mysql_tmp/ /etc /usr/local/etc
mt rewoffl
echo "Full weekly backup of /home/*/public_html /home/*/Mail /var/mail /usr/local/www/ /var/db/mysql_tmp/ /etc /usr/local/etc" > /var/log/backup

My backup policy is in my sig

[Curl E]

Paraphrasing a certain Mr. Torvalds:

Comment removed

[account_deleted]

Comment removed based on user account deletion

Comment removed

[account_deleted]

Comment removed based on user account deletion

Formalise it.

[Anonymous Coward]

Some general comments here; some apply to the original poster, some don't, but they are all points that need to be considered.

First of all: what data needs to be retained, and for how long? Some data has to be kept for several years to satisfy regulatory requirements. That data should be backed up to tape, and several copies made; each copy should be sent to a different site. Make sure that each site also has whatever software and hardware is needed to recover that data - it's all too easy for formats to go out of fashion, and ten years later, find out that you can't read that stuff you desperately need for some lawsuit. Case in point: the BBC had a Doomsday project, in which they stored a large amount of data on a laserdisc. Ten years later, they had to scrounge to find an old Acorn system that hadn't been manufactured for eight years in order to be able to read the disc.

Secondly: as others have said, verify your backups. It's no good writing stuff to tape; you have to be able to read it. One guy I know was called out to a site to do a restore. He asks where the backup tape is, and it's handed to him. He pulls the tape out of the housing: "Hm. I didn't know these things come in clear." Yes: they'd used that tape so many times, the magnetic oxide had flaked off. Completely.

Third: do a cost-benefit evaluation. It's no good spending a million bucks on a backup server if the total value of the data you're backing up is only a hundred thousand dollars.

Fourth: get management involved. They don't need to know all the technical details, but they do have a better idea than you do of how long they need to keep the data. Make sure they understand the cost involved in longer retention periods (twice as long => twice as many tapes), then let them make the call. That way, it's not your butt on the line if you get it wrong.

In my specific case, I work at a major university in Australia. We use Tivoli Storage Manager for our backups: all servers get one initial full backup ("incremental from zero", to be precise), then all backups are incremental. TSM makes two copies of everything it backs up (in the process, it verifies that the original backup is readable), and we've configured things so that one tape silo is on site, one off site (both are live at any given time, thanks to fibre links between our data centres). Initial backups go to the "off site" silo, and the copies go back to the "on site" silo. Both silos use LTO2 (we're in the process of upgrading one to LTO3 drives, then we'll introduce LTO3 media. The LTO2 media will be moved over to the LTO2 silo as it's freed up.)

Data which needs to be kept for longer than 90 days is stored using the Archive function, which makes a copy independant of the backups. Everybody who sets up a server is required to sign off on its backup requirements; if they say "it doesn't need backups", we have it in writing. We also have software in place that alerts us if a server misses its regular backups.

Good luck. It's a thankless job, and if you're lucky, you'll never be called upon to do a restore.

Re:Formalise it.

[draxbear]

I use Tivoli Storage Manager too and it is a very unsung hero of the backup/restore battleground.

You touch on it briefly but it's well worth highlighting how you are converting from LTO2 to LTO3. It doesn't require significant effort to perform this conversion thanks to the "separation" of data from the medium it is stored on by TSM. TSM handles all the underlying work, you just tell it to "move that data from LTO2 to LTO3 storage mediums please" and it chugs away in the background doing it.

No restores and re-archiving on new the new media type.
No shrink-wrapping of "god I hope it works in 1-7 years" old tape hardware.

My Policy: NEVER backup. Archive instead.

[sakusha]

I never NEVER backup. It is futile, a huge waste of time, and a monumental risk. The only time I have ever lost data was while performing backups. Let me give you an example.

Way back around 1979, it was my first serious development job, and as the junior programmer in the shop I had the onerous duty of performing the weekly backups of our production drive, containing all the code for our accounting software development. We had a big 10Gb Corvus hard drive (the original Winchester) networked to our Apple IIs and pre-release Apple /// machines, and it took a whole case of floppy disks to back up the system, IIRC something like 180 floppies. It took goddam forever, so I usually started early in the morning on Fridays, and the network was unusable during the backup, so everyone (except ME) got to sleep in on Fridays (usually they didn't show up at all).
So I spent all morning doing the backup, shuffling floppies in and out of the stupid disk drive, then when it was all done, doing a verification pass, which took just as long as the backup. All was well, I turned the hard drive back to network usage, verified it was operational, and went for a long lunch and a haircut. Nobody was around when I left, so I locked the door and left.
I came back to the office at about 2PM, everyone was there and they all started yelling at me immediately. The moment I left the office for lunch, the Corvus hard drive died. It was totally not my fault, it was a random hardware failure, at least it was a good thing it happened AFTER the backup was finished. The only problem was, it was the only unit we had, it would take 2 weeks to get a replacement drive shipped and installed (hard drives were very hard to obtain back then). Of course everyone jumped to the conclusion that I killed the drive, I protested that it was working correctly when I left, and I tried to reassure them that the backup was verified so no work was lost. Everyone assumed that since I had taken 2 hours for lunch (I had scheduled the haircut a week earlier with my boss, who wasn't in the office that Friday) that I had killed the hard drive, then fled in a panic. No matter what I said, nobody believed me. I got fired that day. Since that day, I have never performed backups, and I have never lost ANY data. I have a totally different strategy.

System and app backups are totally useless. Sys configs and apps can be replaced easily by a fresh install, and much quicker than doing a restore, and you don't have to waste time doing repetitive, useless, time-wasting backups every week/month/etc.
The only data worth saving is irreplaceable data. This should not be backed up, but instead, archived. For the most precious data, for example, photo and graphic design jobs I created from scratch for customers, those are archived immediately upon delivery (I usually work on short deliverable cycles, like a week or two at most). Routine archiving of each job to CDs or DVDS makes it easy to create an extra copy as a deliverable, and one for my archives. Other major file stores, for example, my mp3 collection, is archived each time I acquire a full DVD worth of new data. It is better to archive as you go, than to waste time in full system backups.

Rather than wasting time with backups, it is preferable to make sure your disks don't NEED restoration, by scrupulously maintaining them in optimal condition. Maintenance is better than backups. I buy only high quality hard drives, I monitor their performance and integrity continuously, and as a result, I've never had a single hard drive failure in any system I've owned, and that goes back almost 20 years. I have never lost any data.

Re:My Policy: NEVER backup. Archive instead.

[Ryosen]

That's just ducky when the building burns down, the office is vandalized, the hardware is stolen, someone deletes the files, the fire system malfunctions and triggers the automatic sprinkler system, you hit 'delete' when you meant to hit 'enter', it turns out that your source control didn't quite control your source as much as you thought it had, you fire the wrong person, you hire the wrong person, someone does something they shouldn't have been doing and the equipment gets impounded, the bills weren't paid on time and the equipment gets impounded, your new superstar admin faked his references and actually believed the /dev/null joke mentioned above, your hard drive crashed.

Oh, I know, you were meticulous in its care. Not good enough. Quite simply, 100% of all hard drives fail. It's not a question of 'if', merely 'when'. Only about 38% of data loss is due to equipment failure. (source: University of Texas Center for Research & Information Services).

As for archiving versus backup, you overlook one small detail: business interruption: the cost of downtime incurred while the archived data is reconstructed into a usuable form. A staggering 93% of companies that lost their data for 10 days or more due to a disaster filed for bankruptcy within one year. 50% of those companies files for bankruptcy immediately (source: National Archives and Record Administration). In a recent survey, 42% of companies responded 72 hours as their threshold for survival without their data. The other 58% cited less than 72 hours.

I own a company that provides data recovery solutions and I have seen far too many people who thought they were invulnerable, just like you. We help companies understand that it's not just whether the data is backed up, it's how you plan to recover it.

Re:My Policy: NEVER backup. Archive instead.

[sakusha]

You weren't paying attention. I specifically said that due to my diligent maintenance, I have a 0% hard drive failure rate over the last 20 years. I just retired a server with an Atlas 10K SCSI drive that ran 24/7/365 for over 5 years without a single problem, not even a soft error. That's what happens when you buy quality products, like high-end SCSI drives instead of cheapshit IDE drives.

Yes, I am invulnerable. My OS and apps are backed up on their original distribution discs. My handmade data is archived almost as fast as I create it. Backups are useless. Maintenance and archiving is a better approach. I don't need to hear lectures from someone like you who sells backup systems by slinging FUD and scary statistics.

Re:My Policy: NEVER backup. Archive instead.

[techno-vampire]

We had a big 10Gb Corvus hard drive (the original Winchester)...

That didn't sound right, so I did a little checking. FOLDOC tells me that the drives got their name because they had two 30meg volumes, rather like the Winchester 30-30. If you really were working with a 10Gig drive, it wasn't a Winchester, and it wasn't in 1979, either, because they didn't have drives that big back then.

Re:My Policy: NEVER backup. Archive instead.

[Detritus]

Non-removable disk drives were commonly referred to as Winchesters back then, even if they were not made by IBM. IBM pioneered the technology that later became universal in the disk drive industry. The original poster probably mistyped GB instead of MB.

Re:My Policy: NEVER backup. Archive instead.

[ebh]

Seconded. At one place I worked in 1981-82 we had a Corvus 33MB 12-inch "Winchester", which was likely a follow-on to the drive the GP refers to, hooked up to this weird "multi-user" CP/M box that consisted of a bunch of S-100 single-board Z-80 computers on a common backplane that could all access the "big" drive.

(Anyone remember the brand name? IIRC, the front panel had a power switch and big red square reset buttons for each CPU. It was featured in Byte or one of the other mags of the day in a cover story on multiprocessing vs. multitasking.)

This was a business literally running out of a garage. Our backup (and configuration management) policy consisted of printing out our code and putting the listings in special bins. Good times.

Re:My Policy: NEVER backup. Archive instead.

[nbvb]

Cubix?

I have touched a Cubix machine in ~10 years (since my Banyan VINES days ...) but they were "the" go-to company for blades, long before we called them blades. :)

Re:My Policy: NEVER backup. Archive instead.

[techno-vampire]

Considering how long ago it is, I'd not be surprised if the memory had faded a tad. That's why I just pointed out the discrepancy and didn't call the poster a liar. for the time, the typo you suggest is another good explanation.

Re:My Policy: NEVER backup. Archive instead.

[Ryosen]

FUD? Ignoring for the moment that you've proven yourself to be little more than an elitist ass, one who doesn't seem to have much experience in a larger organization, these are not scare tactics. They are simple facts. Just because they run counter to your ignorance does not mean that they don't have applicability in the real world. And, as I said before, the risk isn't the loss of the data as much as the business interruption. All the diligent maintenance in the world isn't going to make a bit of difference when the building burns down and you have no infrastructure redundancy.

The simple fact of the matter is that many companies either lack the technical experience or the simple motivation to implement a disaster recovery plan, of which data backup is a part. Many companies prefer to think that data loss will never happen to them. Some, like yourself, are either unintentionally unaware or purposefully ignorant of the various risks. And the stats that I cited before are from reputable organizations. You can add other organizations that are not selling a product, such as the American Red Cross or FEMA, to that list. It is common knowledge that you should protect your data. For you to insist that the sole threat is the quality of the hardware is reckless and irresponsible. That your answer to the issue of equipment's quality is to "buy a mac" speaks volumes of your inexperience and immediately discounts you as someone of any consequence. You're a fan boy, and not a particularly good one at that. Step aside and let those of us that take care of large-scale computer systems do the real work.

I'm happy for you that you haven't had a problem in 20 years. I will be sure to cite you as a lucky example of misguided ignorance in my next sales presentation and toast your good fortune as I celebrate my next success.

Oh, and FWIW, the majority of clients that I serve need to recover data that was deleted, accidentally or otherwise. They, too, have had good luck with the equipment that they purchased. And, fancy that, they're not even using Macs.

The fools.

Re:My Policy: NEVER backup. Archive instead.

[sakusha]

I might be wrong about the size, but these were the original "Winchester" models, now that you mention it I don't recall if they were 10, 20 or 30Mb, all I remember is that they were so huge (to us then) and that they took goddam forever to back up to floppies. The drives were big grey boxes with a translucent plastic lid, you could see the heads move across the platter. Then those were placed inside a Corvus white box. My understanding is that these Winchester drives were originally produced for IBM mainframes but IBM didn't like them so the limited production went to OEMs like Corvus. These were the first commercially available hard drives for microcomputers like the Apple II, and ran Omninet for multiuser storage networking. That worked like crap, because the Apple DOS system (and the Apple /// SOS system we were helping develop) didn't have robust file and record locking.

Yeah, the 10Gb was a typo, it's hard to believe that at one time hard drives with 10 or 20 megs was considered huge.

Re:My Policy: NEVER backup. Archive instead.

[Christopher Cashell]

No offense, but you've shown your ignorance here. First of all, the original poster is discussing corporate backups, while you're talking about backups for your own personal stuff. What works for your one or two computers at home is not likely to work for a company with hundreds of workstations and dozens of servers.

Many of your statements just flat out don't make sense when you consider larger scale or corporate computing environments.

For example: System and app backups are totally useless. Sys configs and apps can be replaced easily by a fresh install, and much quicker than doing a restore, and you don't have to waste time doing repetitive, useless, time-wasting backups every week/month/etc.

You've never installed Exchange, have you? Or Oracle? Or Cisco CallManager? ActiveDirectory? Sonexis? RSA ACE Authentication Manager? Nagios? Any large scale "enterprise" application?

These are not trivial things to install. This "fresh install" is not going to be quicker than a restore, it's going to potentially take dozens of times longer. Utilizing a real backup scheme, I can restore our Exchange cluster from backup in about an hour or two. If I were to try to install them from original media and reconfigure them from scratch, it would take me at least 20 hours, and that's assuming I had mail store backups.

I also found the following comment humorous:

I buy only high quality hard drives, I monitor their performance and integrity continuously, and as a result, I've never had a single hard drive failure in any system I've owned, and that goes back almost 20 years. I have never lost any data.

Wait! Didn't you just tell us a story where you said:

The moment I left the office for lunch, the Corvus hard drive died. It was totally not my fault, it was a random hardware failure, at least it was a good thing it happened AFTER the backup was finished.

So you carefully monitor your hardwar and trust that it won't fail, but at the same time, you've experienced a random hardware failure that was totally not your fault? Heck, I run all servers with mirrored hard drives, and I still don't trust them without backups.

The fact that you've been lucky enough that you haven't personally lost data at home is great. But it's because of luck more than anything. A hard drive can fail without any sort of warning, regardless of its quality. Additionally, your scheme doesn't take into account one of the most common reasons that data needs to be restored, and that's user error. All of this is also without considering any "catastrophic" or even remotely close events. What happens if a power strike fries the whole computer, motherboard, drives, and all? Or even a power supply in a computer that dies, and takes out the motherboard and drives (I've personally seen this happen). You never responded to how you would handle something like a fire that destroys your computers. How do you recover from that?

Good luck with your data, I hope you don't experience bad luck and lose important information. And no offense, but I hope you are never involved witih the IT department at my company. ;-)

Tape Storage: Safes and Offsite Drives

[Chris Tyler]

Two thoughts related to storage:

- Consider carefully whether you trust your tape safe. I've seen tapes damaged at temperatures lower than some tape safes are rated for.

- If you have offsite backups, you should also have offsite tape drives. If your main site is destroyed in some catastrophic disaster, it's not too hard to get emergency replacements for server hardware, especially x86. But urgently sourcing the right model of tape drive (in many cases a model that is a few years old) can be a nightmare. While most drives of [insert favorite tape technology family here] generation N+1 will read tapes from generation N, it's very easy for time to slip by to the point that most of the drives on the market are actually generation N+4, and their support for reading generation N is spotty (even if the spec sheet says otherwise). If you're using proprietary backup software, make sure you can get your hands on a copy of it quickly, too.

The most elegant solution..

[Xross_Ied]

is http://www.avamar.com/

The Backup server or cluster of servers store 20KB blocks keyed to the block's SHA-1 hash.

Smart agents on each backup client chunks each new file to be backed up into 20KB blocks and calculates SHA-1 hashes which it compares against the backup server.

If the block is new (not on the backup server) the block itself is transfered.
If the block is old, the backup server stores an extra reference to the block for the client/file.

The end result is..
a) a 1000 windows backup clients will result in only one copy of a windows dll being saved
b) every full backup is like an differential in terms of size/speed.
c) you have weeks (if not months) of daily backups of all backup clients stored in 110% of total backup size.
d) the backup agent on the client has a larger footprint and requires more CPU while running.

Put the backup servers at a remote site with a high speed link and you have disaster recovery as well.
If the high speed link isn't an option, there is support for remote replication; requires another backup server (or cluster) of same size.

This is the way all backups will be done in the future.

Mod parent up

[tengu1sd]

Before you start spending money you need to know what the company requirements are. There are excellent tools and options, including real time raid-1 over mutliple sites, but the business case will drive your requirements.

Servers - how long can they be down? Do you have replacement plans in case your data center gets hit by the next earthquake/hurricane/fill_in_the_disaster. Having tapes off site means nothing if you don't have hardware for restore. Can you get Hardware X if everyone else is looking for X, maybe Y is the new standard and you're application needs X version 1.2.

Desktops, are files on a server or local? Do you have a standard desktop that can be rolled out and copies of the server. Can the desktops go 2 weeks, but you need the servers back in 12 hours. You need a plan before things get ugly.

Speaking of tapes, as mentioned you need to periodically check your restore. Backups don't matter, it's whether or not you can restore your data that counts. How often, incremental or full. Be careful shipping tapes. Since 9/11 I've noticed tapes shipped with certain carriers have read issues at the remote site. Is this X-rays on cargo or just a bad run of tapes?

Veritas (aka Symantec)

[Anonymous Coward]

If cost is a big concern, you're likely going to end up with BackupExec. It's not my best friend in the world, but it generally works. And it's pretty damn cheap compared to NetBackup. If you're more concerned with functionality and ease of use, cough up the extra $$ and use NetBackup. I have several of each and I wish they were all NBU.

Of course, one of the biggest favors you can do yourself is to get reliable tape hardware. Vendors like STK and ATL are pretty expensive, but I can't tell you how much less likely I am to shoot people now that I have a big STK library instead of a collection of small, cheap libraries that were always having problems.

LTO-2 & LTO-3 are popular tapes these days. They're fast, they hold a lot, yada yada yada. Backup to the part about "they're fast". This is important. They have minimum write speeds. If you can't keep up, they'll constantly stop & start. When they do this, they have to rewind, start again, come up to speed, write a bit, stop, rewind, etc. This is referred to as shoe-shine. This is bad. If you're backing up small boxes that can't keep your tape drive going, you have 2 options. 1) multi-stream -- where multiple clients are backing up simultaneously to the same tape and the data is interleaved. I'm not a big fan of this. 2) buy a bunch of disk space for disk staging. Back up to disk and then migrate from there to tape. This is a better option, but it'll cost money (duh).

When you back up Exchange, you need to do 3 backups. 1) system state, c:, etc. Make sure you exclude the store and the M: drive. 2) Information Store. 3) Mailboxes (aka brick-levels). You need #2 if your store becomes corrupted, your RAID controller dies, etc. You need #3 to restore email when execs delete something and then call and say "I deleted something. Can I get it back?"

If you're shipping a lot of tapes on/off site, cough up even more money and get a NBU Vault license. It'll keep track of all that mess for you.

Despite popular belief, fulls don't have to run on Friday. They can also run on other days, like Tuesday and Sunday. If you're having a hard time getting all your fulls done in one day, pick a few servers and run their fulls on another day.

Oh -- one last thing about NBU -- run it on Solaris. I have 1 Windows NBU master/media server and I hate it. I'll not make that mistake again.

If you have some data that's really important, feel free to run fulls more often. I do InfoStore fulls 3 times a week. Some important databases (which aren't really all that big) run fulls every night. You'll be happy you did when you have to restore and don't have to replay 6 (or 13!!) days of transactions.

Having a bunch of different retention periods looks good on paper, but it's a bigger headache to manage. If you really want to do this, NBU Vault can take care of some of the pain. The more stuff like this you do, the more Vault will be worth it.

Backup policy?

[Patrik_AKA_RedX]

I don't bother with backups. I've got a airtight policy in case of a HD crash or any other form of data loss:
1)Look shocked and terrified.
2)Yell.
3)Scream.
4)Pull hear.
5)Bang head to wall.
6)sit quitely sobbing a corner.
7)Kick the cat.
8)Replace HD. (if necessary).
9)Reinstall software.
10)Kick cat again.
11)redownload mp3s, movies, games and pron.
12)Feed cat.
13)Mail goatse.cx pictures to random innocent people as an act of pointless revenge.
14)Make futile threats to a deity that if it happens again "the cat gets it".
15)continue life as normal.

Now what could possibly go wrong with my plan?

Re:Backup policy?

[spuby]

God, you killed me. I'm still trying to stop from laughing. And all because I started visualizing the part with the cat.

AMANDA

[Noksagt]

AMANDA is really great software. In my past job, we used Retrospect (then from Dantz). That was a nightmare--it used some proprietary archiving format & we weren't able to retrieve some things. AMANDA uses standard dump or tar files (well, as standard as 'dump' is, I guess), so I'm confident that that'll never happen. It also has a first-class scheduling system. Every night, we fill almost exactly one full tape. There are very few disks which don't get a nightly incremental & we have it configures so we are virtually guaranteed a full backup of every host at least once a week.

We use it with Linux, FreeBSD, AIX, OS X, and Windows (through Cygwin). The only "ugly" part is that we backup at night & laptops are therefore not part of the regular cycle. For those, we use rsync over SSH to backup to a central file server which does get backed up at night.

Open Source can be enterprise grade

[Noksagt]

BMR has been standard for years.

AMANDA and others have been deployed in large institutions for years too.

I've seen attempts to build large enterprise backup environments with "simple open" software. They melt down somewhat short of the size that the original questioner is asking about, typically.

I've certainly seen a lot of "home-rolled" scripts with tar & what not abused this way. I haven't seen an AMANDA installation that failed to scale. Have you seen problems with "not-so-simple" open source software?

I've been at places which were big enough to use 40, 60, 180, multi-hundred, 5000 tape changers.

One nice feature in AMANDA is support for RAIT. Commercial software can handle multiple tape changers just as easily as AMANDA, but a lot of it certainly doesn't take much advantage of the fact by providing RAIT (which gives you higher capacity and throughput, and lower failure rate).

This is important, and half-assed solutions shouldn't apply.

Just because you can see the source code doesn't mean it is "half-assed."

Re:Open Source can be enterprise grade

[georgewilliamherbert]

I use plenty of stuff for which I have the source code. Going back to the 4.2mumble BSDs, through SunOS, Linux, Solaris, the various x86 BSDs, and plenty of applications (this is Mozilla I'm /.ing with, and before that a long line of other open source browsers). I have no problem with installing large Linux farms, using Apache for an enterprise web deployment, using MySQL for moderate sized databases (or PostgreSQL, though I haven't deployed it personally).

Tape backup... NBU wins. Legato's a close second. Sorry, charlie. Open source as a category does not suck. The open source backup stuff doesn't suck, for small to medium sized sites. It's not enterprise class, though, and most of the trick to succeeding in IT is knowing when the tools you use aren't applicable anymore and how to figure out what are.

NBU can't RAIT, but it can stream across multiple tapes, and can write duplicate tapes if you want redundancy. And you can extract the files off tape with tar if you have to.

Amanda certainly doesn't suck, but it's not NBU.

Re:Open Source can be enterprise grade

[Noksagt]

most of the trick to succeeding in IT is knowing when the tools you use aren't applicable anymore and how to figure out what are.

I agree entirely with this statement.

The open source backup stuff doesn't suck, for small to medium sized sites. It's not enterprise class....Amanda certainly doesn't suck, but it's not NBU.

In what way have you found that Amanda does not scale? How have you found the proprietary software to be better?

NBU can't RAIT, but it can stream across multiple tapes, and can write duplicate tapes if you want redundancy. And you can extract the files off tape with tar if you have to.

I agree that NetBackup is good. It does have features that Amanda lacks (though Amanda has all of those that you list). But Amanda has features which NetBackup lacks too (such as RAIT).

I'm not saying NetBackup isn't worth money. I'm just asking you to give concrete criticism, rather than to dismiss alternatives as "half-assed" or "toys."

Re:Open Source can be enterprise grade

[georgewilliamherbert]

NBU advantages:

• Master server / slave (media) server
  • Central management point for the whole enterprise's backups (master server)
• User friendly restore management for end users
• Application-aware hot / warm backup plugins for enterprise apps like Oracle, SAP, Peoplesoft, Siebel, Informix, Sybase, Exchange ....
• Optional global management of multiple sites from a master master server
• Native clients for all OSes including Windows
• Tape vaulting management software addon
• Support for arbitrarily large tape libraries
• Block-level incremental backup option

Just off the top of my head.

Re:Open Source can be enterprise grade

[Immercenary_2000]

"Amanda certainly doesn't suck"

Well maybe if you bought her dinner first or at least a drink...

I know, I know, cheap joke but I couldn't resist

Rsync...

[IpSo_]

So far the best "backup" software I've used is rsync.

I used to work at one of the worlds most well known web hosting companies where among other things I ran their backup system. It started out with Arkeia and a 120tape library with 6 AIT3 drives. Arkeia was crap though (this was 3yrs ago), it was such a pain to setup and the trying to restore ANY amount of data would literally take days just to scan its local database. Trying to restore just one file would take 6hrs just for it to scan its local database... On a dual processor box with SCSI drives and 1gb of ram.

We moved to Veritas NetBackup, which was a dream to work with compared to Arkeia, but it too had issues (besides its cost). You could tell the software had been around for ages, it was far from being easy to use, or even efficient, but compared to Arkeia, it was a dream. It would start like 10+ processes, and every now and then one would die, causing everything to silently stop working and you would have restart them all. This usually caused at least one days worth of backups to fail. When you had 1TB of data to get in a 8hr window from a few hundred machines, it didn't take much to miss your window.

Tape backups are just a pain to use. They are slow to backup, and even slower to restore from. They need constant cleaning, and from my experience the drives fail more often then harddisks do. It seemed we were replacing about two tape drives a year. They aren't cheap either. Ouch!

The best backup system I've used so far is rsync with its nifty snapshot ability.

I setup the backup system for a company with locations in 10 different cities (connected with broadband), where each location has its own Linux server, and a central backup server at the main branch. Each employee has a H: which maps to a Samba share on the cities local Linux server, they save all their data to this location, and twice a day the main backup server rsync's the data back to the head office. Since this process happens twice a day, the amount of data that changes is quite minimal, a few hundred megs or less across the entire company, so it only takes a couple hours at most. The main backup server keeps these twice daily snapshots for about two months, and each week the main backup server itself is backed up to tapes. Luckily we have never had the need to restore from the tapes...

So basically all data is stored locally on a RAID'd server, then remotely on a RAID'd server at head office (twice daily), then offsite on tapes (weekly). The main benefit though is that executives, or the technical staff can pull data off the main backup server from any date in the last two months immediately, just by using Windows Explorer. No need to restore from tapes, and all the data is redundant in 3 locations. To do a restore, we basically just reverse the rsync script, and push the selected data from the main backup server back out to the local server.

Works like a charm, and its free!

rsync

[DrSkwid]

I run the same version of my OS on QEMU and have it rsync the data.

Using Hardlinks for EveryDayBackup on Disk

[Anonymous Coward]

Using "cp -al" for creating backups using hardlinks has made it possible for us to keep fullbackups online for every day the last 14days, every weeks from the last 14weeks and every month from when we started taking backups. The online backup are stored at two locations (two diffrent offices connected using VPN). Every weekend we take full tapebackups.

This enable us to retrieve files without digging for the tapes. Tapes are rotated out of the building to a remote safe :-) I hope we are good.

You say poe-tay-toe, I say poe-tah-toe ...

[RedDirt]

I guess I'm glad you feel invulnerable with the backup scheme you've crafted (and make no mistake, it is a backup - despite that you choose to call it archiving), but really you're just playing roulette with your data.

Now, having offended you, let me agree with some of the things you say. =)

Your assertion about maintaining a complete system backup is pretty spot on. The data is what you want to keep safe and the applications are perfectly able to be reloaded from the source media (provided, of course, that it's still available - but that's a different topic). I somewhat disagree that configs are unimportant. Though backups aren't necessarily the place for those either, rather they ought to be kept in some sort of change-control mechanism (CVS or whatever) for the sake of repeatability. That way, you can do an install on a fresh box, check out the configuration and proceed to dealing with the important stuff - the data.

Back to disagreeing.

Your current scheme of burning off your work whenever you deliver something makes you vulnerable to significant data loss via accidentally fat-fingering an rm, or catching a virus, or suffering the random angst of an irritated diety (other assorted phenomenon) since you claim immunity to such catastrophe. Your major complaint about backups seems to center around the amount of effort required to do them (well, that and the bad experience you had after performing one). This is why I think you should take some time to look at using rsync to do snapshot-style backups. My backups go nightly and are almost completely automated - I just have to plug in a different external USB drive into the server each morning (though if I forget, it's no big deal). Between that and the hardware RAID, I've reduced my rework window to no more than 24 hours. As an added bonus, I don't have to worry with buying or storing a pile of CDs/DVDs or worring whether they'll still be readable when I need data off the bloody things.

Now to quibble about terms, or the reason I accused you of still doing backups.

Archiving says that you take the project data and move it off primary storage and onto your CD/DVD. Yet, when you were speaking of your MP3 collection, you said you burned a DVD when you had enough new material to fill a disk - implying that the data was not moved off, merely copied off. According to Webster, that'd be a backup. =P Thus you do some of both, probably.

In summary:
    * Automated backups of your data so you don't have to actively worry about doing backups == good
    * Depending on CDs and/or DVDs to keep your data safe == silly
    * Calling a backup an archive == redefinition of terms
    * Asserting that certain components haven't failed in the past and will therefore never fail in the future == past performance is no indication of future returns

Re:You say poe-tay-toe, I say poe-tah-toe ...

[sakusha]

Please do not project your hardware neuroses onto me, I'm not using cheapshit white box PeeCees loaded with Windoze malware, I use pro Macs.

Your backup system isn't a procedure, it is a fetish. You claim your "rework window" is only 24 hours. How many 24-hour time slots have you wasted over the past few years, doing unnecessary backups?

Re:You say poe-tay-toe, I say poe-tah-toe ...

[RedDirt]

Hardware neuroses, Windoze malware, PeeCees and "pro Macs". THAT explains it. You're one of those old-school Macintosh persecution-complex cultists. All becomes clear now. I do happen have a pair of white-box Windows machines for gaming, though the rest of my gear is low to mid-grade server-class x86 hardware running Linux or FreeBSD (Tyan and Supermicro stuff). I also happend to have some Mac hardware (although you'd probably sneer at my Powerbook for not being "pro" enough).

I spend approximately 10 to 15 seconds unplugging and replugging the USB drives on the days that I swap them. How long does it take you to burn a DVD? 20 minutes? 30 minutes? Amortized over a three or four week period (your claimed backup window), I still win on time "wasted". That's why I love the way the rsync method works - I don't have to DO anything, the snapshots happen automatically as long as the target drive is attached.

Since you use Apple hardware, using rsync for backups is even easier, provided you're running OSX, since Apple bundles it with the OS. Here is a good resource for setting it up if you ever decide to explore other, more reliable options for data security.

Alas though, judging from your tone, you don't seem willing to engage in rational discourse. C'est la vie.

Honestly... Tivoli Storage Manager

[Colin Smith]

I've used Amanda, Bakula, Netbackup, Networker and by far the best of the bunch for enterprise size networks is TSM. Easily. Netbackup is something I still have cold sweats and nightmares about, ok, not quite nightmares, just the occasional cold sweat. It's really a small network system which has been kludged to "enterprise" class. TSM was designed for managing large network backups from the start.

Re:Honestly... Tivoli Storage Manager

[thempstead]

I would agree with this, having used various of the products mentioned, with the following comments ...

1. Be aware that TSM is quite expensive!
2. If you go with TSM get decent training for it. I have worked with several systems which have been setup incorrectly because the person(s) setting up the TSM system had not had sufficient training in order to configure things properly.
3. (Related to 2), make sure you know how to recover your TSM system in the event of a full DR, (not difficult if you know what you are doing but can beinteresting if you don't).

t

I love TSM

[Colin Smith]

I once had to move from TSM to Netbackup due to corporate policies. It was horrible. TSM's a fabulously powerful bit of kit i'd recommend to anyone looking for an enterprise backup system.

Rosary backup policy

[SpaghettiPattern]

I have a rosary backup policy. My prefered saints to pray to are Mary, Don Bosco, St. Ignatius of Loyola and St. IGNUcius.

Linux Is Your Friend

[QAPete]

My company does a couple of things that I thought I'd share with you. First, we run a multi-terabyte SAMBA fileserver, on which we have both departmental/project shares and every user's 'My Documents' folder. Second, we have a group policy that maps everyone's 'My Documents' folder to the appropriate SAMBA fileserver directory. Finally, my IT policy is communicated to all new and existing company computer users explaining this setup, and the fact that - aside from mail - the only user data backed up is located in those two areas.

We use a product called Storix http://www.storix.com/, which is a very inexpensively priced commercial .tar generating backup software, and we store our fileserver info on a DLT2 carousel tape changer. This software is very easy to use and maintain, and works on a wide variety of Linux/AIX systems. The software allows for backups ranging from incrementals to full bare sheet metal restores, including a boot CD. For a small additional fee, you can ensure your tape data is encrypted. Highly recommended.

An additional point I'd like to make is I'm a big fan of having MORE backup devices, lots of redundancy. Yes, it's more expensive, but if your data is truly mission-critical, consider having a backup device for every key server. The internal DAT tape backup drives on Dell rack servers are inexpensive, and will provide you with this redundancy, as well as the ability to back up multiple servers in parallel (allowing for more frequent backups). If your server is larger than what DAT will handle, paying approximately $5k more for an additional tape carousel is not what I'd consider an inappropriate insurance policy for this type of data.

Finally, your onsite and offsite storage safes need to be fireproof, lockable and combination/key changeable. Ensure that you have at least one recent full backup on and offsite for all your key servers, whether you do incrementals or not.

An aside: for HP-UX users, Ignite and BRU are a great way to go. Should your boot drives/partitions fail, you can boot from tape and be up and running your system in less than a couple hours.

Re:just what ever you do make sure offsite IS OFFS

[jrmcferren]

Where I'm at, all I have to make sure is that my offsite is not the city. I live in the Johnstown area between Johnstown and Westmont. As far as flooding goes, I'm safe. As far as fire goes, I can store my backups in my girlfriend's room as this building is designed that a fire in F-Dorm (where I'm at) cannot get to the next dorm (E or G). My girlfriend is in B-Dorm. If I would store backups, it would be at home (where my parents live) as it is another part of the state. I would jabber some more, but I have to get my ass to class.

What are your needs?

[ocbwilg]

First, determine your needs. Are you backing data up for disaster recovery purposes, data protection purposes, or archiving purposes to meet regulatory requirements? Or maybe some combination of the three? How long does this data need to be stored?

The most common technique is a weekly full backup with daily incremental backups. Depending upon your file retention requirements, you may be able to re-use the incremental tapes or you may have to append to them and then cycle them out when they are full. Most companies with any sense will send them to off-site storage, which can be records storage company or another company-owned site a good distance away.

As far as hardware goes, again, most companies will have several servers running backups simultaneously to multiple drives or libraries in order to reduce their backup window. The hardware varies depending upon the amount of data that needs to be backed up.

An often overlooked part of DR planning is to store a copy of the backup software and a spare media device with the backup media offsite. If you have a disaster and lose your backup devices and software, you may not be able to get the same hardware/software versions that can read your backup media in the future.

Also, you must do regular DR testing. You would be suprised how many companies use a backup strategy for years, and then when disaster strikes they discover that their strategy was inadequate. Once you have done a couple of DR drills you will undoubetedly discover ways to tweak your procedures to improve performance and reliability.

A Finger drive

[hesiod]

We use what we call a "finger drive" (not to be confused with thumb drive). After a catastrophic failure, we are all driven to finger pointing.

Rsync+remote physical server

[Avuton Olrich]

I personally take care of 1.3GB of personal, very important storage (read: porn). Once every two weeks or so I rsync to another physical computer. I don't believe there's any advantage to RAID over this. Sure, if my main server dies I've lost whatever's been collected in the last two weeks, but for me that's really a non-issue.

I don't maintain 1000 Boxes, but ...

[Qbertino]

But anyway I'd actually try to apply my 1-person-shop strategy if I would be maintaining that much.
It may sound crazy for most people but it goes like this:

1) All critical data on central servers. No critical data on workstations, ever.

2) Critical Stuff for MS stored on Unix via Samba (Asuming your using Ethernet and not some Turbo Protokoll I don't know of)

3) A guy responsible for backup including taking this weeks backup home + a standin for him. Both have necessary root access and have specific payd time devoted to maintaining a working backup policy.

4) Automated regular overturning backups (custom shell script) using an PHATT external USB 2/Firewire HDD or, in your case, a few of these (or something simular).

5) A custom polstered Zarges Box or suitcase large enough to carry a backups worth of those around (home/offsite).

Downside: Doesn't use expensive unreliable ancient-technology tape, which, for some bizar and strange reason I really can't fathom, somehow still is the ultimate way to do backups for most people. Ergo: It will be hard to convince management *and* your IT co-workers that this is actually a very good solution.

Upside: Faster, Cheaper, more reliable, easier to recover from, easier to replace/find spare parts and easier to handle than any other solution I know of - or my IT-expert geek friends use for that matter. I have actually managed to recover from backups done that way. And since I've been hearing the some horror stories about tape for almost two decades now - no matter what type of tape - I consider my observation confirmed.

Footnote: I strongly suggest using ext3 as backup filesystem. It's a slowpoke, adding maybe a few hours to your backup, but it's insanely easy to recover from any Unix without having to do panik purchases of strange GUI FS recovery software that require some strange configuration of Win XP + Service Pack 2 and the additional sacrifice of your firstborn child.

Look at your network, do the math and have some shelves built for a few of those external driveboxes. Your critcal stuff can't be more than a 1-digit sum of TBs.

My 2 cents.

Backups? We don't need no steenking backups!

[Mr. Firewall]

Here's what I do when I need to back up:

1. Depress the clutch pedal.
2. Put the gearshift into "Reverse"
3. Slowly let out the clutch pedal while pressing lightly on the accelerator pedal

It works really well, and I can almost always recover from those backups too.

Re: Make sure offsite IS OFFSITE

[Alien54]

Well I thought that the mud would act as a lubricant to the earthquake faults, setting up Earthquake season. Earthquakes then cause more fires. fires burn off the ground cover enabling floods when the rain comes, which creates mud to enable the earthquakes.

So it becomes a nice natural cycle for California.

Riots work well as part of a slightly different cycle.

So you are the guy with the sideburns? excellent.

Although there seem to be earlier mentions of that phrase in various versions in other groups prior to the 1995 rec.humor.funny posting - It's still all quite good.

I don't mean to minimize the quake. I have lived through many, and this was by far the most terrifying. But I don't want people in Israel and elsewhere with loved ones in Los Angeles thinking that there is chaos here. Remember, the four seasons in our part of California are earthquake, fire, flood and drought. This year we had them all, plus a fifth season: riots. But we pull together and get on with our lives.

and the reply

You forgot the sixth season, pestilence.

Re:just what ever you do make sure offsite IS OFFS

[a9db0]

i would suggest minimum different zip codes different time zones would be best

Sounds funny but very true. Backups across town aren't terriby useful if across town is flat too. Sound farfetched? Ask a sysadmin in Miami how far off he ships his backups. If he was there when Andrew visited, I'll bet they're in New Mexico.

This may seem a tad offtopic, but it is relevant:

You have to think through both distance from and access to your backups as a part of disaster recovery planning. Backup isn't just recovering the CEO's email, though that is a (hopefully) far more frequent occurance than recovering from a hurricane/fire/mudslide/blizzard. Easy access to the backup media is important for daily operations. Recovery from disaster is quite a bit more complex. Your backup solution needs to be able to cover the full spectrum - from yestarday's lost spreadsheet to the area flattened by mother nature.

Personally, I keep two backups - one here locally, one 1000 miles away in another state. Backup to CD here, online rsync in NC.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - Variously attributed, frequently to Andrew Tanenbaum

Re:just what ever you do make sure offsite IS OFFS

[Ungrounded Lightning]

I'm not in IT for my company so I only know part of it, from observation mainly.

First: All important files are to be kept on network fileservers - big RAID boxes which keep backups automagically, as configured, as part of their normal operation. All workstations automount them, all home directories are on them, laptops sync to them when on LAN, etc. (There are also "scratch" filesystems for temporary files - build intermediates, chip simulations and their results, etc. These are cheap, fast, and non-redundant Probably fast drives on fast servers. Files there may persist for months but will be lost if a drive fails.) If you want your files preserved you make sure they're on a backed-up fileserver or you lose.

Second: Backup tapes (generated by the fileserver appliances) are periodically transported, by currier in locked box, to an offsite storage facility. (No idea how often or where. Not my problem.)

Re: Make sure offsite IS OFFSITE

[techno-vampire]

Well I thought that the mud would act as a lubricant to the earthquake faults, setting up Earthquake season.

Not unless the mud can seep 5 miles underground or more. And, yes, I'm The Guy With The Sideburns. It's a long story, and doesn't belong here. Glad to see I'm recognized. I'd have used Sideburns as my handle here but it was taken.

ShadowProtect is great for Windows server backups

[Anonymous Coward]

We use ShadowProtect to backup all of our Windows Exchange and SQL servers. It's a hot (no downtime for backups) snapshot-based backup and disaster-recovery product similar to Symatec's V2i/LiveState products.

In the past we used Symantec's LiveState Recovery but we discovered that it consistently produced corrupt backups of Exchange servers. LiveState is basically useless for backing up Exchange. ShadowProtect has no problem making clean backups of Exchange, even under very heavy load, and it's significantly cheaper than LiveState.

The only problem we have experienced with ShadowProtect so far was that during a restore operation we did not have access to some of our drives when we booted the ShadowProtect Recovery Environment CD. However, StorageCraft (makers of ShadowProtect) support was very helpful and made a custom recovery bootable CD for us (at no cost) which resolved our issue.

One thing we found interesting is that we discovered that Symantec's snapshot technology, upon which their product is based, is actually made by StorageCraft. Take a look at the file properties copyright info of pqv2i.sys or symsnap.sys, their snapshot driver, it's copyrighted by StorageCraft. This means that StorageCraft's ShadowProtect is based on the same widely-deployed/tested snapshot technology, albeit the latest-and-greatest version of it as StorageCraft is the actual maker of this technology, not Symantec.

I wouldn't do this...

[cr0sh]

For a small additional fee, you can ensure your tape data is encrypted.

The problem with encrypting backups is that if - on the backup - one bit of data becomes corrupted, the entire backup is likely to be worthless. Since most times when doing a restoration of data, this corruption happens when you need the data most (Murphy's Law), you will come to regret the decision. At least on an unencrypted tape, you can sometimes (with a lot of work) start in the middle of the tape (or other backup medium) and work around the damaged portion, and only lose a bit of the backup.

With an encrypted backup, you could lose it all.

Unless your backup medium is completely foolproof (I know of no such medium), encrypting the backup seems foolhardy, unless you can stand the potential loss and/or the data, should the backup fall into the wrong hands, is valuable enough. Most data isn't this valuable, and the process for backup should take this into account, and only backup the valuable data separately (and possibly encrypted) from the main backup - so that in the event the backup is needed, the entire thing isn't hosed if an errant bit is corrupted.

BTW - I know I have simplified things a tad and an errant bit isn't likely to corrupt an entire backup, but I am using this as an illustration. If you have ever messed around with backups, you know that in the heat of the moment is when things are found to not work as planned (which should make you test your backups before needing them, but I digress) - so you know that what I am talking about is a real possibility...

Re:I wouldn't do this...

[QAPete]

Problem is, if you are in a Sarbanes-Oxley environment, you're going to be forced to encrypt your backup media. Beyond that, you don't want unencrypted, unprotected raw .tar files on tapes that could be lost or stolen, exposing people's personal information or corporate 'secrets'. Too much of that happening these days.

To not encrypt tape backups simply because of the VERY off-chance the backup has one bad byte is kind of silly. We verify our backups at the time we take them, we have multiple copies of backups available at any time, and the risk of one bad byte of data (or 100, for that matter) of putting us in an untenable situation is extremely remote.

I stand by data encryption, especially when it is on media that is outside of your physical control, and especially when it involves personal information or corporate 'secrets'. To not do this is bad advice.

Re:I wouldn't do this...

[Anonymous Coward]

Time to reread Schneier. There are feedback modes in which a small ciphertext error only damages a limited portion of the plaintext. One even permits random access, which IMHO is an absolute must for large or slow archival media.

Re:I wouldn't do this...

[cr0sh]

You have a point about SOX - I just know that despite all best efforts, typically the time you find out your backup is bad is the time you need it most - your typical "Murphy's Law" scenario.

Hmm, I guess what this will mean is the development (probably already exists in some manner) of RAID for tape backup of encrypted data...

The Bittorrent System

[stinkydog]

I zip all my files and name it "Naked pictures of (insert star name here)". Then I publish the torrent. Cheap distributed offsite backup.

SD

a few ground rules

[dJOEK]

Some tips from a pro:

1. Educate users. Let them know in what circumstances what file can be recovered upto when. Don't allow 'grey' backups

2. Don't backup workstations. Use a solution like Norton Ghost or PXE to distribute network booted generic images and have user homes on a WELL CONFIGURED WELL BACKED UP NAS.

3. Don't do incrementals unless you absolutely absolutely have to. differential backups (every change since last _full_ backup) allows you to have multiple copies on tape.

4. use great backup software. I'm a EMC/Legato NetWorker guy myself. don't trust homebrew freshmeat perl scripts. you don't trust your million dollar data to a 35-year old geek living in his parent's garage.

5. Test your restores. Do it often.

6. keep a copy of your critical data offsite as well

7. Have a disaster plan. test it. TEST IT!

eVault software for backups

[Raver32]

I highly recommend a SAN solution for larger backups, using the eVault software product. We switched over about 6 months ago, I can literally restore a corrupt/deleted/cut and pasted elsewhere file within 5 minutes of the user request. No searching for tapes, no swapping and taking backups elsewhere. A fibre connection to another location that houses the server/backup array gives me gigabit connectivity to all of my backups. Each night is a full backup, even for slow backups such as Groupwise. www.evault.com

your .sig

[Medievalist]

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - Variously attributed, frequently to Andrew Tanenbaum

I was in a meeting with the late Dr. John Hendrickson in the 1990s or 80s and file transfer options for moving large files between the Academy of Natural Sciences in Philadelphia and the Benedict Estuarian Research Labs near Washington, D.C. were being discussed. Today, we'd transfer the data constantly over a broadband connection while making local backup archives. Back then we were running processing and data gathering in batches, and we were running all our voice and data traffic between the sites on a single fractional T1.

John, who was a biostatistician and the finest mathematician I have ever known, turned to me and asked what the capacity of a 9-inch reel tape was (I don't remember, I do remember we were getting 6250 bpi though). He then calculated in his head the rough bandwith of his car (because he knew how big the cargo area was and how long it took to drive to Benedict) and discovered that the fastest and most economical way for us to get the job done was shipping tapes. I was willing to take his word for it - he did that kind of thing all the time - but some of the other scientists had him explain the calculation anyway.

What John said at the time (I don't really know if it was orginal to him, but it was the first time I'd heard it) was "never underestimate the bandwidth of a station wagon full of mag tapes". No hurtling involved, he was a cautious driver.

Commvault is pretty impressive

[Medievalist]

I use a table-driven script calling rsync --link-dest onto coraid aoe racks, then archive offsite to LTO3. I back up everything nightly.

But the guys here who wanted to buy a product, rather than build a solution, spent months researching all the alternatives and they even got demo hardware and software and trialed the majors on site. Their finding was that Comvault knocks the doors off everything out there for really large volumes of data on multiple operating systems. Veritas and Legato were among the ones they trialed, I don't remember any more details (sorry!). We've tried Time Navigator and Arkeia and Retrospect in the past, none of those scaled for us.

We now run both systems (commvault and my custom one) so that we have a backup system and a backup backup system.