So You've Lost a $38 Billion File

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

Comments

Re:Tapes?

[MindStalker]

Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.

BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.

Re:Tapes?

[greginnj]

Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.

Re:Tapes?

[Sobrique]

In a backup system where you're taking full backups once a week, even with no data churn at all, you end up backing 1/7th of your estate every night. Starts getting a real challenge at about the 15Tb mark, and becomes a whole new adventure in pain when you're talking 100-200Tb.

And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.

But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.

Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.

Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.

inject + extract

[mengel]

Every night before backups add a file to partitions being backed up (like '.backuptst.txt') with the date in it.

After you run the backup, memove then restore that file, make sure it has the current date in it.

I've had that as a feature in my backup scripts for over 10 years...

Re:Someone is trying to cover their ass

[AK Marc]

It sounds more like the reality is that none of these ever existed and someone got caught-out.

Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.