Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
...print will never be dead.
Looking for a Rails developer in Chapel Hill?
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Not even close. We use LTO2 tapes and keep them offsite for 18 months. We've run several test recoveries on tapes > 12 months old with success. Some of our tapes have been in circulation for about 24 - 30 months now and are still writing without difficulty. For restoration purposes, the actual media is rarely the problem. Changes in encryption passwords (with a poorly documented history), files in use, and lost/orphaned files are the most common reasons for restoration failure.
Nah, Nothing like that. It isn't really exciting either. I cannot really give any names or anything. But I can generically describe what happened.
A law firm handed me a computer that wouldn't boot. On it was some pictures taken concerning a wrongful death case. It turns out that the pictures were a hindsight and in the middle of fixing it, The task turned from getting the computer to run to getting the pictures from the drive. The drive was failing and was larger then the 137gig 28bit LBA limits. But we didn't know this because it was never booted and XP pre SP1 did not enable 48bit addressing by default. And even after SP1, if you didn't update your ATAPI driver to x.1135 or later, it wouldn't be enabled by default even if you have the ability. So connecting it to another computer made it worse. Eventually the fault in the drive which was a crashed head, made it impossible for us to recover past the boot sector running traditional recovery software. The data recovery specialist were able to get around everything we added to the problem as well as the problem itself and retrieved better then 98% of everything on the drive. I think one file was bad but we weren't concerned with it at all.
Long story short, the pictures showed someone's negligence in a wrongful death case and once they were presented or added to the evidence pile, the defendant's insurance company settled for 2 mill. The lawsuit was for more then that so you could probably guess what it could have been worth. The firms cut was in the area of 40% from what I understand. So it was worth 40% of 2 mill to them. $2500 seems like a little amount in comparison.
It has to be worth more than $200k (or else the would have just written off the file). I agree that the info isn't worth the $38B though. The account would still be there even if those transactions weren't. That said I thought all the states learned the lesson of 9/11 of remote offsite backups especially in a state as geologically active as Alaska. Hope the warehouse with the paper isn't near the data center...
Degaussing scares the bad magnetism out of the monitor and fills it with good karma.
Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
About a month or 2 back, a slip of the fingers turned my root filesystem into a linux swap partition.
Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.
I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.
On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.
The living have better things to do than to continue hating the dead.