Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Seppuku?
tasks(723) drafts(105) languages(484) examples(29106)
For that kind of money, I'd probably just send the HD to data recovery specialists.
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
I'm going to transform myself into a mighty hawk. Either that or I'll just go and work at Dixons, haven't decided yet.
...print will never be dead.
Looking for a Rails developer in Chapel Hill?
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
c++;
Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.
BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.
So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
As their IT consultant I stand by my use of Maxtor drives.
spoonerize "magic trackpad"
Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.
Read the best of all of Slash: seenonslash.com
Because no one ever restores them regularly to test them.
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...
Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Quo usque tandem abutere, Nimbus, patientia nostra?
Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
Easy to offsite???
I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.
And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
That they cuold get recovery for only 200K.
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
The Kruger Dunning explains most post on
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Is it just my observation, or are there way too many stupid people in the world?
Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
If it weren't for deadlines, nothing would be late.
I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.
Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
1) Write "200 000 000 000 USD" on the paper. .txt file /. how big financial disaster you've made, and how you've saved your ass
2) Type what's on the paper into a
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the
No sig today.
Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
Not even close. We use LTO2 tapes and keep them offsite for 18 months. We've run several test recoveries on tapes > 12 months old with success. Some of our tapes have been in circulation for about 24 - 30 months now and are still writing without difficulty. For restoration purposes, the actual media is rarely the problem. Changes in encryption passwords (with a poorly documented history), files in use, and lost/orphaned files are the most common reasons for restoration failure.
An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
About a month or 2 back, a slip of the fingers turned my root filesystem into a linux swap partition.
Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.
I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.
On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.
The living have better things to do than to continue hating the dead.
$38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.
[Insert pithy quote here]