Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.
BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
right - the account is worth 38 billion - the file was apparently worth about 200 grand in labor. of course it didn't cost that much to make the first time, as it was done over a longer period without all the o.t.
It's hard to believe that's how Micronians are made. Why don't we see it right now by having you both kiss one another?
Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.
Read the best of all of Slash: seenonslash.com
Sounds like the file was worth $200,000. The account was worth $38B.
That they cuold get recovery for only 200K.
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
The Kruger Dunning explains most post on
The Alaska Permanent Fund is not tax funded at all. Technically, it's not even part of the state government.
;)
At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.
They passed $30 billion last year, the news story would indicate it's gone up a bit since then.
Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).
Storm
Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
If it weren't for deadlines, nothing would be late.
For super-critical, but not super-massive, data like this especially. I use a couple of external 250GB hard drives to back up my data fast. Then I take one of the HD's home
You take your companies confidential data and leave it lying around at home? Please tell me you're joking.
Don't look now, your bias is showing. I've seen a lot of systems, *nix and Windows where proper partitioning of drives hasn't happened. Even then the article only states that the drive containing the data was erased; it mentions nothing about the OS being on the same drive. And of courser RAID doesn't help in this, unless you have a very delayed snapshot mirroring system; identical mirrored drives don't help when you delete a file because, guess what, mirrors stay the same, so it's deleted everywhere.
Heck my pathetic excuse for an ISP in the UK managed to blow away 3 months worth of mail and that was on a NetApp, moving to a Sun NAS. Admin stupidity happens on all platforms; pretending it doesn't on your chosen one is more stupid than deleting a crucial file.
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
From the CNN article, they spent about $75,000 on data recovery specialists (who could not recover the data), then spent another $125,000 rescanning all the documents from paper.
It doesn't hurt to be nice.
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
riiiiight...wink, wink, nudge, nudge :-)
your denial is beneath you, and thanks to the use of hallucinogenic drugs...i see through you - another dead hero
I guess it depends on what you're backing up...
Of all our clients, the smallest backup we've got is about 14 GB. That's too big for a single DVD-R, but it fits just fine on a DDS3. We can also easily automate a tape backup - just instruct a secretary or someone to swap the tape in the morning. Tapes are reasonably durable too...more rugged, in general, than a removable HDD. Of course, that's all for the little backups...
Some of our clients are backing up 400+ GB of data on a daily basis. I guess you could use some kind of removable HDD...or go through a stack of DVDs every day... Or you could just use a single LTO-3 tape.
And that's just our clients. We don't have any monstrously huge backups to deal with. Some places have literally TerraBytes of data to back up... While I'm sure a good amount of that goes into some kind of RAIDed SAN/NAS...a robotic tape library starts making a lot more sense than a pile of HDDs or DVDs.
"Work is the curse of the drinking classes." -Oscar Wilde
It sounds more like the reality is that none of these ever existed and someone got caught-out.
Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.
Learn to love Alaska
7 tracks, 1/2 inch wide tape.
Mea navis aericumbens anguillis abundat
Get the UBCD or UBCD4Win for some good tools. Particularly, UBCD4Win includes several freeware and open source tools for file recovery. My favorite happens to be testdisk, followed closely by Restoration. (Make sure, if you use the UBCD4Win, you build these tools into ISO. Just follow the directions at the site, it is real easy.)
You forget to mention that such a tape (just the tape) is also as expensive as a 250Gbyte HD.
Add robots, and it gets seriously expensive for large installations...
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?