Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Seppuku?
tasks(723) drafts(105) languages(484) examples(29106)
For that kind of money, I'd probably just send the HD to data recovery specialists.
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
I'm going to transform myself into a mighty hawk. Either that or I'll just go and work at Dixons, haven't decided yet.
...print will never be dead.
Looking for a Rails developer in Chapel Hill?
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
c++;
Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.
BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.
So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
As their IT consultant I stand by my use of Maxtor drives.
spoonerize "magic trackpad"
Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.
Read the best of all of Slash: seenonslash.com
Because no one ever restores them regularly to test them.
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
Tape works great if you are willing to spend big bucks for top-class hardware. Unfortunately, most people try to get by with the cheap stuff, which is very unreliable. Try to explain to a manager why you need a $50K tape system to backup a $10K server. Computers have gotten very cheap, high-quality tape transports haven't.
Mea navis aericumbens anguillis abundat
And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...
Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Quo usque tandem abutere, Nimbus, patientia nostra?
Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
Easy to offsite???
I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.
And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.
--
make install -not war
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
That they cuold get recovery for only 200K.
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
The Kruger Dunning explains most post on
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
The Alaska Permanent Fund is not tax funded at all. Technically, it's not even part of the state government.
;)
At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.
They passed $30 billion last year, the news story would indicate it's gone up a bit since then.
Please report to your nearest Microsoft customer reeducation camp.
Mea navis aericumbens anguillis abundat
Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).
Storm
Is it just my observation, or are there way too many stupid people in the world?
Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
If it weren't for deadlines, nothing would be late.
I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.
Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
1) Write "200 000 000 000 USD" on the paper. .txt file /. how big financial disaster you've made, and how you've saved your ass
2) Type what's on the paper into a
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the
No sig today.
Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
Not even close. We use LTO2 tapes and keep them offsite for 18 months. We've run several test recoveries on tapes > 12 months old with success. Some of our tapes have been in circulation for about 24 - 30 months now and are still writing without difficulty. For restoration purposes, the actual media is rarely the problem. Changes in encryption passwords (with a poorly documented history), files in use, and lost/orphaned files are the most common reasons for restoration failure.
An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
Get the person with the purse strings to go through the 'cost of downtime' calculation.
Lead them throught it, point out all the lovely parts like contractual obligations (engineering companies tend to need to keep designs for long long periods of time) or 'regulations' (Sarbanes-Oxley has a lot to answer for).
Add in the cost of x many people not working for a week.
Include the 'well, can our business still function if we lose our customer database'.
And if that really doesn't work, then clearly your last resort is artificially induced panic, where you raise the possibility of 'something important' being gone, and unrecoverable. Payroll records are a good example, as that's a personal terrror as well as a 'problem for the company'.
Are NOT restore systems. Restoration is a totally different operation, and not one that the backup solutions people invest a lot of effort in. You doubt me? Try to restore an Exchange Server from tapes (Backupexec) after losing and re-building the server.
They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.
HA is not the same as DR
I can have a simple HA cluster that involves two nodes attached to a single disk array, all sitting in the same rack somewhere. Take a guess what happens when the power for the building goes out?
HA is nice, but will do nothing for you in the event of a disaster.
You can structure your site so that you get both, but doing so requires a lot more work (stretched clusters and SAN's spread over miles) and you have to be careful that you dont trash your performance while you are at it. (real time replication over distance involves latency, and you have to be careful about what that will do to your app)
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
riiiiight...wink, wink, nudge, nudge :-)
your denial is beneath you, and thanks to the use of hallucinogenic drugs...i see through you - another dead hero
;rolleyes:
There really isn't anything "Insightful" about pointing out a grammar error. Making personal insults isn't either.
C'mon mods, this is just embarrassing.
Try not to take me more seriously than I take myself.
Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Continuity planning can be complicated based on the environment, and quite often overlooked until the first time it is needed.
Few companies maintain sufficient hardware of their own for true disaster recovery purposes. In most cases like that, the organization will have redundant data centers that are probably used in a load sharing model. Hopefully one data center can carry the full load of the critical activities if the other goes down. For these organizations, backup tapes are really intended for a complete disaster at all sites that would require acquiring new hardware.
Other organizations have agreements with firms like Sunguard and IBM for cold sites. These vendors guarantee a certain square footage in their own data centers. They then work with the client to understand the exact hardware and software requirements that would be needed in case the cold site needs to go hot. In these instances, the tapes are shipped to the Sunguard or IBM site and loaded on machines as quickly as possible. The contracts normally give the vendor a minimum amount of time to stand up hardware and load the software and data, governed a great deal by how much data needs loading.
Just a note, if your company is deciding on going the hosted DR route, make sure before hand that you have agreement from your software vendors that your license allows you to load their software outside your organization. I worked at one company that didn't have that in their original software contracts and had to spend more money with the software vendors when they created a DR plan. Many software vendors won't mention this little detail.
Most often I've seen backup tapes used when for example an important database table was dropped accidentally. The last good backup tape was loaded and the database completely restored to get back to production. This is what you'd think of as single system disaster recovery.
4 Node (2 active and 2 passive) cluster attached to continuous replicating SANs. 2 nodes (1 active and 1 passive) & 1 SAN onsite, 2 nodes & the other SAN in a remote side, CoLo, or hot site. That's our basic design for critical applications. Active nodes provide network load balancing while the passive nodes allow us failover potential. Granted the cost for such a solution is extremely high, but in an enterprise environment where 3 days of downtime cost more than an entire year of housing and bandwidth the cost definitely justifies the cost. One of the nice features of having remote locations is that you can essentially drop nodes in them and use them as hot sites. They are already housing dedicated bandwidth to the central office, so that aspect becomes a non-issue. Additionally, that configuration allows our hot site to be a true DR location for offsite testing and rebuilding boxes and services. Again, this solution is not cost effective for all businesses, and your assessment of the HA design you've given is fully accurate. There's an element of better availability, but the architecture leaves a lot to be desired for many enterprise concerns. Ultimately, both HA and DR are financial concerns that dictate what a company can afford to spend... and that ultimately determines architecture.
About a month or 2 back, a slip of the fingers turned my root filesystem into a linux swap partition.
Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.
I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.
On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.
The living have better things to do than to continue hating the dead.
I guess it depends on what you're backing up...
Of all our clients, the smallest backup we've got is about 14 GB. That's too big for a single DVD-R, but it fits just fine on a DDS3. We can also easily automate a tape backup - just instruct a secretary or someone to swap the tape in the morning. Tapes are reasonably durable too...more rugged, in general, than a removable HDD. Of course, that's all for the little backups...
Some of our clients are backing up 400+ GB of data on a daily basis. I guess you could use some kind of removable HDD...or go through a stack of DVDs every day... Or you could just use a single LTO-3 tape.
And that's just our clients. We don't have any monstrously huge backups to deal with. Some places have literally TerraBytes of data to back up... While I'm sure a good amount of that goes into some kind of RAIDed SAN/NAS...a robotic tape library starts making a lot more sense than a pile of HDDs or DVDs.
"Work is the curse of the drinking classes." -Oscar Wilde
$38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.
[Insert pithy quote here]
No, you _CANT_.
For that, this "scratch" has to be a massive gash, which goes more than halfway through the depth of the plastic disc. Additionally, it has to span the entire radius of the disk just to make it "difficult" to recover a significant portion of the data.
Your backup solution should NOT involve throwing a bunch of bare DVD-Rs on the dash board of your car.
Any backup solution involves climate control, and light-proof packaging. Your tapes would crumble in no time, otherwise.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
7 tracks, 1/2 inch wide tape.
Mea navis aericumbens anguillis abundat
"The tapes are about $100, which is close to par with hard drives now (400 GB native), but in the not too distant future, LTO4 will be out, which doubles the capacity."
Yeah, but when LTO4 is out you can't use them without buying a new _expensive_ LTO4 drive.
Whereas in the not too distant future when new hard drives with double the capacity are out, you can still use them with your existing computers (as long as they still support SATA).
Basically HDDs = media + drive, and they are about the same price as tapes on a per GB basis if not cheaper. Multiple HDDs have better bandwidth than multiple tapes with one tape drive.
And I've heard horror stories where backup tapes can only be read by the same drive the backups were made on.
When you factor all that in, tape isn't that great, it's still better in some areas, but it should be cheaper for all its disadvantages.
A file restore is as different from a full system restore as an engine bench test is different from a full rocket launch.
As an IT auditor, I do ding IT shops when they don't do full system restores (which has the dual benefits of verifying that the techs are capable, and verifying that the media is readable). I'm going to be printing out this story and showing it to people who don't do full system restores... I get along fine with BOFHs, and I can sympathize with them about the burden of SOX, but while I'm doing the audit, I don't let them slide on this.
Read the best of all of Slash: seenonslash.com
Yes you will, or should, when the user's manager gets involved. If you would like to think that because a user trashes a month's accounts, that you can wave some magic hand and say "Yes, I know that data is in pristine condition on last week's backup, but no, you're not getting it just because Waldo over there is as dumb as dogshit to have trashed it in the first place", you either work in some kind of hell hole out of Dilbert (think "I am Mordac, the Preventer of Information Services"), or have been interpreting your job description far too literally. (We wonder why so many people have so much distate for certain IT people.) However,
Should the user's department be charged the cost that your DR service bills for the retrieval of said tape? Absolutely.
Should this retrieval work be prioritized accordingly within your task list? Absolutely.