Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Seppuku?
tasks(723) drafts(105) languages(484) examples(29106)
Just curious... do most people still use *tape* for backup? Personally, I use multiple hard drives and a DVD burner on a daily basis.
I don't respond to AC's.
But maybe that's just me, someone who opted not to work in government after studying political science.
For that kind of money, I'd probably just send the HD to data recovery specialists.
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
I'm going to transform myself into a mighty hawk. Either that or I'll just go and work at Dixons, haven't decided yet.
...print will never be dead.
Looking for a Rails developer in Chapel Hill?
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
c++;
If you're going to back up a file (actually a set of files) that is worth that much, wouldn't it be smart to go a bit further than keeping a backup copy on magnetic media?! Maybe in more than one place too?!
The original generic sig.
So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
As their IT consultant I stand by my use of Maxtor drives.
spoonerize "magic trackpad"
Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?
The information was worth $38 billion, but they rescanned it at a cost of only $200K?
Sounds like RIAA accounting to me.
Wtf?!? They call to ms and dell, but not to people who actually recover data? My first call would been to http://www.ibas.com/
ps. I don't work at ibas
"Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable..."
Ok, so you go to the primary hard drive and make another backup.
Another question, doesn't anyone test their backup systems?
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
RIAA / MPAA mathematicians
I'm a consultant - I convert gibberish into cash-flow.
Because no one ever restores them regularly to test them.
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
1) No surprise reading this:
Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.
2) Always make sure the backup really works.
3) Better procedures are needed if a single tech can reformat both hard drives in the same session
4) Much better hardware and software are needed for data worth $38 billion.
5) Paper backups are a good last resort and as a check on data integrity
Sounds like the file was worth $200,000. The account was worth $38B.
I'd be fired and leave with $30 Million. Then I'd move to the Cayman Island, having a sunbath, and asking myself between two cocktails how hard my former employees and my former state have been working to clean my mess. Well maybe I won't even ask cause I don't give a shit about it.
No matter what they might have lost, it wasn't worth $38 billion. The state gross product of Alaska is only $33 billion, and their tax revenue will only be a fraction of that. And it's not like they lost any of that either, just some files.
What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.
He who lights his taper at mine, receives light without darkening me.
And, I have ot say those numbers were smelly. Not sure where they pulled them from.. but...
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Quo usque tandem abutere, Nimbus, patientia nostra?
This should actually be a best-practice kind of situation really. Accidents happen. Both hard disks got formatted. The tapes were shit. BUT, we had a paper trail. It was painstaking, but it was recoverable. EXCELLENT! This is exactly why banks have paper records of everything, and why they pay a LOT of money to have them properly stored.
That said... the excerpt is a bit misleading. The data was worth $38 billion. They didn't lose $38 billion. They managed to get it back in shape for $200,000, which is not pennies, but probably well worth the effort.
I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.
--
make install -not war
The cost of the loss was the $200,000 to rescan the documents plus the cost of not having those documents online for part of the past 9 months, plus any related costs.
Think of it this way:
If you lose a $38 billion dollar check in a fire, but it only costs you $100 to get the bank to re-issue the check, your loss is $100 plus a few days' interest. A few million dollars in interest is a far cry from $38 billion.
If they'd shredded the boxes they'd be in real trouble.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
But I'll admit that about 6 years ago, when moving from one old laptop to another, I accidentally erased all of the data from my main and my backup Jaz disks before ensuring that every piece of data from 6th grade through my first year of college was backed up onto new media. It wasn't that I didn't have a good strategy for backup: it was adequate for my needs at the time. It's not that I didn't know I shouldn't format disks with important stuff on them: I figured I had transferred it already. At age 18, I didn't have the money for recovery services, nor did I have much of a printed record of anything.
What happens when you fuck up that big? Take it like a man, and live with the shitty consequences. Know that there's nobody to blame but yourself, but learn from that catastrophe so it never happens again.
That they cuold get recovery for only 200K.
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
The Kruger Dunning explains most post on
Why were the drives being formatted anyway? You don't need to do such things with some OSes.
It reads: Now imagine wiping out a disk drive containing information for an account worth $38 billion (29 billion).
Thats "For an account worth"....not...."files worth"
What is the big deal, $200K is nothing compared to $38BN. Besides, the accident created (temporary) jobs.
Lets say you have a small software development company. You have 6 employees that cost you on average of $90k per year each (including taxes, 401k match, health care, salary, office space, etc...) Now lets say that it takes you 2 years to build your application. Over those two years your team spent 20% of the time in meetings, 15% of the time goofing off, 10% of the time debugging, 20% of the time designing/planning, 5% of the time training, and the rest (25%) actually making progress on code.
Your investment for that final product is roughly $1.08 million dollars.
Now, imagine the day before you are sending the code off to the press to go gold, you lose everything. Luckily, you retain all of your staff, and they are all very familiar with the project. At this point, they can start working on recoding the entire project. No more meetings, no more design decisions, no more planning, no more training, and less goofing off. You can now re-create that final product in about 35% of the total time.
Your opportune cost to replace the final product is roughly $378,000.
So yes, the file could have absolutely been worth $38 Billion, yet only cost $200,000 to recreate.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
To err is human, to really f-up requires root.
From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone."
Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.
It's not about the backup, it's about the restore.
If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.
obviously no deficiencies vs. no obvious deficiencies
Where I work, we finally phased out our tape backup drives last year in our server room. A product called the "CRU Dataport" is a very nice removable, hot-swap hard drive frame and carrier assembly. You just install their frame in an external 5.25" drive bay, and buy a set of carriers for it. Install suitable SATA drive drives in each carrier, and switch them out nightly just like backup tapes. Backup software like "Backup Exec" can still be used, but it will treat each cartridge just like it did the backup tapes.
Tape still has a few advantages though. For one thing, they're less fragile than a hard drive. They're also less valuable to the average employee than a SATA hard drive, so there's less worry of one disappearing if you give it to someone to take off-site regularly.
But all in all, we like the switch. Backups complete in less time, and it's faster restoring selected files. (No rewinding or tape re tensioning needed, etc.)
..Alaska.
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Alaskans....
Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).
Storm
Is it just my observation, or are there way too many stupid people in the world?
Having worked with people from "statewide" (what they call statewide IT in Alaska), this is a minor screwup when compared to what is possible there. They should thank the tech for reminding them that they should back up data just like 50% of other government organizations do.
Actually there are some well-intentioned people in government IT in Alaska, but because the work is tricky (everything in Alaska is quirky), and the pay is poor, and the cost of living is relatively high, most of the good people take off for the 48 sooner or later, where their ability to do tricky work is better compensated.
Ontrack Easy Recovery
alright, it's proprietary, so for us open source evangelists it is evil
but this data recovery tool does wonders...
The MAFIAA is a bunch of mindless jerks who will be the first up against the wall when the revolution comes
costs to reissue a check?
$100? sure... try losing an unemployment check some time... paperwork out the fanny.
every day http://en.wikipedia.org/wiki/Special:Random
At least they did not deleted the pr0n stuff
- This can't be... - Be what? Be real?
Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
... is how the tech told the boss. I can't even imagining how I would explain that big of a "Oops!" to my boss and he is relatively tolerant of mistakes.
how about sixish?
http://www.disc-storage.com/nsm/index.htm
every day http://en.wikipedia.org/wiki/Special:Random
Surely they could just have outsourced the job to India for $2,000?
So, if you're already manning an outpost in Alaska, and you screw up, where do you get banished to from there?
Even with a reformat, if it is the same file system, as long as you didn't start overwriting the data, then data recovery is usually not an issue with the correct software. Overwriting by formatting with a different file system usually requires the more expensive file recovery option. Even then, spending a few thousand dollars to recover 38 BILLION dollars worth of data is probably justifiable
If you then made the disk a linux swap partition, you might be hosed. They don't say what the file system was. Even so it is hard to imagine that they couldn't do a data recovery on it. On the other hand, making people rescan the data is possibly a good exercise on why we do backups.
"It is a greater offense to steal men's labor, than their clothes"
I know this is Slashdot, not a reputable source of information, but the headline and write-up are so sensational that the editor of the National Enquirer is feeling uncomfortable just reading it. (Yeah, they read /. They read EVERYTHING!)
Seriously, they lost data that had to be rescanned into the computer.
Total cost of mistake: $220,700
(including $71,800 for consultants!)
Hmmmm... Seems a bit less than $38 Billion.
By this logic if you lose a file that belongs to one of the large brokerages, the data loss could be in the Trillions!!!
Can I claim losing one of my personal files as a certain percentage of my total life-long income? If so that scan of Her with the grits that my stupid little brother deleted would be worth tens of thousands of dollars, at least!
.. assuming that only a quick-format has been done.
Might even be able to recover from a full format, though I've not had experience with that..
An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
Fortunately this is the government we're talking about so it's not a big deal.. If this was a software company, or studio there'd be bloody hell to pay.
I don't own a snook, and if I did I wouldn't leave it cocked.
They've been overwriting the same set of backup tapes since 1983? I'd be surprised if it was a dedicated tape system -- sounds like it's some cheap tape drive sitting on someone's system. Sounds like this mistake will cost them more than a pretty hefty storage solutions from one of the big companies that specialize in that sort of thing. Think they'll go shopping around once they're done scanning all that data back in? I'd be a bit surprised if they do...
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
The data on the drive should be recoverable with the right tools...none of which are expensive. There are even some companies that specialize in recovering data from this kind of loss. I have done it several times for customers, it's no biggy if you know how to deal with it.
"...a civilian some of the time, a soldier part of the time and a patriot all of the time." -Brig. Gen. James Drain
Use DAT tapes for near-line storage.
Breakfast served all day!
Although to be fair, we've not had much need for tape recovery for quite some time - filesystem snapshots are really really good for 'user has been a muppet and deleted his excel file'.
"staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Sounds like the data was worth $200,000...
I often have trouble dealing with business leads that can't seem to determine the value of their data. I ask them "What would the Russian Mafia pay for it?"...
Are NOT restore systems. Restoration is a totally different operation, and not one that the backup solutions people invest a lot of effort in. You doubt me? Try to restore an Exchange Server from tapes (Backupexec) after losing and re-building the server.
This is a classic case of government incompetance. Its also an example of why we test DR plans. A very simple DR test would have most likely spotted the bad tapes. That and a decent tape rotation system, which they obviously did not have since they were forced to recall the paper and re-input.
Former Revenue Commissioner Bill Corbus said no one was ever blamed for the incident.
"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," Corbus said.
Yeah, that pretty much sums up working for the government.
They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.
Did they bother to look on the P2P networks for a copy of their data files? You can still find just about anything there . . .
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
Why wouldn't you simply spend a couple grand on sending the drives to somebody like Ontrack? Several years ago one of the techs at a firm I worked for did the same thing to an accounting firms server. Our insurance paid for the Ontrack recovery. During the process I got a mini-tour of one of their facilities. These guys are seriously hard core about recovery. If it was just a simple format they could have probably handled it via a remote connection.
All the money used to pay labor to recreate the files comes right out of our PFD fund, which means we'll get paid less for some id10T who can't tell what hard drive he's working on.
Any of you IT admin work on more than one HD at a time? How many times have you pooched the wrong drive?
There are many up here in Alaska who believe this was intentional, as the PFD has been under attack for years by those working in the government who want to keep the money and use it for stupid things like installing round-a-bouts. The one at the University sucks! And people have voiced their concern, yet the gov says they will install more!
The money comes from the oil revenues generated, which is supposed to be divided up among the residents of Alaska. Ours last year was about $1200, and it was supposed to go up this year. Hopefully this years has already been calculated and locked in.
But if next year the amount is lower because of this goof, I can tell you that someone in Alaska will put a death threat on this poor saps tail. You don't mess with a sourdough's pfd!
L8r
They depended solely on TAPE backup? ... sigh ... they aren't alone in that. But it's time proven really, really bad idea. If I had a dollar for every tape drive that I have encountered that was apparently working but was either creating unreadable tapes, or was omitting important files because of misconfiguration, I'd be a wealthy man. Tapes are fine for "we'll try to get it back, but we may not be able to" type storage. And there is a valid need for that kind of storage. But for the only copy of really important data?
Sure, other backup media can have problems also, but IMO tapes never have been and probably never will be reliable enough to be the sole backup media for anything really important.
And where, pray tell, was the off-site backup? Billions of dollars worth of Acoounts Payable data, and they don't have a copy off site? What is their plan if there is a fire, flood, explosion, etc?
You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
My corp has no (that's zero) client data retention policies. That leave lil' ol' me justifying the backups and archives for the eight years wort of client files. Here's what I have to cover my backside:
1) Hourly disk snap shots for the fat-finger mistakes
2) Weekly full tape bakups on a three week schedule for disaster recovery
3) Nightly incremental tape backups to keep the weekly backups up to date
4) Tape archive system to hold all of the old files that would otherwise be deleted after the job is finished (can't have that)
5) Double copies of the archive tapes - Just In Case
6) Off-site storage of all backups and duplicate archives - Just In Case
7) Weekly test of backups BEFORE those are shipped off-site.
8) Clause in my contract saying I can't be fired if this system fails - I really wish that were true.
Most of this stuff was standard issue, out of the box, flip the switch and use the defaults.
Why is this so hard for a government agency to do the same?
Here will be an old abusing of God's patience and the king's English.
Sure, a decade ago people needed things like zip disks and tape drives, but I don't know anyone who puts any faith in those things in todays world. Tapes rarely work for restores and the degradation times are not nearly what people claim them to be. Also, have you ever tried digging for something in a mountain of tapes?
At my current employer we backup out databases each night to several locations. The first are a couple of terra servers (network drives) next to the server that have a few terrabytes worth of storage a piece. We also ftp the files to an off-site backup as well. The files end up getting pasted to another server as well that is used for both maintaining the backup files as well as a point for development and testing of applications on a current structure and data. Going this route is far more dependable then trusting a glorified VHS with our data (yeah yeah they are encoded and read differently).
Besides, how many times have you opened a tape and had it bad before you even get files on it? That never served to comfort me.
how much would have been lost?
$38Bn
therefore the file was worth $38Bn.
Creating a new $30,000 car costs less than $10,000. the car is still worth $30,000 on the profit sheet.
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
After this story, I guess I should stop whining about the 2 months worth of files I lost yesterday when I mentioned to our tech support guy that my computer was running super slow. I returned from a meeting to see Ghost wiping my hard drive. (Argh! I used to be the tech guy elsewhere, but now I'm busy teaching.) Apparently I missed that memo that went out 6 months before I joined stating that files should only be stored on the school file server, not locally. (And this guy was miffed because he noticed that caps lock and insert - the two keys which are the scourge of my existence -were removed from my keyboard. )
Morals of the story.
- Back up the day before you are expecting data loss.
- Data loss can originate from the *strangest* places.
- Off site backups are mandatory - even for personal work.
- Don't assume that someone is competent just because they have a title.
So, does anyone have a good recommendation for easy off site backup. Ideally, any file I save to my documents or a mapped drive gets uploaded to some server and archived. I'm trying OmniDrive. Flash drives are an alternative but I'd prefer something simpler (all that plugging) and harder to lose. Linux (home) and Windows (work) compatible a plus. Free is good too.
-Jon
They probably based that number on the number of various criminal groups involved in identity theft and how much they could potentially sell the data for.
----- If communism is a system where the government owns business, what do you call a system where business owns govern
A real BOFH doesn't TEST, we do live restores frequently enough for our dumbass users that we don't need to test. And with a blocksize of one, maybe two if we like you.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Greaaat I am currently in negotiations for a contract with Alaska. 200k out of their pockets is going to make contract negotiation a little more difficult :/
The video is funny.
http://www.backuptrauma.com/
Though, using rsync to backup to rotating partitions works as well.
On odd days, rsync to B1
On even days, rsync to B2
On odd weeks, rsync to B3
On even weeks, rsync to B4
On odd months, rsync to B5
On even months, rsync to B6
On every two odd months, rsync to R7
On every two even months, rsync to B8
On every odd year, rsync to B9
On every even year, rsync to B10
So with 10x the space, you can have easy instant access to:
a day or two ago.
a week or two ago.
a month or two ago
4 or 8 months ago.
a year or two ago.
There's no reason to format unless you're changing filesystems. Just install to a new directory and clean out the rest by hand if you want to. Looks like he learned this lesson the hard way.
This is why people persist in storing paper in addition to electronic copies. Most people know how to assess the risks of loss/destruction of stored paper. They understand how to put a process in place that minimizes the significant risks.
To the layman (i.e., suits), risk assessment and mitigation in electronic storage is like magic. How many management levels up do you suppose the first guy was who said, "How could something like this just happen?" Two levels? One? Zero?
"We reject as false the choice between our safety and our ideals." --The American President (20.1.2009)
Another job well done by the Bastard Operator From Hell and his trusty bulk eraser.
With Google having many, many terrabytes of data, do they even bother doing backups? Practically speaking, could clusters of redundant servers with large hard drives be a faster, better, more cost-effective solution in lieu of the clunky, time-consuming backup tapes in the long run? It just fascinates me that as the amount of total storage increases, backup tapes seems less and less as practical. What do you think?
Coderz 4 Life
Hi All
I used to do tech support for backup software (ARCserve).
The stories I could tell you about people's bad day's when they did not think
through their backup plans, not to mention the problems CA introduced to ARCserve itself.
I talked with one lady who overwrote her live Lotus Notes database with the backup when she thought she
had restored it to another location. She had not read the page on how to restore it to another location. Once we figured this out, her response was "I think I can resign with dignity".
Another customer had a nice hot-swap RAID array with the nice monitor that shows the drives
stripes. He loved to shows guests how the hot swap worked. He did so at least every other week (yes, this is a bad thing to do). So one day when he decided to show off the array again, the whole thing went down. Well, he thought, I will just restore from backup. Oh wait he never tested his backups. Look
all he ever backed up was 50 meg out of the 60 Gig RAID array (circa 1997). Once he figured this out all he could say was "I'm fired, I'm fired".
Yup, managment always thinks backups are expensive until they loose thier data.
Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
didn't I just see DJ_hacK0rZ on the AOL IRC offer up this same database for $200, or $2000 if it includes credit card *and* social security numbers?
http://www.southwest.com/cgi-bin/buildItinerary2?r ef=bktrvl_gn
I can see Apple using this case in Alaska for blitzing use with Time Machine adds for OS X 10.5. You know it will be the PC guy in a parka or some type Eskimo gear talking about losing some thing important and the Mac guy in a ratty tee shirt laughing at him ask why can't he just have his computer travel back in time to retrieve the file like he's Marty McFly or John Tutor.
"... And all will know that 300 boxes gave their last breath to recover it"
Truer words haven't been said.
This guy is way out there
$38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.
[Insert pithy quote here]
I used to work at a company that did it's own advertising, they had the catalog and all data for the advertising (pictures of models, the full finished running catalog, the full catalog everybody was working on, the financial data to calculate prices for each catalog), was standing in the middle of a department desk, near the entrance, around 12 disks varying from 80G -> 250G (they bought the largest available for each year, so the 80G has been there for 3 years) daisy chained with FireWire to a single PowerMac G4. No backups whatsoever, just copied the data from one disk to another every night, but only for 2 or 3 older disks.
I could buy stock in their competitors, come into their office right before the catalogs have to be printed (1-2 months before the beginning of fashion season), either steal (nobody would notice, they make too much noise for anyone to sit close by) or destroy the disks (they smoke up if you slightly force the power adapter in at a 45 degree angle) and wait for them to go out of business and my stock to rise.
The data on that is worth millions if not billions, almost $500 million per year in sales alone.
Custom electronics and digital signage for your business: www.evcircuits.com
Mid 90s. A nameless mid-sized ($100 million) company (were I was a short-term contractor) kept all corporate data in a ERP system, from accounting to customer orders and inventory. They had a RAID-5 disk configuration, and faithfully backed up the system nightly with carefully planned tape rotations. The RAID-5 controller failed in a way that destroyed all data unretrievably. Turns out the backup tape had been running off the end for months, but the tape operator ignored the cryptic error message. It was a MAJOR disaster that almost wiped out the company.
It surely is some kind of world record...
-x- Sorry my bad English. I'll have him tarred and feathered. -x-
"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," Corbus said.
Translation: "We were going to fire the technician but then he produced printouts of his written orders and of management's assurances that the current backup scheme was adequate."
Reminds me of my first day on the job a long, long time ago. I wanted to copy the contents of a floppy disk to the hard drive on an HP minicomputer. Unfortunatly, the HP "Copy" command really meant "Replace", so the entire computer hard drive was reformatted as a 5 1/4" floppy.
or x millions in bribe for 'accidents'?
Tough choice.
Maybe the story should be $38billion lost because someone forgot 'mt -f /dev/st0 setblk 0'.
We deal with drives from hundreds of customer offices where we provide thousands of computers for free. A very simple policy makes something like this data loss exceedingly unlikely. We don't take the drive from any computer and reuse it straight away EVER.
The process for every drive that comes in is to label it with:
* The customer's name
* The model of computer (in case we need to boot it)
* The OS (in case we have to attempt recovery)
* The date received
The drives are stored in a secure area and we don't reuse a drive until it has "ripened" for at least 45 days. Drives that are reused are wiped then reimaged. Drives that aren't going to be reused because they are damaged or because they are too small, are destroyed after 6 months with a welder's slag hammer.
(Also you can remove the magnets from the drives and use them to hang things in your cube!)
Gods above man.. the OP asked for feedback when 2-3 became available.
I pointed out 6TB exists today!
every day http://en.wikipedia.org/wiki/Special:Random
Kroll Ontrack
Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone
In state that was formerly Tsarist Russia, state government pays you!
[Insert pithy quote here]
... one time we had a hugely corrupted SQL Server database and I had no good backups.
I had one backup that wasn't that old (a day maybe) but ended up accidentally deleting it when I was trying to setup a system to recover the data... don't ask.
It was a nightmare... an all-nighter but still no DB (this was the entire corporation's PeopleSoft financials DB so all accounting was shutdown on Monday morning), consultants flown in from afar, the works.
Closest backup was 4 days old. The company wrote off the time lost to re-enter all the accounting data. All told it was probably about $25,000 of lost time/labor.
I walked with my head down for about a month. The PeopleSoft programmers brutalized me behind my back. Ah, what fun. Call me crazy but I've been overly paranoid about backups ever since.
Some little dinky departmental servers can get by with NAS and/or FTP backup. But when you need to backup a few TB every single night, FTP starts to become very rediculous, very quickly.
Yes, for low-end backup of relatively small amounts of data, tape makes no sense. Quality drives cost a fortune, and keeping track of what is on which tape is a real pain for just a few total TB.
But for many users of modern tape, a backup solution with a total capacity of just few TB is a mere rounding error. Heck, modern tapes can usually store 1.5 to 2 TB per tape. (Depends on compressiblity, which is done by the drive itself.) It is not uncommon to see a libary full of a few dozen drives and several hundred of those tapes. (This bloat is largely due to expanded archival requirements, and the current regulatory environment.)
And yes, tape is quite reliable, and has been reliable for decades. Yeah, if you try and archive on some consumer-quality QIC/Travan floppy-tape piece of crap, you are gonna lose data. (This is why nobody sells tape drives to consumers anymore.) But enterprise tape is a different animal altogether, and is far more reliable long-term than hard drives. (Especially for off-site long-term archival.) I have customers that still read reel-to-reel tapes, even today.
SirWired
People wonder why banks etc. are required by law to keep paper copies of transactions securely stored for years. This is why. Similar debate as to paper record for voting machines....
If guy who touched the computer didn't get fired, then it's a true crime. He should have to pay the $200k too.
As several have pointed out before, if they spent $200,000 fixing the mistake, it wasn't a file worth $38 billion, it was a file worth $200,000. An essential file is exactly worth what it costs to replace it.
This is not a pedantic point. This a catastrophic failure of people's ability to understand the magnitude of numbers. How can a contradiction of this magnitude make front page without anybody realizing its absurdity? 38 000 000 000 / 200 000 = 190 000. This story is like saying to your boss, "I accidentally destroyed a piece of our inventory worth $190,000, but don't worry, 'cause I walked across the street and bought an identical replacement from a vending machine for one dollar." If your sense of magnitude is properly callibrated, this story should become 190,000 times less interesting when you realize this mistake. The worthiness of this story should shift from being a footnote in a highschool newspaper about someone stealing a dollar from a locker to a New York Times story about a CEO fleeing the country with his company's retirement fund. This is the difference between a guy getting stabbed in midtown Manhattan and a small city getting flooded with nerve gas. And yet, no distinction is made. 38 billion and 200 thousand are, after all, pretty much the same number -- it's just that one has a few more zeroes for your eyes to skip over.
To put it as nicely as possible: You are all fools, and your innumeracy will be the downfall of civilization.
Though I do agree with you, I'd say the bigger debate would be whether or not CEO x or CEO y in fact fucked up in the first place. Same with a possible sacrificial lamb of an IT person.
In my home country, most people spend only 5 min to file the tax returns (fill the form) and receive the refund in less than 2 weeks.
Log in to your online bank, select the proper link to local "IRS", make sure the numbers are OK on the form, approve some realestate loan info (prepared by the bank and ready to use) and you are done! No W-whateverforms and other BS I had to deal with in US.
Welcome to Estonia!
PS! Do you even have to pay taxes in US? I think not!
A DMB I worked with a few years back was trying some sort of fancy trick working on the production database. Well he did something wrong then realized he did it wrong and all you could here was "fuuuuuuuuuuuck". The poor guy was beet red and shaking when I walked over. Couldn't get the development database to work for the production. Couldn't get the backups to come up. I think he stayed up for like 2 days. In the end we ended up replicating the original installation. I have no idea why the development setup wouldn't go over. The company owner put some absurd price tag on the amount of money he thought it cost him. Something like $250,000. I thought that was strange since all he lost was stats that were never used.
For some reason I refuse to use either spell check or the spacebar properly.
OneCare.
parasight.de
You know, you can just send the hard drive to a data recovery center and let the Pros handle it. It may cost a couple grand, but when weighing that with $38 bazillion dallars, the cost seems overwhelmingly acceptable.
What a great time to pick up your copy of the Hichhiker's Guide to the Galaxy and figure out where to relocate!
Stop the brainwash
Correction: should be OJ's wife's killer, not OJ's killer. Sorry.
Table-ized A.I.
I think this guy gets 2 days time out in the low oxygen server room.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
After all Paul Bremer was awarded one after having only lost $12 Billion
True story. Sadly. So one day whilst I was an undergrad computer science student, the department sysadmin decided it would be a good day to back up the student directories. There wasn't enough free space on the hard drive, so he decided to clear some up. He found some random file that seemed to be a good candidate for deletion, as it was a) rather large and b) he couldn't quite figure out what it really was. So he canned it. And then when he went to copy the student directories.... oops! Where did they go? It just so happens that this "mystery" file was actually the student directories. Needless to say, much irritation ensued. I suppose it is one thing to lose all of your department's students' files, and its another thing to lose them all while trying to back them up, but to do so by intentionally deleting them without knowing what they were. That, my friends, takes a very, very special sysadmin.
With "routine" reformatting, you know the system in question was the very best Dell and M$ can provide. I've never lost a file since going to gnu/Linux. Never, and that includes the survivors transferred from the Windoze word dating back to 1989. These jokers managed to lose a file in less than a year because they had to do a "routine" reformat of a machine.
Friends don't help friends install M$ junk.
Typical thoughts, if you can have them over the rushing sound in your ears are, "My career is over," and "I hate Microsoft/Dell." If his boss made him use the system as is, he's hating his boss too but both of them are going to be scapegoated by billion dollar companies.
The root cause of the accident was system instability. The article talks about "routine" reformatting. No stable system needs a routine reformat. Storing the data onto another partition may have helped, but that's just a mitigation strategy for a situation that should never have happened. It's possible they had put their trust and data into a hardware RAID they did not and could not understand. Finally, the tape backup should have been a sufficient mitigating strategy but was not. Blame for that can only be laid at the feet of the people who sold the system and said that it worked. You have to wonder if Dell/M$ picked up the tab for this or if they charged consulting time for the failure of their systems.
Friends don't help friends install M$ junk.
It was not the tech's fault, unless you believe reformatting is routine. The system failed the tech as did a reasonable mitigation strategy. They thought they had the data in three places, but it was not in any.
Friends don't help friends install M$ junk.
I should have guessed it.
So, M$/Dell made $71,800 on their systems failure that had already cost the people of Alaska $150,000. I wonder how many minutes of overtime Bill Gates himself charged.
Friends don't help friends install M$ junk.
I confess I didn't RTFA, but this seemed pretty clear in the writeup -- the data loss was NOT in the billions.
"Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
I.e., the loss cost them 200K. And a huge headache, of course, but how much Advil did they have to buy to cost billions?
UFB.
The higher the technology, the sharper that two-edged sword.
I don't go through a very large volume of LTO2 tapes but I have had a few I can't read - most likely from the same source. IBM3480,3490,3590 tapes of a wide range of vintages have been OK. I've had only a couple of problems with a lot of 8mm cartridges and still managed to get the data in the worst case (three days to read a tape!). Various types of DLT tapes have had problems, including SDLT tapes that are DOA - can't even write to them once. Another company reads in the reels of 20+ year old stuff for me so I don't know what happens there but I think it requires treating the tapes before reading.
As the leader, you are responsible for everything that happens below you. That is why you get paid so much, right? Of it it just the pretty face?
Seriously, that's what's wrong with corporate America (well, the world). I happen to be a professional engineer. I run a small company (a corporation). Because I am a licenced professional, the corporation does nothing to shield me from liability if something goes wrong with my designs. If one of my employees screws up and it goes out the door, it's my fault. I go to jail and/or get sued if someone dies. Personal responsibility. I don't care where you are in te chain of command, when somebody underneath you fucks up, it's your fault. If the fuck up happened four layers below you, then there are four fuckups in the chain, but you're still to blame. Even if you never met the IT peon who made the error, you hired the guy who hired the guy who hired the guy who screwed up.
If boards and CxOs were held accountable for all the really big screwups, they'd be a little more concerned about the organization and a little less about maintaining that single digit handicap.
Is it just my observation, or are there way too many stupid people in the world?
They should publicly flog this person. Either that or have him hear Kas's reason as to why the PS3 costs $600...
_____
"If I wanted to hear this recording, I would have pressed a button to hear it!"
It would have cost *far* less if they'd paid for an unformat - what's the worst case, $10k?
mark
It's amazing anyone still using it.
http://www.high-rely.com/index.cfm?action=article
Sudden Disruption
--
Sudden View...
the radical option for editing text
http://www.sudden.net/
http://suddendisruption.blogspot.com/