Slashdot Mirror
So You've Lost a $38 Billion File
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Seppuku?
tasks(723) drafts(105) languages(484) examples(29106)
Just curious... do most people still use *tape* for backup? Personally, I use multiple hard drives and a DVD burner on a daily basis.
I don't respond to AC's.
But maybe that's just me, someone who opted not to work in government after studying political science.
For that kind of money, I'd probably just send the HD to data recovery specialists.
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
I'm going to transform myself into a mighty hawk. Either that or I'll just go and work at Dixons, haven't decided yet.
...print will never be dead.
Looking for a Rails developer in Chapel Hill?
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
c++;
If you're going to back up a file (actually a set of files) that is worth that much, wouldn't it be smart to go a bit further than keeping a backup copy on magnetic media?! Maybe in more than one place too?!
The original generic sig.
So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
As their IT consultant I stand by my use of Maxtor drives.
spoonerize "magic trackpad"
"Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable..."
Ok, so you go to the primary hard drive and make another backup.
Another question, doesn't anyone test their backup systems?
Because no one ever restores them regularly to test them.
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
1) No surprise reading this:
Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.
2) Always make sure the backup really works.
3) Better procedures are needed if a single tech can reformat both hard drives in the same session
4) Much better hardware and software are needed for data worth $38 billion.
5) Paper backups are a good last resort and as a check on data integrity
Sounds like the file was worth $200,000. The account was worth $38B.
No matter what they might have lost, it wasn't worth $38 billion. The state gross product of Alaska is only $33 billion, and their tax revenue will only be a fraction of that. And it's not like they lost any of that either, just some files.
What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.
He who lights his taper at mine, receives light without darkening me.
And, I have ot say those numbers were smelly. Not sure where they pulled them from.. but...
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Quo usque tandem abutere, Nimbus, patientia nostra?
This should actually be a best-practice kind of situation really. Accidents happen. Both hard disks got formatted. The tapes were shit. BUT, we had a paper trail. It was painstaking, but it was recoverable. EXCELLENT! This is exactly why banks have paper records of everything, and why they pay a LOT of money to have them properly stored.
That said... the excerpt is a bit misleading. The data was worth $38 billion. They didn't lose $38 billion. They managed to get it back in shape for $200,000, which is not pennies, but probably well worth the effort.
I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.
--
make install -not war
The cost of the loss was the $200,000 to rescan the documents plus the cost of not having those documents online for part of the past 9 months, plus any related costs.
Think of it this way:
If you lose a $38 billion dollar check in a fire, but it only costs you $100 to get the bank to re-issue the check, your loss is $100 plus a few days' interest. A few million dollars in interest is a far cry from $38 billion.
If they'd shredded the boxes they'd be in real trouble.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Well, let's say it's all of Alaska's tax income data for the 2006 year. Of course it's not impossible to recreate the data, but since it's basically electronic money, you could consider it "lost" if only temporarily.
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
But I'll admit that about 6 years ago, when moving from one old laptop to another, I accidentally erased all of the data from my main and my backup Jaz disks before ensuring that every piece of data from 6th grade through my first year of college was backed up onto new media. It wasn't that I didn't have a good strategy for backup: it was adequate for my needs at the time. It's not that I didn't know I shouldn't format disks with important stuff on them: I figured I had transferred it already. At age 18, I didn't have the money for recovery services, nor did I have much of a printed record of anything.
What happens when you fuck up that big? Take it like a man, and live with the shitty consequences. Know that there's nobody to blame but yourself, but learn from that catastrophe so it never happens again.
That they cuold get recovery for only 200K.
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
The Kruger Dunning explains most post on
Lets say you have a small software development company. You have 6 employees that cost you on average of $90k per year each (including taxes, 401k match, health care, salary, office space, etc...) Now lets say that it takes you 2 years to build your application. Over those two years your team spent 20% of the time in meetings, 15% of the time goofing off, 10% of the time debugging, 20% of the time designing/planning, 5% of the time training, and the rest (25%) actually making progress on code.
Your investment for that final product is roughly $1.08 million dollars.
Now, imagine the day before you are sending the code off to the press to go gold, you lose everything. Luckily, you retain all of your staff, and they are all very familiar with the project. At this point, they can start working on recoding the entire project. No more meetings, no more design decisions, no more planning, no more training, and less goofing off. You can now re-create that final product in about 35% of the total time.
Your opportune cost to replace the final product is roughly $378,000.
So yes, the file could have absolutely been worth $38 Billion, yet only cost $200,000 to recreate.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
To err is human, to really f-up requires root.
From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone."
Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.
It's not about the backup, it's about the restore.
If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.
obviously no deficiencies vs. no obvious deficiencies
Where I work, we finally phased out our tape backup drives last year in our server room. A product called the "CRU Dataport" is a very nice removable, hot-swap hard drive frame and carrier assembly. You just install their frame in an external 5.25" drive bay, and buy a set of carriers for it. Install suitable SATA drive drives in each carrier, and switch them out nightly just like backup tapes. Backup software like "Backup Exec" can still be used, but it will treat each cartridge just like it did the backup tapes.
Tape still has a few advantages though. For one thing, they're less fragile than a hard drive. They're also less valuable to the average employee than a SATA hard drive, so there's less worry of one disappearing if you give it to someone to take off-site regularly.
But all in all, we like the switch. Backups complete in less time, and it's faster restoring selected files. (No rewinding or tape re tensioning needed, etc.)
..Alaska.
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Alaskans....
Please report to your nearest Microsoft customer reeducation camp.
Mea navis aericumbens anguillis abundat
Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).
Storm
Is it just my observation, or are there way too many stupid people in the world?
I wouldn't be surprised if the formatted drives were used for other purposes before they realized their mistake. I don't think the data recovery firms can restore info that has been overwritten, can they?
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
If it weren't for deadlines, nothing would be late.
I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.
Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
Having worked with people from "statewide" (what they call statewide IT in Alaska), this is a minor screwup when compared to what is possible there. They should thank the tech for reminding them that they should back up data just like 50% of other government organizations do.
Actually there are some well-intentioned people in government IT in Alaska, but because the work is tricky (everything in Alaska is quirky), and the pay is poor, and the cost of living is relatively high, most of the good people take off for the 48 sooner or later, where their ability to do tricky work is better compensated.
Ontrack Easy Recovery
alright, it's proprietary, so for us open source evangelists it is evil
but this data recovery tool does wonders...
The MAFIAA is a bunch of mindless jerks who will be the first up against the wall when the revolution comes
costs to reissue a check?
$100? sure... try losing an unemployment check some time... paperwork out the fanny.
every day http://en.wikipedia.org/wiki/Special:Random
1) Write "200 000 000 000 USD" on the paper. .txt file /. how big financial disaster you've made, and how you've saved your ass
2) Type what's on the paper into a
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the
No sig today.
Hey. From the article, consultants from Microsoft and Dell were called in. As they said in Raiders of the Lost Ark, top...men.
Read the best of all of Slash: seenonslash.com
Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
... is how the tech told the boss. I can't even imagining how I would explain that big of a "Oops!" to my boss and he is relatively tolerant of mistakes.
how about sixish?
http://www.disc-storage.com/nsm/index.htm
every day http://en.wikipedia.org/wiki/Special:Random
With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.
No no no! You don't understand. It says clearly that consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data. Who needs data recovery experts after that? Sure, data recovery expert can retrieve data from disks that have been submerged in acid for three weeks, but hey, this is a Microsoft consultant!
Yes, I am the one with the legendary sig.
I assume government agencies are required to wipe hard drives before releasing them to be junked, given to another dept, sold, etc. The info on those drives was full of very personal info (birth certificate images, etc).
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
So, if you're already manning an outpost in Alaska, and you screw up, where do you get banished to from there?
Even with a reformat, if it is the same file system, as long as you didn't start overwriting the data, then data recovery is usually not an issue with the correct software. Overwriting by formatting with a different file system usually requires the more expensive file recovery option. Even then, spending a few thousand dollars to recover 38 BILLION dollars worth of data is probably justifiable
If you then made the disk a linux swap partition, you might be hosed. They don't say what the file system was. Even so it is hard to imagine that they couldn't do a data recovery on it. On the other hand, making people rescan the data is possibly a good exercise on why we do backups.
"It is a greater offense to steal men's labor, than their clothes"
After they formated the drive they did a clean install of the OS over it. That makes it a tad more difficult to recover the data.
.. assuming that only a quick-format has been done.
Might even be able to recover from a full format, though I've not had experience with that..
An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
They've been overwriting the same set of backup tapes since 1983? I'd be surprised if it was a dedicated tape system -- sounds like it's some cheap tape drive sitting on someone's system. Sounds like this mistake will cost them more than a pretty hefty storage solutions from one of the big companies that specialize in that sort of thing. Think they'll go shopping around once they're done scanning all that data back in? I'd be a bit surprised if they do...
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Use DAT tapes for near-line storage.
Breakfast served all day!
Although to be fair, we've not had much need for tape recovery for quite some time - filesystem snapshots are really really good for 'user has been a muppet and deleted his excel file'.
Are NOT restore systems. Restoration is a totally different operation, and not one that the backup solutions people invest a lot of effort in. You doubt me? Try to restore an Exchange Server from tapes (Backupexec) after losing and re-building the server.
This is a classic case of government incompetance. Its also an example of why we test DR plans. A very simple DR test would have most likely spotted the bad tapes. That and a decent tape rotation system, which they obviously did not have since they were forced to recall the paper and re-input.
Former Revenue Commissioner Bill Corbus said no one was ever blamed for the incident.
"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," Corbus said.
Yeah, that pretty much sums up working for the government.
They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.
Did they bother to look on the P2P networks for a copy of their data files? You can still find just about anything there . . .
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
Why wouldn't you simply spend a couple grand on sending the drives to somebody like Ontrack? Several years ago one of the techs at a firm I worked for did the same thing to an accounting firms server. Our insurance paid for the Ontrack recovery. During the process I got a mini-tour of one of their facilities. These guys are seriously hard core about recovery. If it was just a simple format they could have probably handled it via a remote connection.
All the money used to pay labor to recreate the files comes right out of our PFD fund, which means we'll get paid less for some id10T who can't tell what hard drive he's working on.
Any of you IT admin work on more than one HD at a time? How many times have you pooched the wrong drive?
There are many up here in Alaska who believe this was intentional, as the PFD has been under attack for years by those working in the government who want to keep the money and use it for stupid things like installing round-a-bouts. The one at the University sucks! And people have voiced their concern, yet the gov says they will install more!
The money comes from the oil revenues generated, which is supposed to be divided up among the residents of Alaska. Ours last year was about $1200, and it was supposed to go up this year. Hopefully this years has already been calculated and locked in.
But if next year the amount is lower because of this goof, I can tell you that someone in Alaska will put a death threat on this poor saps tail. You don't mess with a sourdough's pfd!
L8r
They depended solely on TAPE backup? ... sigh ... they aren't alone in that. But it's time proven really, really bad idea. If I had a dollar for every tape drive that I have encountered that was apparently working but was either creating unreadable tapes, or was omitting important files because of misconfiguration, I'd be a wealthy man. Tapes are fine for "we'll try to get it back, but we may not be able to" type storage. And there is a valid need for that kind of storage. But for the only copy of really important data?
Sure, other backup media can have problems also, but IMO tapes never have been and probably never will be reliable enough to be the sole backup media for anything really important.
And where, pray tell, was the off-site backup? Billions of dollars worth of Acoounts Payable data, and they don't have a copy off site? What is their plan if there is a fire, flood, explosion, etc?
You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
My corp has no (that's zero) client data retention policies. That leave lil' ol' me justifying the backups and archives for the eight years wort of client files. Here's what I have to cover my backside:
1) Hourly disk snap shots for the fat-finger mistakes
2) Weekly full tape bakups on a three week schedule for disaster recovery
3) Nightly incremental tape backups to keep the weekly backups up to date
4) Tape archive system to hold all of the old files that would otherwise be deleted after the job is finished (can't have that)
5) Double copies of the archive tapes - Just In Case
6) Off-site storage of all backups and duplicate archives - Just In Case
7) Weekly test of backups BEFORE those are shipped off-site.
8) Clause in my contract saying I can't be fired if this system fails - I really wish that were true.
Most of this stuff was standard issue, out of the box, flip the switch and use the defaults.
Why is this so hard for a government agency to do the same?
Here will be an old abusing of God's patience and the king's English.
Sure, a decade ago people needed things like zip disks and tape drives, but I don't know anyone who puts any faith in those things in todays world. Tapes rarely work for restores and the degradation times are not nearly what people claim them to be. Also, have you ever tried digging for something in a mountain of tapes?
At my current employer we backup out databases each night to several locations. The first are a couple of terra servers (network drives) next to the server that have a few terrabytes worth of storage a piece. We also ftp the files to an off-site backup as well. The files end up getting pasted to another server as well that is used for both maintaining the backup files as well as a point for development and testing of applications on a current structure and data. Going this route is far more dependable then trusting a glorified VHS with our data (yeah yeah they are encoded and read differently).
Besides, how many times have you opened a tape and had it bad before you even get files on it? That never served to comfort me.
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
They probably based that number on the number of various criminal groups involved in identity theft and how much they could potentially sell the data for.
----- If communism is a system where the government owns business, what do you call a system where business owns govern
A real BOFH doesn't TEST, we do live restores frequently enough for our dumbass users that we don't need to test. And with a blocksize of one, maybe two if we like you.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
The video is funny.
http://www.backuptrauma.com/
Though, using rsync to backup to rotating partitions works as well.
On odd days, rsync to B1
On even days, rsync to B2
On odd weeks, rsync to B3
On even weeks, rsync to B4
On odd months, rsync to B5
On even months, rsync to B6
On every two odd months, rsync to R7
On every two even months, rsync to B8
On every odd year, rsync to B9
On every even year, rsync to B10
So with 10x the space, you can have easy instant access to:
a day or two ago.
a week or two ago.
a month or two ago
4 or 8 months ago.
a year or two ago.
There's no reason to format unless you're changing filesystems. Just install to a new directory and clean out the rest by hand if you want to. Looks like he learned this lesson the hard way.
This is why people persist in storing paper in addition to electronic copies. Most people know how to assess the risks of loss/destruction of stored paper. They understand how to put a process in place that minimizes the significant risks.
To the layman (i.e., suits), risk assessment and mitigation in electronic storage is like magic. How many management levels up do you suppose the first guy was who said, "How could something like this just happen?" Two levels? One? Zero?
"We reject as false the choice between our safety and our ideals." --The American President (20.1.2009)
With Google having many, many terrabytes of data, do they even bother doing backups? Practically speaking, could clusters of redundant servers with large hard drives be a faster, better, more cost-effective solution in lieu of the clunky, time-consuming backup tapes in the long run? It just fascinates me that as the amount of total storage increases, backup tapes seems less and less as practical. What do you think?
Coderz 4 Life
Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
http://www.southwest.com/cgi-bin/buildItinerary2?r ef=bktrvl_gn
I can see Apple using this case in Alaska for blitzing use with Time Machine adds for OS X 10.5. You know it will be the PC guy in a parka or some type Eskimo gear talking about losing some thing important and the Mac guy in a ratty tee shirt laughing at him ask why can't he just have his computer travel back in time to retrieve the file like he's Marty McFly or John Tutor.
"... And all will know that 300 boxes gave their last breath to recover it"
Truer words haven't been said.
This guy is way out there
$38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.
[Insert pithy quote here]
I used to work at a company that did it's own advertising, they had the catalog and all data for the advertising (pictures of models, the full finished running catalog, the full catalog everybody was working on, the financial data to calculate prices for each catalog), was standing in the middle of a department desk, near the entrance, around 12 disks varying from 80G -> 250G (they bought the largest available for each year, so the 80G has been there for 3 years) daisy chained with FireWire to a single PowerMac G4. No backups whatsoever, just copied the data from one disk to another every night, but only for 2 or 3 older disks.
I could buy stock in their competitors, come into their office right before the catalogs have to be printed (1-2 months before the beginning of fashion season), either steal (nobody would notice, they make too much noise for anyone to sit close by) or destroy the disks (they smoke up if you slightly force the power adapter in at a 45 degree angle) and wait for them to go out of business and my stock to rise.
The data on that is worth millions if not billions, almost $500 million per year in sales alone.
Custom electronics and digital signage for your business: www.evcircuits.com
Mid 90s. A nameless mid-sized ($100 million) company (were I was a short-term contractor) kept all corporate data in a ERP system, from accounting to customer orders and inventory. They had a RAID-5 disk configuration, and faithfully backed up the system nightly with carefully planned tape rotations. The RAID-5 controller failed in a way that destroyed all data unretrievably. Turns out the backup tape had been running off the end for months, but the tape operator ignored the cryptic error message. It was a MAJOR disaster that almost wiped out the company.
Yea, nothing like sending your citizen's personal information oversees. Sounds like a hell of a plan.
That's nothing. Once, I wrote "$500 trillion" on a notepad file, and then accidentally deleted it. Boy was my face red.
"When the atomic bomb goes off there's devastation...but when the atomic bong goes off there's celebraaaaation!"
"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," Corbus said.
Translation: "We were going to fire the technician but then he produced printouts of his written orders and of management's assurances that the current backup scheme was adequate."
Reminds me of my first day on the job a long, long time ago. I wanted to copy the contents of a floppy disk to the hard drive on an HP minicomputer. Unfortunatly, the HP "Copy" command really meant "Replace", so the entire computer hard drive was reformatted as a 5 1/4" floppy.
Maybe the story should be $38billion lost because someone forgot 'mt -f /dev/st0 setblk 0'.
We deal with drives from hundreds of customer offices where we provide thousands of computers for free. A very simple policy makes something like this data loss exceedingly unlikely. We don't take the drive from any computer and reuse it straight away EVER.
The process for every drive that comes in is to label it with:
* The customer's name
* The model of computer (in case we need to boot it)
* The OS (in case we have to attempt recovery)
* The date received
The drives are stored in a secure area and we don't reuse a drive until it has "ripened" for at least 45 days. Drives that are reused are wiped then reimaged. Drives that aren't going to be reused because they are damaged or because they are too small, are destroyed after 6 months with a welder's slag hammer.
(Also you can remove the magnets from the drives and use them to hang things in your cube!)
Gods above man.. the OP asked for feedback when 2-3 became available.
I pointed out 6TB exists today!
every day http://en.wikipedia.org/wiki/Special:Random
Data overwritten once or twice may be recovered by subtracting what is expected to be read from a storage location from what is actually read. Data which is overwritten an arbitrarily large number of times can still be recovered provided that the new data isn't written to the same location as the original data. For this reason it is effectively impossible to sanitise storage locations by simple overwriting them, no matter how many overwrite passes are made or what data patterns are written. However by using the relatively simple methods presented in this paper the task of an attacker can be made significantly more difficult, if not prohibitively expensive.
Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone
In state that was formerly Tsarist Russia, state government pays you!
[Insert pithy quote here]
Of course. You don't take any chances when you are defrauding a $38 billion account. I'd be watching the departments personal bank records if I were the police.
... one time we had a hugely corrupted SQL Server database and I had no good backups.
I had one backup that wasn't that old (a day maybe) but ended up accidentally deleting it when I was trying to setup a system to recover the data... don't ask.
It was a nightmare... an all-nighter but still no DB (this was the entire corporation's PeopleSoft financials DB so all accounting was shutdown on Monday morning), consultants flown in from afar, the works.
Closest backup was 4 days old. The company wrote off the time lost to re-enter all the accounting data. All told it was probably about $25,000 of lost time/labor.
I walked with my head down for about a month. The PeopleSoft programmers brutalized me behind my back. Ah, what fun. Call me crazy but I've been overly paranoid about backups ever since.
Some little dinky departmental servers can get by with NAS and/or FTP backup. But when you need to backup a few TB every single night, FTP starts to become very rediculous, very quickly.
Yes, for low-end backup of relatively small amounts of data, tape makes no sense. Quality drives cost a fortune, and keeping track of what is on which tape is a real pain for just a few total TB.
But for many users of modern tape, a backup solution with a total capacity of just few TB is a mere rounding error. Heck, modern tapes can usually store 1.5 to 2 TB per tape. (Depends on compressiblity, which is done by the drive itself.) It is not uncommon to see a libary full of a few dozen drives and several hundred of those tapes. (This bloat is largely due to expanded archival requirements, and the current regulatory environment.)
And yes, tape is quite reliable, and has been reliable for decades. Yeah, if you try and archive on some consumer-quality QIC/Travan floppy-tape piece of crap, you are gonna lose data. (This is why nobody sells tape drives to consumers anymore.) But enterprise tape is a different animal altogether, and is far more reliable long-term than hard drives. (Especially for off-site long-term archival.) I have customers that still read reel-to-reel tapes, even today.
SirWired
People wonder why banks etc. are required by law to keep paper copies of transactions securely stored for years. This is why. Similar debate as to paper record for voting machines....
If guy who touched the computer didn't get fired, then it's a true crime. He should have to pay the $200k too.
Though I do agree with you, I'd say the bigger debate would be whether or not CEO x or CEO y in fact fucked up in the first place. Same with a possible sacrificial lamb of an IT person.
> Yea, nothing like sending your citizen's personal information oversees. Sounds like a hell of a plan.
It's a much better idea to bring the overseas workers in on H1-B visas instead!
Or, there's no way an underpaid American worker would compromise the data for some extra cash on the side. Only ferreners would do that! Terrorists!
My other car is first.
A DMB I worked with a few years back was trying some sort of fancy trick working on the production database. Well he did something wrong then realized he did it wrong and all you could here was "fuuuuuuuuuuuck". The poor guy was beet red and shaking when I walked over. Couldn't get the development database to work for the production. Couldn't get the backups to come up. I think he stayed up for like 2 days. In the end we ended up replicating the original installation. I have no idea why the development setup wouldn't go over. The company owner put some absurd price tag on the amount of money he thought it cost him. Something like $250,000. I thought that was strange since all he lost was stats that were never used.
For some reason I refuse to use either spell check or the spacebar properly.
OneCare.
parasight.de
My first call would been to http://www.ibas.com/
Indeed, very odd. During a visit (with my class) at Ibas, they said they would have little to no problem recovering almost all of the data even after several reformat/rewrite cycles. And while Ibas ain't cheap, I'm pretty sure Ibas would be an order of magnitude or two cheaper than rescanning all the documents, not to mention faster.
What a great time to pick up your copy of the Hichhiker's Guide to the Galaxy and figure out where to relocate!
Stop the brainwash
Correction: should be OJ's wife's killer, not OJ's killer. Sorry.
Table-ized A.I.
I think this guy gets 2 days time out in the low oxygen server room.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
True story. Sadly. So one day whilst I was an undergrad computer science student, the department sysadmin decided it would be a good day to back up the student directories. There wasn't enough free space on the hard drive, so he decided to clear some up. He found some random file that seemed to be a good candidate for deletion, as it was a) rather large and b) he couldn't quite figure out what it really was. So he canned it. And then when he went to copy the student directories.... oops! Where did they go? It just so happens that this "mystery" file was actually the student directories. Needless to say, much irritation ensued. I suppose it is one thing to lose all of your department's students' files, and its another thing to lose them all while trying to back them up, but to do so by intentionally deleting them without knowing what they were. That, my friends, takes a very, very special sysadmin.
With "routine" reformatting, you know the system in question was the very best Dell and M$ can provide. I've never lost a file since going to gnu/Linux. Never, and that includes the survivors transferred from the Windoze word dating back to 1989. These jokers managed to lose a file in less than a year because they had to do a "routine" reformat of a machine.
Friends don't help friends install M$ junk.
Typical thoughts, if you can have them over the rushing sound in your ears are, "My career is over," and "I hate Microsoft/Dell." If his boss made him use the system as is, he's hating his boss too but both of them are going to be scapegoated by billion dollar companies.
The root cause of the accident was system instability. The article talks about "routine" reformatting. No stable system needs a routine reformat. Storing the data onto another partition may have helped, but that's just a mitigation strategy for a situation that should never have happened. It's possible they had put their trust and data into a hardware RAID they did not and could not understand. Finally, the tape backup should have been a sufficient mitigating strategy but was not. Blame for that can only be laid at the feet of the people who sold the system and said that it worked. You have to wonder if Dell/M$ picked up the tab for this or if they charged consulting time for the failure of their systems.
Friends don't help friends install M$ junk.
It was not the tech's fault, unless you believe reformatting is routine. The system failed the tech as did a reasonable mitigation strategy. They thought they had the data in three places, but it was not in any.
Friends don't help friends install M$ junk.
I should have guessed it.
So, M$/Dell made $71,800 on their systems failure that had already cost the people of Alaska $150,000. I wonder how many minutes of overtime Bill Gates himself charged.
Friends don't help friends install M$ junk.
I confess I didn't RTFA, but this seemed pretty clear in the writeup -- the data loss was NOT in the billions.
"Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
I.e., the loss cost them 200K. And a huge headache, of course, but how much Advil did they have to buy to cost billions?
8) ...???
9) Profit!
Caution: May contain nuts.
UFB.
The higher the technology, the sharper that two-edged sword.
I don't go through a very large volume of LTO2 tapes but I have had a few I can't read - most likely from the same source. IBM3480,3490,3590 tapes of a wide range of vintages have been OK. I've had only a couple of problems with a lot of 8mm cartridges and still managed to get the data in the worst case (three days to read a tape!). Various types of DLT tapes have had problems, including SDLT tapes that are DOA - can't even write to them once. Another company reads in the reels of 20+ year old stuff for me so I don't know what happens there but I think it requires treating the tapes before reading.
As the leader, you are responsible for everything that happens below you. That is why you get paid so much, right? Of it it just the pretty face?
Seriously, that's what's wrong with corporate America (well, the world). I happen to be a professional engineer. I run a small company (a corporation). Because I am a licenced professional, the corporation does nothing to shield me from liability if something goes wrong with my designs. If one of my employees screws up and it goes out the door, it's my fault. I go to jail and/or get sued if someone dies. Personal responsibility. I don't care where you are in te chain of command, when somebody underneath you fucks up, it's your fault. If the fuck up happened four layers below you, then there are four fuckups in the chain, but you're still to blame. Even if you never met the IT peon who made the error, you hired the guy who hired the guy who hired the guy who screwed up.
If boards and CxOs were held accountable for all the really big screwups, they'd be a little more concerned about the organization and a little less about maintaining that single digit handicap.
Is it just my observation, or are there way too many stupid people in the world?
They should publicly flog this person. Either that or have him hear Kas's reason as to why the PS3 costs $600...
_____
"If I wanted to hear this recording, I would have pressed a button to hear it!"
It would have cost *far* less if they'd paid for an unformat - what's the worst case, $10k?
mark
It's amazing anyone still using it.
http://www.high-rely.com/index.cfm?action=article
Sudden Disruption
--
Sudden View...
the radical option for editing text
http://www.sudden.net/
http://suddendisruption.blogspot.com/