Slashdot Mirror
Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
Hermes: What do we do when we break somebody's window?
Dwight: Pay for it?
Hermes: Heavens, no! We apologize! With nice, cheap words.
I am officially gone from
The Maryland/VA/DC metro area is really starting to go down hill, from an infrastructure standpoint. Things are just falling apart around here...oh, and what's that? Instead of investing in fixing aging infrastructure, they instead are spending billions to build the ICC? Oh, and what's that? It's STILL going to be a toll road?
I've lived in Montgomery County my whole life, but I'm quickly getting tired of this place -_-;;
Living With a Nerd
Oh no, their contract specifies that they get paid one million dollars for each time they say they are sorry!
Pretty weak apology if you ask me. I guess these military contractors are used to the "boys will be boys" pat on the hand they get from the Department of Defense when they screw up there.
Blar.
I have a project in a separate NG hosted dataspace, in Virginia. They are killing us with incompetence and their sub contractors are worse then they are. We are still trying to get things certified and they won't provide us information about their hosting. We think they have us on virtual servers that belong to another project, and the reason they don't want to tell us anything is that it would reveal they are in breach, since we are paying for dedicated servers.
Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong fix.
To be fair, there is no evidence that Northrop is doing worse than anyone else would have done. We are talking about an enormously complex IT infrastructure here (or so I assume, since it is a government network), and this is not exactly a uniquely bad failure. A week may seem extreme, but I have seen smaller scale systems go down for that long.
I am not an apologist for Northrop, I am just saying that this is not exactly one-of-a-kind incompetence.
Palm trees and 8
The "old business man discovering the internet" IT bubble culture is still alive and well in the defense industry. They have such a bad track record with networking technology it borders on scary. Transformation comes to mind quickly and they keep repeating the same mistakes.
I don't know how many years on this Earth I got left. I'm going to get real weird with it. - Frank Reynolds
Fresh out of school with my CS degree I went to work on a project for my employer that involved partnering and working directly with folks from NG. The original deadline for shipping the solution was something like 6 months after I started. The complete and utter incompetence of the NG side of things wound up stretching this out more than 18 MONTHS longer, and the final delivery lacked a lot of the original stated requirements.
Being the newbie to the whole corporate culture, I was shocked that people were not bothered at all by blown deadlines, missed estimates, and huge cost overruns. Shortly after the project finally delivered, I bailed to work for a much smaller company (fewer than 10 employees) where I discovered that I really love the smaller, more dynamic environment that only small companies can provide.
Working for huge corporations just sucks.
I work with a fair number of ngc.com; I'm a contractor myself. At one point, I had a few interviews for Northrop and I'm so glad they over looked my talents because they seem to abuse the talent they have.
Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.
See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).
I only point out Northrop because while all government contracting is essentially this contract+play model, Northrop has a reputation of placing people on furlough much more often than other companies such as CACI, Raytheon, General Atomics etc. Some Northrop employees seem to live the lives similar to actors and actresses in Hollywood, and I'm not talking about Tom Hanks acting, but maybe those actors that get little spots from time to time on your sitcoms. They literally live in apartments, and wait for the phone to ring day after day. Northrop employees seem to wake up in the morning, wondering if they'll still have a job at the end of the day.
What does this observation have to do with the op? Well, it seems that moral and motivation might be a bit low on a large scale at Northrop, so such blunders are no surprise to me.
Kinda like brother Bluto in Animal House when he smashes the guitar, then looks at the shocked people and goes, "Heh, sorry."
It was EMC storage that failed:
http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/
"A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.
Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."
"Northrop's $2.6 billion service contract with Virginia's government..."
What could they possibly be doing for Virginia that should cost $2.6 billion?
It lead to the Great Pr0n Outage of '10.
Lechers had to actually go outdoors and look at live, clothed people for a change.
A feeling of having made the same mistake before: Deja Foobar
That "they'll fix it so it never happens again."
I used to work in IT and I hated it whenever some manager said that to a customer. Things fail, they can fail multiple times and different failures can have the same results so the statement that "it'll never happen again" would be a bold faced lie.
"Northrop Grumman Says 'I'm Sorry' For Virginia IT Outrage"
Yeah, them Indians, stealing all our jaaaaabs and doing crappy work on top of it!
Oh, wait! Don't tell me these were American IT staff, locally recruited in the good ol' US of A?? Meh, this has got to be someone else's fault. Americans in IT are simply flawless!
This is happening because all of the best IT talent these days are doing startups or working at cool companies like Google and Facebook. This means that Northrup Grumann is staffing their teams with people that can't get those cooler IT jobs. That's the real cause of this disaster for Virginia.
good thing Northrup Grumman doesn't do anything important, like, say, vital national security support.
oh, wait... .
if this is supposed to be a new economy, how come they still want my old fashioned money?
I recently turned down a offer from NG. I turned them down for a number of reasons but mainly past employee experience. Once I had learned that they unexpectedly cut employee salaries in the past to make numbers that was a very large part of the decision. There where other issues like competence and attitudes, that I expect in govt related positions. They are simply not going to attract top talent with their track record.
can't find my account info sorry.
More background for those interested.
http://www.govtech.com/gt/749378 this is the same state agency which last summer fired the head of the agency because he attempted to make Northrop Grumman comply with its obligations.
For some background on their previous problems try the IEEE.
http://spectrum.ieee.org/riskfactor/computing/it/virginia-information-technologies-agency-believes-in-the-perfect-network-fairy
And yes, this is the same agency which has been help up as an example of the benefits of taking diverse IT services and pulling them into a central cloud.
NG reports increased profits and record bonuses for key executives
Northrop-Grumman just needs to merge with Fairchild and their company initials will be complete - NFG
Go see it yourself here.
On Wednesday, August 25, at approximately 3 p.m., the Commonwealth of Virginia experienced an information technology (IT) infrastructure outage that affected 27 of the Commonwealth's 89 agencies and caused 13 percent of the Commonwealth's file servers to fail. The failure was in the equipment used for data storage, commonly known as a storage area network (SAN). Specifically, the SAN that failed was an EMC DMX-3.
According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than one billion hours of run time. A root cause analysis of the failure is currently being conducted.
Anybody else read this like some middle age guy after "finishing a bit too quickly" and telling his , "I swear honey, this the first time this has ever happened to me..."
So Virginia taxpayers will continue to get screwed, but Northrop Grumman has now extended a reach-around?
"Somebody has to do something. It's just incredibly pathetic it has to be us."
--- Jerry Garcia
I used to work at Lockheed Martin and I worked with a tremendous amount of talented people. The people tended to be a little more risk-averse, career-wise.
The likelihood of working 60 hour weeks is less, which is an attractions. Benefits used to be good. So it just attracts different people.
Except for ending slavery, the Nazis, communism, & securing American independence, war has never solved anything.
Ironically, Microsoft has a case study touting the Virginia DMV project as a success story.
http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2005-Enterprise-Edition/Virginia-Department-of-Motor-Vehicles/Virginia-DMV-Enhances-Decisions-Boosts-Safety-Through-Integration-with-Other-Agencies/4000004307
Look at me! I'm a "Systems Engineer", or "IT Engineer" or "MCSE"...I want the fancy "Engineer" title, but I don't want to be held financially accountable, legally acountable, and be held blameless when I do something wrong/stupid/illegal.
I love IT "Engineer" folks who can just get away with "oops" or "i'm sorry" when they piss away hundreds of thousands, or millions of dollars.
That's some fine "Engineering" there...maybe we can design cars and life support systems that way too...
where I come from, you swap out the suspicious or dead junk, and roll the data back on from backups.
this, of course, requires you know how to find dead junk, swap it, and retrieve tapes that you knew how to make, and roll them back on according to your training.
remove all of the above, and it sounds like the error they have.
monkeys cost less, and can do exactly the same job. just sayin'....
if this is supposed to be a new economy, how come they still want my old fashioned money?
maybe they did not test a hardware fail and the fall over back did not take over / was not setup right.
Northrop Grumman was cobbled together over several years from about 1999-2006 from Northrop, Grumman, TRW and several other players. It is so dysfunctional because it is composed of so many competing units that don't operate like a single company. In fact, when I briefly worked for them out of college, most of my coworkers were from TRW and hated the idea of being NGC employees.
I might say more...
These agencies like to place the blame on the contract vendor where they can. Often times it's their own failures that they refuse to acknowledge that cause these over-runs, missed deadlines and so on.
Let me tell you, I used to think it was insane that a toilet seat might cost $300 dollars. Now that I've seen how the government contracts at the federal level it seems like that price was probably a bargain. I don't expect governments at the state level to behave any better than the feds either.
I don't know how NG's state sector runs their business but if it's anything like what I've seen in other sectors then VA KNOWS exactly what they bought and APPROVED it before it ever went live. Then again... well, never mind, I don't think I can give my perspective on it.. so read about it for yourself... http://www.wired.com/dangerroom/2010/08/hp-holds-navy-network-hostage/
Getting charged out at $100/hour while getting paid $10-14/hour was depressing, especially since the contracting company did nothing in that case apart from collect and hand out money (insurance, health care etc was my responsibility). When the client (actually former workplace encumbered by a requirement for layoffs and a hiring freeze) looked as if they would be able to employ me directly the contract company took near criminal steps to poison my relationship with that company. I left after being blamed for a quite few things that happened long before I'd even heard of the place and decided it was easier to work in a different industry.
Many places screw over both their clients and their employees.
I've found the root problem!
From NG's own site: "Mills was vice president of Mission Assurance/Six Sigma at Northrop Grumman’s former Mission Systems sector, where she led the strategy, planning and execution for Six Sigma to increase customer satisfaction through streamlined operations and improved quality."