Slashdot Mirror
Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
I have a project in a separate NG hosted dataspace, in Virginia. They are killing us with incompetence and their sub contractors are worse then they are. We are still trying to get things certified and they won't provide us information about their hosting. We think they have us on virtual servers that belong to another project, and the reason they don't want to tell us anything is that it would reveal they are in breach, since we are paying for dedicated servers.
Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong fix.
It was EMC storage that failed:
http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/
"A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.
Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."
Here you go, direct from our local news radio station.
"Northrop Grumman holds a $2.4 billion, 10-year contract with the Virginia Information Technologies Agency to build, operate and maintain the state's 7-year-old, problem-plagued consolidated computer services bureaucracy. It is the largest single-vendor contract in Virginia history. The partnership has been repeatedly criticized in JLARC studies for poor and tardy delivery of services, cost overruns and system failures."
These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.
Living With a Nerd
This kind of thing seems to be a growing trend in government IT. I'm posting anonymously because while I don't think I'm going to say anything that violates an NDA, it's better to be vague and sure.
The county that I live in recently made a move like this: they fired basically all of their IT staff and replaced them with the lowest bidding consulting company.
The upside of that from a certain fiscal standpoint is that they've eliminated a bunch of positions with pensions and good (and therefore expensive) benefits. These people have been replaced, in some departments, by the exact same people now subcontracting through the consulting company. This isn't really cheaper -- they won't have additional pension obligations to those people, but they're drawing much higher salaries than before, and obviously the consulting company gets a sizeable cut too.
In other departments, all the long-term employees have been replaced by new consultants. This is a problem in that, probably the people who had those jobs should have documented their networks and tasks much better, but the fact is, they didn't. The whole memory of those departments has been flushed. Inexperienced people are now trying to figure out how to maintain processes that literally no one who works there knows anything about. It's a disaster and it's going to get a lot worse before it gets better.
Meanwhile, the county executive is running for governor, touting the above as a great accomplishment. Hey, he shrunk the size of government by eliminating many permanent positions! By the time people realize that, not always, but sometimes, a lifer IT person is worth their pay because of the institutional memory they have of a thousand important things that were never documented, the election will be over.