Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage

Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."

Comments

Apt Futurama quote

[dkleinsc]

Hermes: What do we do when we break somebody's window?
Dwight: Pay for it?
Hermes: Heavens, no! We apologize! With nice, cheap words.

My Project

[WED Fan]

I have a project in a separate NG hosted dataspace, in Virginia. They are killing us with incompetence and their sub contractors are worse then they are. We are still trying to get things certified and they won't provide us information about their hosting. We think they have us on virtual servers that belong to another project, and the reason they don't want to tell us anything is that it would reveal they are in breach, since we are paying for dedicated servers.

Re:My Project

[snspdaarf]

I do some work with NG sub contractors. One is a delight to work with. Working with the other is like staring at the sun through binoculars. People don't answer email, sometimes because they are lazy, sometimes because they quit. When an answer does come in, it is worse than none at all. It's like the worst troll postings on slashdot, full of errors, run-on sentences that confuse more than clarify, and off topic. Absolute nightmare.

Re:My Project

[Cylix]

Unless they have gone to some very lengthy steps to hide this you can probably discover the information on your own.

http://www.dmo.ca/blog/detecting-virtualization-on-linux/

This page details steps for many different types of virtualization environments. Though I think it would be just as fast to sort through the output of dmidecode and look for an identification in the mess.

I'm afraid this is rather linux centric, but even so similar data sets can be collected on windows.

Re:My Project

[Monchanger]

As much as Northrup is being bashed here, I don't think this problem is specific to NG, but common to many large contractors and their subs.

In the few times I've worked with subcontractors doing IT for the government I've been unimpressed. Even being one step away from the prime contract seems to allow for many problems, both technical and managerial. Requirements and deadlines aren't met, and they pull the BP-Halliburton-Transocean trick of avoiding responsibility by blaming each other (as well as everyone's favorite scapegoat: the government). Trying to get a subcontractor to build things they way they were supposed to will often require waiting for the next spiral, which means going way over budget.

I understand the difficulty with pushing too hard, punishing contractors who screw up and scaring them away from government work, but it seems we've gone too far in accepting very expensive third-rate work. As much as the public likes to say government can't do anything right, how much worse off would we be than $2.4B in the hole with nothing to show for it but a mediocre datacenter run by amateurs? I don't think I'm asking for much, I haven't touched on the very messy political poisoning of contracting.

IT Bubble Syndrome

[darien.train]

The "old business man discovering the internet" IT bubble culture is still alive and well in the defense industry. They have such a bad track record with networking technology it borders on scary. Transformation comes to mind quickly and they keep repeating the same mistakes.

Re:To be fair...

[bsDaemon]

Meanwhile, the last week has marked the first time where there was really a valid excuse for apparently unmoving lines at Virginia DMV branches... glad I don't have to get my license renewed until 2017. They should be back up by then.

I was involved with NG on a project once....

[quangdog]

Fresh out of school with my CS degree I went to work on a project for my employer that involved partnering and working directly with folks from NG. The original deadline for shipping the solution was something like 6 months after I started. The complete and utter incompetence of the NG side of things wound up stretching this out more than 18 MONTHS longer, and the final delivery lacked a lot of the original stated requirements.

Being the newbie to the whole corporate culture, I was shocked that people were not bothered at all by blown deadlines, missed estimates, and huge cost overruns. Shortly after the project finally delivered, I bailed to work for a much smaller company (fewer than 10 employees) where I discovered that I really love the smaller, more dynamic environment that only small companies can provide.

Working for huge corporations just sucks.

Re:I was involved with NG on a project once....

[Nadaka]

I work for a smaller DoD related company (17k employees, so not that small really) and I hear nothing but horror stories about NG, and have experienced one of them myself. NG originally developed our project but was not allowed to bid on the re-compete where we picked it up. It was bad, but we are fixing it. I really can't say much more than that at the moment.

Re:I was involved with NG on a project once....

[GrumblyStuff]

Probably but you die on the inside every day you have to work up bullshit excuse why this was bound to happen and why it's a good thing.

NGC Culture

[CherniyVolk]

I work with a fair number of ngc.com; I'm a contractor myself. At one point, I had a few interviews for Northrop and I'm so glad they over looked my talents because they seem to abuse the talent they have.

Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.

See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

I only point out Northrop because while all government contracting is essentially this contract+play model, Northrop has a reputation of placing people on furlough much more often than other companies such as CACI, Raytheon, General Atomics etc. Some Northrop employees seem to live the lives similar to actors and actresses in Hollywood, and I'm not talking about Tom Hanks acting, but maybe those actors that get little spots from time to time on your sitcoms. They literally live in apartments, and wait for the phone to ring day after day. Northrop employees seem to wake up in the morning, wondering if they'll still have a job at the end of the day.

What does this observation have to do with the op? Well, it seems that moral and motivation might be a bit low on a large scale at Northrop, so such blunders are no surprise to me.

Re:NGC Culture

[XorNand]

See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

I do a lot of business with both the federal gov't and private sector businesses on IT projects. You've over-simplified things to the point of painting an inaccurate picture. Federal contracting is extremely complex and there are myriad types of contracts that can be awarded, each with different terms. It sounds like what you're describing is a labor-hour contract. The contractor bills the gov't for the "fully burdened cost" of putting a warm butt in a seat. This includes the worker's salary, overhead, G&A (general & administrative), and profit. All together, it's typically a lot more than a 50% markup of the staff's straight salary.

Unlike most private sector contracts, when doing a fully burdened labor hour contract with the feds, the contractor will spell out exactly what their profit margin is. Generally this is only 6-10%, which is considerably lower than the private sector. Despite what everyone thinks, doing business with the gov't isn't all that lucrative. It's an extremely competitive market in which the bottom-line cost is almost always the most important factor. Contracting officers are even prohibited by law to give preferential treatment to companies that have previously done a great job.

I can't really comment on forced furloughs, because I'm not familiar with how Northrup operates. But just because they do "government contracts" doesn't necessarily mean they can afford to keep highly-skilled staff on the payroll until they find a new project for them. Federal contracts can really help with sales revenue because they can be large awards and the government *always* pays. However, the trade off is all the red tape (which increases G&A costs) and the low profit margins. Next time you hear about Company X getting a $10M contract, don't just roll your eyes. Get a hold of their proposal and the contract and see what their actual profit is on the contract. Both documents are public property and available upon request from the federal contracting officer that made the award. (Defense related contracts might need to be pried from gov't with a FOIA request though)

It was EMC storage failure

[Anonymous Coward]

It was EMC storage that failed:

http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/

"A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.

Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."

Re:It was EMC storage failure

[Lunix Nutcase]

I would sue the living shit out of Northrop. That's insane.

Based on what would they sue? The contract they signed had no requirement of redundancy. As much fault as NG has in this, it's not like they broke the contract or anything. This is as much the fault of the incompetents in Virginia's IT Agency as on NG's.

Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

Re:$2.6 billion service contract?

[Pojut]

Here you go, direct from our local news radio station.

"Northrop Grumman holds a $2.4 billion, 10-year contract with the Virginia Information Technologies Agency to build, operate and maintain the state's 7-year-old, problem-plagued consolidated computer services bureaucracy. It is the largest single-vendor contract in Virginia history. The partnership has been repeatedly criticized in JLARC studies for poor and tardy delivery of services, cost overruns and system failures."

These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.

Re:$2.6 billion service contract?

[Anonymous Coward]

This kind of thing seems to be a growing trend in government IT. I'm posting anonymously because while I don't think I'm going to say anything that violates an NDA, it's better to be vague and sure.

The county that I live in recently made a move like this: they fired basically all of their IT staff and replaced them with the lowest bidding consulting company.

The upside of that from a certain fiscal standpoint is that they've eliminated a bunch of positions with pensions and good (and therefore expensive) benefits. These people have been replaced, in some departments, by the exact same people now subcontracting through the consulting company. This isn't really cheaper -- they won't have additional pension obligations to those people, but they're drawing much higher salaries than before, and obviously the consulting company gets a sizeable cut too.

In other departments, all the long-term employees have been replaced by new consultants. This is a problem in that, probably the people who had those jobs should have documented their networks and tasks much better, but the fact is, they didn't. The whole memory of those departments has been flushed. Inexperienced people are now trying to figure out how to maintain processes that literally no one who works there knows anything about. It's a disaster and it's going to get a lot worse before it gets better.

Meanwhile, the county executive is running for governor, touting the above as a great accomplishment. Hey, he shrunk the size of government by eliminating many permanent positions! By the time people realize that, not always, but sometimes, a lifer IT person is worth their pay because of the institutional memory they have of a thousand important things that were never documented, the election will be over.

Re:$2.6 billion service contract?

[AB3A]

Mod parent way up. I've seen it in other places too.

Often when you ask people in the accounting department, they'll say they don't really know what's going on because IT made the system. When you talk to IT, they'll say they don't know because the system was specified by Accounting. The truth is that some smart guys in each department got together and forged some sort of system together. Then the smart guys went on to bigger and better things, while the peons were left with some special voodoo system.

Now you want to move said voodoo system over to a consulting company. The assumption is that the Accountants know every detail of what the old system did. Well, they don't. But nobody is willing to step forward and say that. So the new consultants come along and gosh, nobody knows what the systems do.

Then people ponder why it "doesn't work." Sigh.

This is how shit happens.

Why NGC is so FUBAR

[MikeRT]

Northrop Grumman was cobbled together over several years from about 1999-2006 from Northrop, Grumman, TRW and several other players. It is so dysfunctional because it is composed of so many competing units that don't operate like a single company. In fact, when I briefly worked for them out of college, most of my coworkers were from TRW and hated the idea of being NGC employees.