Slashdot Mirror
New Virginia IT Systems Lack Network Backup
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?
Palm trees and 8
Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?
But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?
Remember how Virginia's health records were compromised earlier this year?
http://it.slashdot.org/article.pl?sid=09/05/05/1232240
Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!
Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.
None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.
While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.
The IT Community Frowns Upon Your Shenanigans...
Northrop Grumman outsources part of it's own IT as well and it does not own some of it's systems they rent them or at least they did 1-2 years ago.
PHB's who say it's cheaper to let some out side people run there IT and likely say no to the higher cost plan that has backup with it.
Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.
Either that was a terrible attempt at irony or you really really weren't watching closely enough. The Galaxy class had at least two redundant paths for every key primary system.
i thought government programs never fail. /s
This is the same fiasco that the Navy went through with EDS and NMCI. Does no one at the state IT level read trade mags? This crap was all over the IT news magazines when the EDS contract was blowing up.
But how stupid can these people be to not include backups? i dont see this as a matter of improperly specifying their needs. I see this as a matter of stupidity. thats all.
again, sorry for the troll post, but this is ridiculous.
Is seems nobody RTFA (no surprise). The problem they're having is network outages at branch offices. I assume they're using DSL or such, with no way to connect if/when it goes down. Any one office probably has >99% up time, but when you have hundreds of offices and the remnants of a hurricane come through you can expect several of them to go offline, which is what's happening.
The article does not mention "backups" as in tape drives and off-site storage.
The article does mention lack of redundancy at the network carrier level.
My guess is that Northrop Grumman designed a network around single circuits connecting offices to data centers, and did not design the network to tolerate WAN link failures.
A stupid oversight for sure, but nothing that can't be easily remedied by ordering redundant WAN circuits from your telco of choice. Redundant routing gear would also be smart.
For all that are blaming government for this - they outsourced the design and implementation to a private company. That company screwed the pooch in design and implementation. Shame on both parties for not recognizing the risk of WAN failure.
-ted
I've been in the business of running networks for many years. I generally find that, as the company gets smaller, the amount of redundancy and unknown problems goes up. I expect every small business that I walk into to have no redundancy and backup systems that work 10% of the time. However, the fact that someone as large as NG and the government overlooked this critical part of system design is amazing to me.
I always explain to my small business clients that 'backups are worthless, restores are priceless.' It generally takes a few seconds to set in before the questions or statements start coming. "But, I have a backup scheduled every night. I take the tapes offsite every Friday."
It's always an eye-opener when I show them the console on their server with lots of red errors indicating that the data on the tapes are incomplete at best. Then, I start talking about the fact that if their server were to die, not only would it take a day or so to get parts for that old thing, but that the likelihood of being able to get it running at 100% again is slim or very costly at best.
Even small businesses will invest in a backup/business continuity device after hearing that and reading our blog that contains 'my company saved the day for a client yesterday'.
Shameless plug: Backups are worthless, restores are priceless blog.
If any story deserves this tag it is this. from the article:
"Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.
But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.
"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.
"It took VITA four hours to open the help ticket" and begin to solve the problem."
4 hrs on a critical system seriously just to get started solving it?
"During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."
wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)
Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable
I *knew* there had to be some other reason why they closed half the interstate rest stops in VA, this is obviously where the money was (mis)spent...
From TFA:
Suddenly, I don't feel so bad for that 2 1/2 hour glitch last week :)
naw the system was down which told the rest stops to lock their doors. The should be open again now, assuming it hasn't crashed again... well fudge
... for every loss in productivity they are responsible for. This "mistake" is sheer incompetence.
From TFA "The 10-year, $2.3 billion project aims to modernize 85 state government agencies' computer networks, PCs, phones, servers and e-mail systems, while holding down costs."
So basically they gave 2.3 billion to a giant gov't contractor because they said they could do it better and cheaper. News flash ... they lied. The cheaper comes from eliminating anything you didn't specify as absolutely essential, like network backups. So far it looks like the only thing they do better is cash your checks. The state will now scramble around trying to add in some redundant network connections, starting with their high priority targets. Anyone want to bet that when they finally get everything stabilized the new outsourced system that was supposed to be "holding down costs" will end up costing significantly more than anyone imagined. That is how gov't contractors work. They put in a low ball bid, and make sure that anything that is not explicitly defined in the contract is billed at rates that make up the difference. Your fault VA.
You think you get actual people when you outsource? People cost money, which reduces profitability.
As long as the systems are inside SLA, what's the problem?
Deleted
There is a lot of bad sentiment here in VA as this consolidation/outsourcing of IT cost quite a few State jobs which obviously upset a lot of people. Of course, they forget the fact that Northrop Grumman is one of the State's biggest employers and that they were hiring people to work at the new location. I still hear of lot of anger on radio stations and in the news when I'm in that region.
You mean outside people as in low-cost Indians on H1-B visa status? Yep, thought so.
The OP would make a heck of a lot more sense if they used better terminology. Rather than Network Backup, call it Network Redundancy or High Availability. When people hear backup, they think storing data to tape.
RTFG - Read The F#$%ing Google!
Hope you used an anonymizing proxy to post that. Nobody dares announce that the Emperor is indeed stark raving naked, lest the proclaimer get his head chopped off.
Northrop Grumman's core business is making airplanes (at least it was). So what we have here is a non-core business effort on behalf of a state government contract. I'll bet it was staffed by the B team at Northrup Grumman because real IT hot shots just are not motivated to get out of bed in the morning to chase state government contracts. On top of this staffing issues, I'm sure the government had lots of non-standard 'requirements' from insecure bureaucrats that need to justify their jobs. This is a lethal combination of doom for this project. The solution is standard requirements shared by lots of similar customers that is chased after by multiple competing vendors but I suspect the politics of patronage in Virginia would never let that happen.
.... getting a drivers license in Texas was bad!
and has involved itself in the market in some way in the past
therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow"
my favorite is how free market fundamentalists wish to blame the market crash of 2008 on government policies. rather than gee, i dunno, the clinton and bush administration deregulation policies? you know, deregulation: having the government less invovled int he market?
"what? my free market bubble and pop? nah, impossible! government's fault! pffft"
please study your banking panics of the 1800s: without regulation, free markets have innate imperfections which always result in catalcysmic failures. all you need is simple human psychology, no government need apply, to cause a market to crash. you either regulate it, leveling the playing field artificially, and therefore making it truly "free", or you leave it alone, letting it bubble and pop like mad, and allow monopolists to take advantage of natural imperfections in the market to leverage unfair behavior
free market fundamentalism is dead. your ideology is dead. fact: you need government involvement in the market for the market to run efficiently. fact: you need government policing and regulation of the marketplace to keep it "free" and egalitarian and equal for all players
if you don't understand these simple truths by now, or refuse to believe that despite the obvious proof, you're an idiot
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
But the site is down, so you can't apply.
(I'm trying to start a new meme, here)
No comprende? Let me type that a little slower for you...
Living in the DC area i see all sorts of crap going on with contractors. The amount of money the fed wastes on contractors should cause us to rise up and slay them. But so many profit (my GF and i included) that it seems unlikely that it will change soon. The CIA for instance has crippled itself by using so many contractors, people who SHOULD be on the payroll. Instead they line the pockets of executives and share holders, shitting away millions upon millions in overhead costs. This means much of the talent and experience belongs to companies like NG and not to the country. They can jack up their rates all they want and the gov't will have to pay. If they don't hire the contractor it doesn't get done. It's wasteful and potentially dangerous.
Utilizing the synergization of benchmark e-solutions to pre-workaround action items!
http://www.coolforsale.com/ spam troll low quality badly designed broke late delivery failed to deliver cheap material shoddy workmanship shady business dissatisfied customer unfriendly support poor service
Leave, or you'll continue to get negative search association here, spammer.
--
Help me out by posting this anonymously in response to all this spammer's posts. Maybe he'll give up.
would you like actually come out and refute my statement that you need government regulations to keep markets fair?
or are you just raging against a truth that hurts, but that your mind accepts?
thought so
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
as free market fundamentalists
it is a hallmark of the triumph of your fear over your intellect that you think that's what i am advocating for
examples of fear triumphing over intellect:
"gay marriage should be legal"
reaction:
"why do you think pedophiles should be allowed to marry boys and bestiality practioners to marry animals?"
#2:
"marijuana should be legal"
reaction:
"why do you want to legalize methamphetamine and heroin?"
#3:
"the government needs to regulate the market in order for it to be stable"
reaction:
"why do you want communist central planning"
do you see the hysteria at work in these examples?
in the future, i suggest you react to what i am actually saying, rather than projecting your irrational fears onto what i am saying, and reacting to that in hysteria
fact: an unregulated marketplace bubbles and pops due to nothing but simple human psychology, and naturally degenerates into a few powerful players dominating everyone else. without regulation, there is no such thing as a "free" market. a stable free market of equals, without regulation by some entity, is something that has never existed in the history of humanity. its a myth cooked up by libertarian fundamentalists, their garden of eden. its a fantasy of blindness in direct contradiction to obvious well-established human behavior:
1. people will take advantage of others, take advantage of natural imperfections in the market, and establish domination and exploitation of later arrivals to the marketplace
2. people will react in panic and fear at rumors, and destroy the market on nothing but emotion: calm rational decisions does not dominate the market
do you care to defy these simple obvious truths?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
When they talk about auxiliary power/thrusters/life-support whatever, that's the secondary/backup system. Except it's generally not a full-scale-system, but more of a scaled-down system meant to get by until the main system is repaired/replaced.
I'm more concerned about the lack of bathrooms though. I'm not sure I'd want to be on a Starship if the crew are all wearing some futuristic version of Depends...
I arrived at a private sector job where they thought they had backups for the past four years. They were just switching an 80 GB tapes every day on the Windows 2k3 servers without running a backup program, had 'floppy-punched' tapes into being accepted as DDS-4 tapes, and had tar backups conking out at 2 GBs because of filesystem limitations of SCO OpenServer which didn't sync properly with the pirate backup server anyway as it had run out of space eons ago and there was no cleanup schedule. Also the RAID card's battery to hold config data was toast (one power loss away from losing everything) and a disk had failed in the array anyway.
As for security, there was a dial up line to the system. No password required to get in and full-access to the system was nothing but a Ctrl-C away. Locally, there was also unprotected wi-fi to the network.
The system held lots of private, government and research data. If it was publicly compromised the business would have lost out on many a contract in the future. Did they care about any of this? Not really. Bottom line was all that mattered. So the story isn't too surprising. Businesses don't care about their own systems, let alone the systems of others.
Looks like Big Government screwed up again, should have gone with the private sector!
The 1st job of IT/IS is lobbying management. How many times was the project reviewed by State IT/IS professionals? How professional can they be if they can't even review a project for the most basic needs? How professional can Northrop Grumman be if they don't even notice a missing basic need in there analysis? This is just another example of the expense of outsourcing and no one being responsible for any IT/IS function. With the unemployment rate and 500,000 IT/IS professionals out of work, the US needs to start hiring in IT/IS to save money to offset their losses from outsourcing. I taught a lot of IT/IS security and none have info security jobs. With all the data stolen in the US, no one hires security.
its just not a rational thing for you. its this bizarre fundamentalist religion for you, the magical mythical "free market". how dare i question its dominion!
if i draw a picture of the free market with a bomb in its turban, will you riot?
fucking fundamentalists of this world, beyond reason, blind as fuck
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
i say we need a vibrant market, but it should be regulated. wow, how radical
in response you regurgitate the usual moronic, hysterical reaction: "why do you want communist central planning!"
what?!
here's an amazing new concept for you: how about you react to what someone actually says, rather than projecting your panty twisted fears on to what they are saying... and then reacting to that stupid insanity as if that is what i am actually representing to you
in your head, fear triumphs over reason
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Response is simple. Companies get a big check from government to design/install something then another to fix it.
you believe the market left alone takes care of itself, and the government makes it unstable
this is the opposite of reality: a free market is inherently unstable. government involvement stabilizes it
i'm sorry i don't have any books by crackpots to cite to prop up the fucking obvious truth for you
not like you would accept it
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
If every city in a the state has their own DMV office there would be about 30 offices with 100 outages over 6 months.
That means every office was out 3.3 times in six months.
Didn't they get a massive flood?
Doesn't sound bad to me my office would go offline if it was flooded with 5ft of water.
The system is not that critical. Still though for 2.3B they should have had business continuity built in.
shoulda woulda coulda....how is this surprising to anyone? I wonder if the GSA will get an overhaul under the new administration to potentially help nick these kinds of things in the butt. More importantly, where was ioSafe when they needed them?
to me about their literature after knocking on my door
i choose not to join your ridiculous religion
a market without regulation bubbles and pops, and is dominated and manipulated by insiders
and you are apparently their willing fool
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Are you fucking kidding me? Unlike religion, one can actually analyze cause and affect in the economy. You know, with numbers and stuff.
Or we could have the equivalent of virginia's it dept running our financial sector. 25/26 days down time... yeah baby!
with someone who has seriously embraces an absurd premise: that markets left to their own devices are stable and egalitarian
markets left to their own devices bubble and pop, and are manipulated and dominated by entrenched insiders
to not understand this is equivalent to someone refusing to accept that that the sun rises and sets or that the tides go up and down. how can you have a rational discussion with someone who refuses to see and accept obvious factual aspects of the reality they live in?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
"the government is involved in a modern economy... therefore if anything goes wrong with a market, we can creatively invent convoluted lines of cause and effect back to the government being the source of the problem somehow"
all we have to is completely remove the government from the marketplace and everything will be roses and wine, right? but of course, even THEN, if something DOES go wrong (and it will), people like you will still say its the government's fault... because the government touched the market at some point in the past, and you see, that magical taint has ruined the magical invulnerability of the self-correcting natively egalitarian always rational marketplace
fucking bullshit ...or, alternatively, maybe an unregulated marketplace naturally bubbles and pops out of nothing more than simple human psychology, and entrenched interests take advantage of natual imperfections in the market to dominate and manipulate smaller players and latecomers
NAAAAAAAAAAAH. that's impossible!
pfffffffft
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
no supplemental reading material from libertarian crackpots about the virginal holiness of the unregulated market?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Having worked in several VA state agencies, all of which have to deal with the state IT agency, I've seen some of the best and some of the worst in that state. I suspect there are some good people at the IT agency, but I have neither first hand experience with any such people nor hearsay nor even anecdotal evidence of any such existence. That agency MIGHT assist a few agencies with their IT needs, but for most agencies it hinders. If you want your project to fail, become mired, or be cancelled, just get the Department of Information Technology (or whatever it's called today) involved. I'm surprised when there's a success in which they're involved, so it's no biggie when such an "oversight" occurs. Why is it this way? Partly because political hacks in the Gov's office and in the state legislature have no clue about IT.
Northrop Grumman PM: "So, let's talk about backup requirements"
Virginia State Pointy Haired Boss: "But the SAN's RAID, isn't it? We'll be fine"
It gripped her hand gently. 'Regret is for humans,' it said.