Slashdot Mirror
New Virginia IT Systems Lack Network Backup
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?
Palm trees and 8
Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?
But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?
Remember how Virginia's health records were compromised earlier this year?
http://it.slashdot.org/article.pl?sid=09/05/05/1232240
Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!
Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.
None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.
While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.
The IT Community Frowns Upon Your Shenanigans...
Is seems nobody RTFA (no surprise). The problem they're having is network outages at branch offices. I assume they're using DSL or such, with no way to connect if/when it goes down. Any one office probably has >99% up time, but when you have hundreds of offices and the remnants of a hurricane come through you can expect several of them to go offline, which is what's happening.
The article does not mention "backups" as in tape drives and off-site storage.
The article does mention lack of redundancy at the network carrier level.
My guess is that Northrop Grumman designed a network around single circuits connecting offices to data centers, and did not design the network to tolerate WAN link failures.
A stupid oversight for sure, but nothing that can't be easily remedied by ordering redundant WAN circuits from your telco of choice. Redundant routing gear would also be smart.
For all that are blaming government for this - they outsourced the design and implementation to a private company. That company screwed the pooch in design and implementation. Shame on both parties for not recognizing the risk of WAN failure.
-ted
"During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."
wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)
Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable
From TFA:
Suddenly, I don't feel so bad for that 2 1/2 hour glitch last week :)
and has involved itself in the market in some way in the past
therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow"
my favorite is how free market fundamentalists wish to blame the market crash of 2008 on government policies. rather than gee, i dunno, the clinton and bush administration deregulation policies? you know, deregulation: having the government less invovled int he market?
"what? my free market bubble and pop? nah, impossible! government's fault! pffft"
please study your banking panics of the 1800s: without regulation, free markets have innate imperfections which always result in catalcysmic failures. all you need is simple human psychology, no government need apply, to cause a market to crash. you either regulate it, leveling the playing field artificially, and therefore making it truly "free", or you leave it alone, letting it bubble and pop like mad, and allow monopolists to take advantage of natural imperfections in the market to leverage unfair behavior
free market fundamentalism is dead. your ideology is dead. fact: you need government involvement in the market for the market to run efficiently. fact: you need government policing and regulation of the marketplace to keep it "free" and egalitarian and equal for all players
if you don't understand these simple truths by now, or refuse to believe that despite the obvious proof, you're an idiot
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it