Slashdot Mirror
State of Virginia Technology Centers Down
bswooden writes "Some rather important departments (DMV, Social Services, Taxation) in the state of Virginia are currently without access to documents and information as a technology meltdown has caused much of their infrastructure to be offline for over 24 hours now. State CIO Sam Nixon said, 'A failure occurred in one memory card in what is known as a "storage area network," or SAN, at Virginia's Information Technologies Agency (VITA) suburban Richmond computing center, one of several data storage systems across Virginia.' How does the IT for some of the largest departments in a state come to a screeching halt over a single memory card? Oh, and also, the state is paying Northrup Grumman $2.4 billion over 10 years to manage the state's IT infrastructure."
Reader miller60 adds, "Virginia's IT systems drew scrutiny last fall when state agencies reported rolling outages due to the lack of network redundancy."
How does a fault in a single SAN controller cause an outage of the entire data storage network? Expensive SAN solutions are expensive & highly redundant for reason. This smells like a "Let's buy the cheaper solution" and/or an infrastructure design fail.
sounds like nobody in Virginia knows either
Our primary concern should be a complete audit of World of Warcraft server hardware, to ensure that this vulnerability does not exist in other, more vital networks.
Add in politics: Get a couple of representatives arguing over where the money (if any) should be spent, and all possibility of real redundancy and fault-tolerance go out the window.
It's true in larger government organizations than this. The failures just haven't occurred yet.
'Sensible' is a curse word.
Umm, so what's the point of having a SAN if it weren't redundant? Me thinks there is more to this story.
There's not a lot of money left over for redundancy after you take out the kickbacks, graft and bribes.
Everyone seems to think that a network outage is no big deal, until the network goes down. That's when people start thinking of the burn rate of an entire organization sitting on their thumbs while that network of off-the-shelf Linksys routers is replaced by some kid at Best Buy. Or how that 5k dollars per year for a backup external line suddenly pales in comparison to the 5k dollars per hour your organization is wasting because you were a cheap bastard.
The ______ Agenda
That's insane. Terry Childs failed (he was arrested and unable to make changes to the network)--and the city kept running.
My 'preconceive bubble' is based on my current job for the US government, and the situation we have in our department.
It might be true on average that government agencies are better at keeping their own infrastructure, especially if they can manage to keep their accounting and design of that infrastructure at a lower level. However, once those decisions pass the level from the internal to the external (or: From those hired for the job, to those elected/appointed into it), that long-term planning appears to break down, in favor of political squabbles.
'Sensible' is a curse word.
This is what Politics in VA is all about.
Favors handed out; tax money wasted.; public screwed.
Rinse and repeat.
That they were going to do "Remediation" of the NAS when the problem started, and they had EMC guys on site, and everything. They must have killed the primary when they attached the secondary. Don't wait for a weekend outage window, lets just do it now on a Tuesday afternoon at 2pm. No one will know... oops....