Slashdot Mirror
Anatomy of the VA's IT Meltdown
Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"
Awesome, sorry if someone already posted but I just couldn't resist the following quote:
Instantly, technicians present began to troubleshoot the problem. "There was a lot of attention on the signs and symptoms of the problem and very little attention on what is very often the first step you have in triaging an IT incident, which is, 'What was the last thing that got changed in this environment?'" Raffin said.
p.s. I am shocked at how many junior cowboy IT people remain employed, given the supposed glut of hire-able and knowledgeable folks.
stuff |
DOH! Looks like it was all just due to someone's assumption that someone else would do their job.
DOH! Looks like someone was making assumptions without reading the article. They considered switching to the backup, but since they didn't know whether the problem was on their end or the server's end, they were afraid that switching to the backup data center would destroy that one as well.
If I have been able to see further than others, it is because I bought a pair of binoculars.
I can follow up a bit on this, since I worked for the DVA for a few years in the early 90s. Even then, just about all records were online and searchable. A veteran that went from Albany, NY to Tampa, FL and got sick could get his records transferred overnight (electronically) between the two hospitals, and there were ways to get metadata about the veteran immediately, including recent visits at any location and reason for the visit. I imagine that improvements in networks mean that these records can be viewed immediately.
At the time, there seemed to be a lot of waste (think $10,000 CD burner in 1993ish, optical cards with images and data impressed on them, etc). But they really were trying to be ahead of the game - a friend of mine showed me his green card and it was almost identical to a design I was working with when I was at the DVA. They also had mechanisms for charging back to private insurance companies in the event a veteran was only partially covered for a visit.
Oh, and just about all the software that was written and in use by those hospitals are in the public domain and downloadable for free - many other hospitals use VistA as their base.
I beg to differ. If you think the VA is crap, go to a private hospital. The VA consistently ranks better than any hospital system in the US. The following article is 2 years old, but it outlines how it beats the crap out of other hospital systems:
http://www.washingtonmonthly.com/features/2005/0501.longman.html
If you think the VA is bad, you can always go to your favorite HMO and have a higher chance of death.
Did I mention that the VA is a leader in hospital IT infrastructure and is decades ahead of other hospitals?
http://en.wikipedia.org/wiki/Veterans_Health_Information_Systems_and_Technology_Architecture
The VA is the largest hospital system in the US and its budget is decreased most years after adjusted for inflation. Given the predicament that Congress puts them in, they've done pretty well.
However, every single mistake they make is a public headline. Private hospitals have the luxury of being sued and quietly settle for $$$. Instead, the VA has to endure lots of bad publicity.
If the VA was a corporation, costs would skyrocket and even more corners would be cut. If you want to make it better, how about you ask Congress to provide adequate funding for the avalanche of people they are getting?