Anatomy of the VA's IT Meltdown

Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"

Comments

In other words....

[Like2Byte]

Business as usual for the VA.

Once again, the VA shows its true colors and mucks up another project funded by taxpayers for the well-being of our nations Veterans. A more screwed up organization one will not find.

Re:In other words....

[LurkerXXX]

You think things like this don't happen at private hospitals? I work with one and I can tell you right now they do.

Re:In other words....

[AvitarX]

Or maybe examine medicare.

I won't say it's perfect, but it has quite low overhead (relative to private insurance) and if there was no debate about who was allowed on and who wasn't it could be streamlined further.

Very few people want a single source of healthcare providing everything.

Re:In other words....

[Enry]

I can follow up a bit on this, since I worked for the DVA for a few years in the early 90s. Even then, just about all records were online and searchable. A veteran that went from Albany, NY to Tampa, FL and got sick could get his records transferred overnight (electronically) between the two hospitals, and there were ways to get metadata about the veteran immediately, including recent visits at any location and reason for the visit. I imagine that improvements in networks mean that these records can be viewed immediately.

At the time, there seemed to be a lot of waste (think $10,000 CD burner in 1993ish, optical cards with images and data impressed on them, etc). But they really were trying to be ahead of the game - a friend of mine showed me his green card and it was almost identical to a design I was working with when I was at the DVA. They also had mechanisms for charging back to private insurance companies in the event a veteran was only partially covered for a visit.

Oh, and just about all the software that was written and in use by those hospitals are in the public domain and downloadable for free - many other hospitals use VistA as their base.

Re:In other words....

[Like2Byte]

The VA is far more than just another hospital. It is supposed to aid US Veterans of all service branchs to see to the needs of them from educational loans, purchasing a home, medical care/assitance and others. See their site: http://va.gov./

If any one hospital or chain of hospitals peformed as consistantly lousey as the VA has that hospital would have been sued into oblivion decades ago. Hundreds of thousands of vets who've used the VA's services can attest. But, we can't neccessarily sue the VA because they're part of the government. Go to any VA hospital in the US. Odds are that after you pass through the pretty facade they've set up you'll find patient after patient sitting in a wheel chair or bed lined along some wall waiting for some over-worked, over-stressed and under-staffed doctor and not getting the care they deserve.

The VA needs to take a lesson from the corporate world and change it's face. Rename itself, start fresh. AND START DOING THEIR G-D JOB! That's the best dismal chance they've got to make things right. As it is right now there isn't a Vet in the US or abroad that thinks highly of the VA. And if there is, I'd find 100 that would refute any positive statement made about the VA.

And, yes - I'm a Vet. My Father is a Vet. My Grandfather is a Vet. My Uncle is a Vet. I don't recall them looking forward to communicating with the VA, either.

In closing, if the VA *did* do their job the homeless wouldn't consist of 25% US Veterans that couldn't re-adjust to civilian life after witnessing the horrors of war!

http://www.cnn.com/2007/US/11/08/homeless.veterans/
http://www.cnn.com/HEALTH/blogs/paging.dr.gupta/2007/05/mia-in-plain-sight.html

Re:In other words....

[leoxx]

You should all be so lucky.

Re:In other words....

[bockelboy]

I beg to differ. If you think the VA is crap, go to a private hospital. The VA consistently ranks better than any hospital system in the US. The following article is 2 years old, but it outlines how it beats the crap out of other hospital systems:

http://www.washingtonmonthly.com/features/2005/0501.longman.html

If you think the VA is bad, you can always go to your favorite HMO and have a higher chance of death.

Did I mention that the VA is a leader in hospital IT infrastructure and is decades ahead of other hospitals?

http://en.wikipedia.org/wiki/Veterans_Health_Information_Systems_and_Technology_Architecture

The VA is the largest hospital system in the US and its budget is decreased most years after adjusted for inflation. Given the predicament that Congress puts them in, they've done pretty well.

However, every single mistake they make is a public headline. Private hospitals have the luxury of being sued and quietly settle for $$$. Instead, the VA has to endure lots of bad publicity.

If the VA was a corporation, costs would skyrocket and even more corners would be cut. If you want to make it better, how about you ask Congress to provide adequate funding for the avalanche of people they are getting?

Re:In other words....

[moosesocks]

The Walter Reed travesty was indeed one that never should have happened.

HOWEVER, it is a brilliant example in which a public outrage was sparked, and the government was forced to do its job, and did indeed clean things up after the horrible conditions were brought to light.

If it were a private hospital, I fear that things would have been kept hush-hush for far longer through lawsuits and settlements. Even then, the worst that the government could do to the place would be to either impose fines, or shut them down entirely. Neither of these scenarios would benefit the patient.

90% of the time, patients at hospitals don't have an active choice in which hospital they arrive at. If you're severely injured or sick, you're not going to drive 2 extra hours to the "better" hospital. You're going to want to get immediate medical attention. In this regards, hospitals are prime candidates for nationalization, as they are hardly ever in competition with each other, which in turn results in all sorts of nasty side effects like inefficiency, negligence, etc.... It really does make sense to hold every hospital to the same exact standards across the nation.

You're also conveniently ignoring the fact that Walter Reed was a single entity in a very large system. There are always going to be a few outliers. Considering that the VA isn't funded nearly as well as it should be (we cut funding to the VA, and used the cash to fund an illegal war) it's still a pretty darn nice health system. An outrage was sparked because a government-run institution that was part of system in which all of the hospitals should have been more or less equally, had fallen behind. This simply would not have occurred in a private system.

To use these reasons to argue against a nationalized healthcare system in the US is to be completely ignorant. We're one of the only industrialized nations *without* a nationalized healthcare system open to all. The idea works. Reconciling the general incompetence of the US government is another issue entirely.

Oh, great.

[Penguinisto]

Now MSFt will take this and start trumpeting a victory for Vista.

(of course, it would be a first for 'em... even if it's the "wrong" Vista we're talking here).

/P

I see the problem

[moogied]

Too many discplines combined..

Anatomy Medical.

centralize IT systems IT.

four regional Topographical.

Fault lines appear Seismology.

There clearly is just not enough synergy..

Assumption junction, what's your function?

[digitaldc]

Volpp assumed that the data center in Sacramento would move into the first level of backup -- switching over to the Denver data center. It didn't happen.

DOH! Looks like it was all just due to someone's assumption that someone else would do their job.
From my experience, you can assume things happened, but if you don't verify that they actually happened - you are DOOMED.

Re:Assumption junction, what's your function?

[Qzukk]

DOH! Looks like it was all just due to someone's assumption that someone else would do their job.

DOH! Looks like someone was making assumptions without reading the article. They considered switching to the backup, but since they didn't know whether the problem was on their end or the server's end, they were afraid that switching to the backup data center would destroy that one as well.

my 2 cents.

[Brigadier]

unfortunately one of the best ways to learn how well your disaster recovery system works is to have a disaster. The problem with scheduled drills is the scenarios themselves are planned out and typically not run system wide ie test the part of the system then that part of the system etc. on RTFA it seems much of the breakdown occurred because too many people assumed. There was also no centralized decision making identities who had access to all the information. All scenarios when view from there individual perspective seemed to have made the right decision. However sometimes when implementing a global recovery plan one system may have to be sacrificed by another.

awesome!

[192939495969798999]

Awesome, sorry if someone already posted but I just couldn't resist the following quote:

Instantly, technicians present began to troubleshoot the problem. "There was a lot of attention on the signs and symptoms of the problem and very little attention on what is very often the first step you have in triaging an IT incident, which is, 'What was the last thing that got changed in this environment?'" Raffin said.

p.s. I am shocked at how many junior cowboy IT people remain employed, given the supposed glut of hire-able and knowledgeable folks.

Re:awesome!

[Richard Steiner]

Many companies don't know enough to fire people who are damaging to their operations.

Re:awesome!

[TooMuchToDo]

Mod parent up. As a small business owner, I've found that one reason our clients love us is because we manage their entire environment (be it hosting or internal network) and we provide them with all the documentation. I tell them "If you can't fire us at any time and keep running with no problems, we haven't done our job." Luckily, our clients love us, and we haven't been fired yet (in business 7 years).

Zonk, you retard

[sootman]

I'm sure I'll get modded to -5, Flamebait, but fucking A, Zonk, Slashdot isn't a newspaper. You don't need to be so economical in your headlines. When I saw the headline, I first thought of VA Linux--you know, the guys who kinda sorta own you. "Medical centers" threw me, so I thought for a second that it might mean the state of Virginia. Then it dawned on me that you probably meant the United States Department of Veterans Affairs. I'm sure I'm not the only one.

Please, God, isn't there some kind of Editing 101 correspondence-school course we can send all these guys to? I mean, I love Slashdot to death, but please God, can you give the staff just one ounce of basic editorial skills: spelling, grammar, etc? Teach them to write for clarity, not just brevity? Maybe go for broke and touch on dupe-checking, fact-checking, changing links so they point to the original article instead of some guy's AdSense-laden blog page that says nothing more than "here's the story"?

You're EDITORS, for God's sake (even if in name only), you are indeed allowed to EDIT submissions.

VA Acronym?

[bmomjian]

Isn't it obvious that the acronym "VA" isn't good to use in a title? FYI, it stands for "U.S. Veteran's Administration".

Why always centralizing?

[guruevi]

I wonder why higher management always wants to centralize their resources. The internet protocol and subsequent many IT applications were built to be efficient in small and decentralized environments.

1) Trying to centralize gives us large expensive computers that are made out of the same components as smaller ones and thus fail just as the smaller ones do, however, ever trying to cram more crap on the same machine will bring down everything at once whenever it fails.
2) Trying to centralize has the ultimate goal to eliminate jobs but they need those people since they know all the little details and hickups their systems have. If people know a project is going to eliminate their job, they won't be cooperative. IT not being cooperative is very bad in this world where everything is computerized.
3) Eventually the same number of people is going to have to work in the centralized system just because you also centralize the problems and more problems will bring more people, more people will bring more overhead and inefficiency, more inefficiency will bring more people (at least that's the default in today's business world, throwing more people at an IT problem doesn't make it disappear faster)
4) More people in a project that was designed to be more cost efficient means the managers will have to cut expenses. Cut expenses brings underpaid people, underpaid people bring less or no experience and higher turnover, higher turnover means more cutting expenses.

Therefore: keep your local IT guy(s) and infrastructure although you can't squeeze 100% of work/day and it will bring a little more expense. The end-users have a better relationship with the guy(s) and that makes happier people. Centralizing brings more overhead, less customer-interaction with IT and thus more inefficiency throughout the business.

Re:Why always centralizing?

[bobaferret]

There are certain this that centraalization brings to the table. Such as this this guy just came into the hospital unconcious , and we know that when he was in a VA hospital accross the country last week he was given a drug that would interact badly with what we want to give him right now. Or what is the chnage in his cat scan since last week whe he was someplace else and had one.

Obviously not all of this data needs to be centralized, but it's existance should be. We don't know to what level the VA was doing this, but I've met a large number of people who work in it's IT branch, and they love what they do, and are very good at it as well. Sometimes things just go wrong, and sometimes things get pushed out there for beuracratic reasons, but most of the time the VA is very IT savy.

Re:Why always centralizing?

[TheSkyIsPurple]

1) Trying to centralize gives us large expensive computers that are made out of the same components as smaller ones and thus fail just as the smaller ones do, however, ever trying to cram more crap on the same machine will bring down everything at once whenever it fails.

If that's how you're doing it, you're doing it wrong.

On how many smaller systems can you upgrade your disk controller's firmware without having to reboot or even stop access to the disks? Not a problem on a good SAN system.
And those systems only get economical when your data storage needs get big.

2) Trying to centralize has the ultimate goal to eliminate jobs but they need those people since they know all the little details and hickups their systems have. If people know a project is going to eliminate their job, they won't be cooperative. IT not being cooperative is very bad in this world where everything is computerized.

It doesn't always have that ultimate goal, but very often does. And very often, if done correctly, it can achieve that goal.
Take 8 sites with 2 admins each that are only doing 50% duty running that service. (You need at least 2 so someone gets to have an occasional vacation).
That's 16 people, doing the workload of 8.
Bring that down to 1 site, and odds are you could do the exact same job with 8 people (since now there are 7 others to back you up)

And now you're all on one system, so you don't have lots of little variances, so you can be more efficient, etc...

Yes, we have lost some of the little details by losing those people, but in general you've got other problems if some information is only known by one person.
As it turned out, a lot of that "critical" information got passed along to other folks anyway, most of what was left turned out to be unimportant, and that small remaining percentage?
Well, the rest of us are smart, and the ones with the info weren't idiots... we were able to figure it out.

3) Eventually the same number of people is going to have to work in the centralized system just because you also centralize the problems and more problems will bring more people, more people will bring more overhead and inefficiency, more inefficiency will bring more people (at least that's the default in today's business world, throwing more people at an IT problem doesn't make it disappear faster)

Starting with bad assumptions.
A small focused skilled team can do pretty much anything. =-)
In fact some would say they're the only ones who do anything.

One example: We used to repeatedly run into situations where we had the same problem at x sites, so we had at least x people trying to solve it. We didn't realize other's were duplicating our effort, so there was a lot of wasted effort, with solutions from different angles, so the sites ended up getting more and more out of sync in their setups.

4) More people in a project that was designed to be more cost efficient means the managers will have to cut expenses. Cut expenses brings underpaid people, underpaid people bring less or no experience and higher turnover, higher turnover means more cutting expenses.

Every centralization project I've been on has had its hiccups, but in the end has resulted in reduced costs overall. We always started off with the people we had, and a contractor or to who was an "expert" in the field we were working in, just to make sure we had an outsider's view. We didn't always believe the contractor, but we'd at least use them for everything they were worth. We then "centralized", and kept most of the folks around to keep everything running everywhere... then the layoffs.

The main problem we have on from our last centralization is that many in our small team are very shy about sharing issues before they know everything about it. They're afraid of looking bad, because they won't be as valuable. (Hadn't run into that one before)

Re:Why always centralizing?

[henryhbk]

Centralizing is often highly-advantageous in health care, because you need the records RIGHT NOW and the patients are allowed to go to any VA medical center in the country. I had a VA patient who lost his medications from Alaska while on vacation on the east coast, and I was able to retrieve his medication list push to our clinic's pharmacy in minutes, but it took 5 minutes to pull his records. Now I know most folks laugh, that 5 minutes is considered a long time, but anyone who works in a very busy walk-in clinic where you see patients every 15 minutes, realizes that a 5 minute hit is a giant wrench in to the works.

You also need to realize that these system use a hierarchical database for speed, so "joins" are much more complicated after the fact, than simply centralizing onto regional servers. Also regionalization is how the VA works as well with local integrated delivery networks (sort of hub-and-spoke with clinics, small and large medical centers) where the patients are often going between institutions in the region.

I recently left the VA system, and have to say their system is better than almost anyone else's, and has a similar failure rate to those at other institutions; as someone pointed out, it is just that this makes bigger press. In general there are very few health systems which have a single pervasive medical record of this scale (heck most places don't have electronic medical records at all) such as Kaiser Permanente here in the US, and the NHS in the UK, so these stories seem all the more spectacular since they are so rare.

Re:Good thing they weren't running Ubuntu

[Anonymous Coward]

UbuntuDupe seems to have a major attitude about Ubuntu. For anyone who doesn't know the story here's why. Basically, UbuntuDupe ran into problems installing Ubuntu and, when he asked for help on Ubuntu Forums, immediately started attacking the people that were sincerely trying to help him. Even with his major attitude the Ubuntu folks still tried their best to help him until they just couldn't put up with him any more. Read it for yourself and you'll see UbuntuDupe's Slashdot postings on Ubuntu in a new light.

+1 C'mon Editors

[ggvaidya]

I had a real fun time parsing this article.

1. Looks at title: omg! Slashdot's parent company had an IT meltdown! ha-ha! But waitaminute ...
2. Looks at icon: a ... crown? The Queen? Perhaps they mean *our* overlords, VA Linux? Or is VA Linux a monarchist organisation now?
3. Looks at summary: and ... medical? Why are th... oh HANG ON WAIT A MINUTE
4. Looks at icon: I remember that! It means ... government! Crown, government, get it? So, VA Linux screwed up a government's medical system? That makes ...
5. Looks into the inner recesses of my mind: ... sense, but ... something's out of place, something's ... just ... not ... quite ...
7. Looks at lightbulb over head: of course! There *is* no VA Linux! It's Sourceforge, Inc now! But that must mean ...
6. Looks at summary: ... carefully ... the VA, why the VA, shouldn't it be ... Vir..ginia?!

Gee thanks, Zonk, just what I needed before going to sleep. Now I'll dream of the Queen in Virginia melting down medical computers for Slashdot's open source overlords. Again.

Last thing I needed ...

Poor VMS.

[juuri]

staffers from Hewlett-Packard Co. conducting a review of the center's HP AlphaServer system running on Virtual Memory System and testing its performance.

We hardly knew ye.

It happens

[ACMENEWSLLC]

What they were doing was a major change to their IT infrastructure. That's massive. Things happen. The fact that they were down at 17 of 128+3 (131) data centers because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.

Seems to me that things worked otherwise well is a major accomplishment. They are still on the old system and are entering in data back into that system and migrating into the new system. But it seems things went well otherwise.

Anytime you do a major shift like this, it's hard. The users hate it because they can do their job very quickly on the system they are use to, but now have to learn a new system and slow down.

Things happen.

Re:It happens

[Critical Facilities]

because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.
I don't know if I agree with that. "Change Control" or "Change Management" is a crucial part of any Data Center. The fact that these ports were changed without being properly "run up the flagpole" is a glaring mistake with very unfortunate results. I'll bet anyone swapping ports in the future will ask permission several times over before trying it again.

Re:They messed up everything they could mess up.

[Nkwe]

Um... they are not running the Vista you think they are. From TFA:

Vista, Veterans Health Information Systems and Technology Architecture, is the VA's system for maintaining electronic health records. It sounds like they are running something much older. Again from TFA:

According to Director Eric Raffin, members of the technical team were at the site with staffers from Hewlett-Packard Co. conducting a review of the center's HP AlphaServer system running on Virtual Memory System and testing its performance. "Virtual Memory System" on an Alpha would be "VMS" would it not? Note the article only states that some folks were working on VMS at the same time when the Vista system (not the Microsoft OS) went down. It doesn't say that they were the same system, but you should consider that their environment is a bit more older and complicated that you make it out to be.

The article also states that the cause was a network configuration error. While in a perfect world you would have test and QA systems that are identical to production, it is rarely feasible. The cost to set up large parallel networks with exactly the same configuration in addition to software with the same configuration is generally cost prohibitive. By "same configuration" I mean same IP addresses, port assignments, routing rules, nationwide WAN links, etc.

I work at the heart of this...

[Anonymous Coward]

1st off... VISTA is not Windows VISTA. It's the "Veterans Health Information Systems and Technology Architecture". Do a google search on that.

VISTA runs on HP's VMS, and on top of that it runs Cache from Intersystems. (And yes it costs the tax payers a lot! But a lot less since we've been centralizing it over the last 3 or 4 years.)

It is a HUGE system.

The centralization that we're currently undergoing is massive, this problem was (IMHO) scape goated to a poor change control process.

I know what was change, I know who changed it, and I know when they changed it. However, this 'melt down' has happened three times... (Not to the same drastic outcome.) It comes down to VMS locking out logons because locks aren't being released properly. (Now you could argue that the reason locks got behind was this change... But I don't think that is the real reason because of our previous problems.)

It's that simple. Ask the VISTA manager over lunch sometime. They weren't afraid of data corruption. They were afraid if they moved the systems, the other system would lock up too with too much user load.

There goes "VISTA". Everyone logged in is fine. Everyone not on... Isn't getting on.

Now comes the bad part... No procedures!

We take 32 medical centers, and throw their IT into a data center. You 'had' clear lines of who owns what, and what happens when they go down. Now you centralize all that... Who raises the flag when something bad happens? Is it the site that has the problem? Is it someone who now controls the system at the data center? Who is responsible for what?

Oh wait... OI&T only has a dozen staff... And almost NONE of those people are technical. Everyones pay was simply moved from one appropriation to another. But what about the IT systems?!?! We moved those too, but didn't hire any permanent staff to take care of it? We just rubber banded a bunch of people together that work across the whole west coast and hand them a pager and say good luck?

Suffice it to say, we have some REALLY REALLY hard working people... And some really bad management. (Congress forcing us to do things on a time table is really annoying. Especially since they expect results, but don't expect any documentation... What do you think is going to get skipped?)

Congress: How is that data center move going!
Howard: We've moved 28 sites!
Congress: Good Job!
Howard: .:Thinks:. Too bad they don't know about everything we've short changed to make such an obscene deadline!

Then again... Howard doesn't even know everything we skip to get things done.

Bah

Like the budgie!

[PHAEDRU5]

Cowboy IT people remain employed because they're cheap!

First thing I learned in the military: your weapon was made by the lowest bidder.

Cache is part of the problem

[FBodyJim]

I work for a company that uses the Intersystems Cache database and I have to say that I imagine that Cache is a large part of the problem. The amount of good documentation for Cache lies between very little and none and my company has been on a nationwide search for people experienced with Cache and they too seem few and far between. Of course, I don't know that Cache really is a "worse" or "better" database that Oracle, SQL Server or MySQL for that matter, however, what I do know that is when it comes to experience, common tasks, documentation, examples and just getting things done, Cache lags far behind the others, not to mention Universities are still teaching relational db theory, not object db theory, at least when I graduated Rutgers a few short years ago. I suspect that given the task of merging databases, even large databases, there are plenty of experienced and knowledge SQL Server, Oracle, mySQL guys out on Monster or some other job site that know how to get the job done, efficiently and correctly, and have done the job a few times before. Based on our current and past searches for people capable of even easier tasks within Cache, there aren't many people out there with any Cache experience, never mind good people with Cache experience, and it's easy to fudge a task when you aren't given much good documentation, examples or experience. In a past career, I worked for a healthcare company that used SQL server for electronic medical records (EMRs) and the system worked rather well. There might have been better ways to design the database, stored procs or application code, however, we never had a problem hiring good staff that understood the database design, SQL queries, T-SQL/stored procs and as i said, I can't say the same about trying to hire good people who know and understand Mumps ("M" the language, not the disease) or Cache ObjectScript or find the Cache tools to be easy and intuitive. Just my $.02, and I don't mean to start a DB debate, just stating that it might just also be time for the VA's to move on from MUMPS/Cache to a more widely used and documented database and programming language, find some new blood.

Re:Cache is part of the problem

[mikelieman]

I don't know about that. First, the judicious use of whitespace might help people understand you argument better.

M is what T-SQL/stored procs wants to be when it grows up. I'm pretty sure getting help from Intersystems isn't an issue at the VA.

This is a Management/Change Management issue. Not a technical issue.