Slashdot Mirror
Anatomy of the VA's IT Meltdown
Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"
Why is Virginia trying to get into the federal record management arena? Shouldn't they keep to themselves?
Stop sending our troops to fight bullshit wars, and you won't have to worry about Veteran's medical records.
Business as usual for the VA.
Once again, the VA shows its true colors and mucks up another project funded by taxpayers for the well-being of our nations Veterans. A more screwed up organization one will not find.
(of course, it would be a first for 'em... even if it's the "wrong" Vista we're talking here).
Quo usque tandem abutere, Nimbus, patientia nostra?
There clearly is just not enough synergy..
So basically, -1 troll/offtopic is really slashdots way of saying "I hate that you thought of something before me."
it's worth a try. in reality, yOUR 'sway' is determined by yOUR lifestyle/spending habits.
destruction based 'economy'/management has never worked. it's an illness. tie that with life0cidal aggression & gangster style bullying, & what do we have? a greed/fear/ego based recipe for disaster.
we're intending for the nazis to give up/fail even further, in attempting to control the 'weather'.
http://video.google.com/videosearch?hl=en&q=video+cloud+spraying
&oe=UTF-8&um=1&ie=UTF-8&sa=N&tab=wv&oi=property_suggestions&resnum=0&ct=property-revision&cd=1
the creators will prevail. as it has always been.
corepirate nazi execrable costs outweigh benefits
(Score:-)mynuts won, the king is a fink)
by ourselves on everyday 24/7
as there are no benefits, just more&more death/debt & disruption.
fortunately there's an 'army' of angels, coming yOUR way
do not be afraid/dismayed, it is the way it was meant to be. the only way out is up.
the little ones/innocents must/will be protected.
after the big flash, ALL of yOUR imaginary 'borders' may blur a bit?
for each of the creators' innocents harmed, there is a debt that must/will be repaid by you/us, as the perpetrators/minions of unprecedented evile, will not be available.
beware the illusionary smoke&mirrors.con
all is not lost/forgotten.
no need to fret (unless you're associated/joined at the hype with, unprecedented evile), it's all just a part of the creators' wwwildly popular, newclear powered, planet/population rescue initiative/mandate.
or, is it (literally) ground hog day, again? many of US are obviously not interested in how we appear (which is whoreabull) from the other side of the 'lens', or even from across the oceans.
vote with (what's left in) yOUR wallet. help bring an end to unprecedented evile's manifestation through yOUR owned felonious corepirate nazi glowbull warmongering execrable.
we still haven't read (here) about the 2/3'rds of you kids who are investigating/pursuing a spiritual/conscience/concious re-awakening, in amongst the 'stuff that matters'? another big surprise?
some of US should consider ourselves very fortunate to be among those scheduled to survive after the big flash/implementation of the creators' wwwildly popular planet/population rescue initiative/mandate.
it's right in the manual, 'world without end', etc....
as we all ?know?, change is inevitable, & denying/ignoring gravity, logic, morality, etc..., is only possible, on a temporary basis.
concern about the course of events that will occur should the life0cidal execrable fail to be intervened upon is in order.
'do not be dismayed' (also from the manual). however, it's ok/recommended, to not attempt to live under/accept, fauxking nazi felon greed/fear/ego based pr ?firm? scriptdead mindphuking hypenosys.
consult with/trust in yOUR creators. providing more than enough of everything for everyone (without any distracting/spiritdead personal gain motives), whilst badtolling unprecedented evile, using an unlimited supply of newclear power, since/until forever. see you there?
"If my people, which are called by my name, shall humble themselves, and pray, and seek my face, and turn from their wicked ways; then will I hear from heaven, and will forgive their sin, and will heal their land."
The article said the project was pulled back and will be looked at - that doesn't necessarily mean cancellation
//m
No organization that I know of has EVER had good luck with the name VISTA.
Volpp assumed that the data center in Sacramento would move into the first level of backup -- switching over to the Denver data center. It didn't happen.
DOH! Looks like it was all just due to someone's assumption that someone else would do their job.
From my experience, you can assume things happened, but if you don't verify that they actually happened - you are DOOMED.
He who knows best knows how little he knows. - Thomas Jefferson
Why didn't the build the second "centralized" system in parallel to the one that already existed? This way, when the new system failed miserably, just flip the switch (or DNS record) back to the old servers and retool the "solution" that you were testing.....
that brings another point to mind...
DIDN'T THEY TEST THE FREAKING THING!?
NewslilySocial News. No lolcats allowed.
unfortunately one of the best ways to learn how well your disaster recovery system works is to have a disaster. The problem with scheduled drills is the scenarios themselves are planned out and typically not run system wide ie test the part of the system then that part of the system etc. on RTFA it seems much of the breakdown occurred because too many people assumed. There was also no centralized decision making identities who had access to all the information. All scenarios when view from there individual perspective seemed to have made the right decision. However sometimes when implementing a global recovery plan one system may have to be sacrificed by another.
Awesome, sorry if someone already posted but I just couldn't resist the following quote:
Instantly, technicians present began to troubleshoot the problem. "There was a lot of attention on the signs and symptoms of the problem and very little attention on what is very often the first step you have in triaging an IT incident, which is, 'What was the last thing that got changed in this environment?'" Raffin said.
p.s. I am shocked at how many junior cowboy IT people remain employed, given the supposed glut of hire-able and knowledgeable folks.
stuff |
and sourceforge, too?
"Flyin' in just a sweet place,
Never been known to fail..."
I'm sure I'll get modded to -5, Flamebait, but fucking A, Zonk, Slashdot isn't a newspaper. You don't need to be so economical in your headlines. When I saw the headline, I first thought of VA Linux--you know, the guys who kinda sorta own you. "Medical centers" threw me, so I thought for a second that it might mean the state of Virginia. Then it dawned on me that you probably meant the United States Department of Veterans Affairs. I'm sure I'm not the only one.
Please, God, isn't there some kind of Editing 101 correspondence-school course we can send all these guys to? I mean, I love Slashdot to death, but please God, can you give the staff just one ounce of basic editorial skills: spelling, grammar, etc? Teach them to write for clarity, not just brevity? Maybe go for broke and touch on dupe-checking, fact-checking, changing links so they point to the original article instead of some guy's AdSense-laden blog page that says nothing more than "here's the story"?
You're EDITORS, for God's sake (even if in name only), you are indeed allowed to EDIT submissions.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Isn't it obvious that the acronym "VA" isn't good to use in a title? FYI, it stands for "U.S. Veteran's Administration".
-Um, we can't access any data for some reason.
"Are you using the latest version, Hairy Hardon?"
-No, but...
"I guess your CD must not have been burned properly."
-No, it says, quite clearly, GRUB error 25 at Stage 1.5. What does that mean?
"What you need to do is go download the Live CD."
-But you said the Install CD *is* the Live CD!
"Can't you just wait till tomorrow or something? Asking for Ubuntu to work in a hospital is totally non-standard, you should have said so at the beginning, I mean, who the hell expects it work in such an unusual environment?"
-Can you just tell me what GRUB error 25 is?
"What version of Windows did you use before switching? LOL, Windows users..."
Apology to Ubuntu forum.
I wonder why higher management always wants to centralize their resources. The internet protocol and subsequent many IT applications were built to be efficient in small and decentralized environments.
1) Trying to centralize gives us large expensive computers that are made out of the same components as smaller ones and thus fail just as the smaller ones do, however, ever trying to cram more crap on the same machine will bring down everything at once whenever it fails.
2) Trying to centralize has the ultimate goal to eliminate jobs but they need those people since they know all the little details and hickups their systems have. If people know a project is going to eliminate their job, they won't be cooperative. IT not being cooperative is very bad in this world where everything is computerized.
3) Eventually the same number of people is going to have to work in the centralized system just because you also centralize the problems and more problems will bring more people, more people will bring more overhead and inefficiency, more inefficiency will bring more people (at least that's the default in today's business world, throwing more people at an IT problem doesn't make it disappear faster)
4) More people in a project that was designed to be more cost efficient means the managers will have to cut expenses. Cut expenses brings underpaid people, underpaid people bring less or no experience and higher turnover, higher turnover means more cutting expenses.
Therefore: keep your local IT guy(s) and infrastructure although you can't squeeze 100% of work/day and it will bring a little more expense. The end-users have a better relationship with the guy(s) and that makes happier people. Centralizing brings more overhead, less customer-interaction with IT and thus more inefficiency throughout the business.
Custom electronics and digital signage for your business: www.evcircuits.com
It's another one of these monstrous systems integration projects that will/never work and every hospital/med center is doing them. They want everything to talk to everything and the only reason behind it is really more big brother Total Info Awaren. BS. I worked for a hospital in the 90's. It was started before I got there and still going on years later after I left. I'm sure it's still going on. It is a monstrous bureaucracy that costs millions (billions?) and you can expect problems of this scale to increase as they continue to centralize their vulnerabilities.
And I've never heard of anyone running even a piece of a datacenter on Vista. Everyone complains that outsourcing companies are too expensive, but honestly, we're a LOT smarter than the fools who implemented this. We would never have this mistake.
I had a real fun time parsing this article.
... ... crown? The Queen? Perhaps they mean *our* overlords, VA Linux? Or is VA Linux a monarchist organisation now? ... medical? Why are th... oh HANG ON WAIT A MINUTE ... government! Crown, government, get it? So, VA Linux screwed up a government's medical system? That makes ... ... sense, but ... something's out of place, something's ... just ... not ... quite ... ... ... carefully ... the VA, why the VA, shouldn't it be ... Vir..ginia?!
...
1. Looks at title: omg! Slashdot's parent company had an IT meltdown! ha-ha! But waitaminute
2. Looks at icon: a
3. Looks at summary: and
4. Looks at icon: I remember that! It means
5. Looks into the inner recesses of my mind:
7. Looks at lightbulb over head: of course! There *is* no VA Linux! It's Sourceforge, Inc now! But that must mean
6. Looks at summary:
Gee thanks, Zonk, just what I needed before going to sleep. Now I'll dream of the Queen in Virginia melting down medical computers for Slashdot's open source overlords. Again.
Last thing I needed
Yeah, time to fire your IT organization's management. And a few of their leads, too. And maybe some of the techs.
Couple of reasons: First, they're running Vista. I'm not trying to be all "You must only run Linux or ur a n00b" here -- you can run Windows servers just fine, but no reasonable IT planner should ever, *ever* consider using an OS that new for a mission-critical enterprise application. If it doesn't have two or three years in the field, don't even consider it.
Second, their failover plan sucked. Live data syncs are good for physical disasters (fires, earthquakes, zombie attacks) but, as the VA discovered, they leave you shitting your pants when you run into an issue that may or may not be data-related. The solution to this, of course, is to keep a day or week-old copy someplace along with an up-to-date (but not implemented!) transaction log that you can go through and update with once you've sanity-checked it.
Third, letting the vendor run "tests" on your production system. Nobody, and I mean nobody, should ever get to touch any production system unless they're implementing a specific change that's been tested in an identical environment, passed QA and review by folks who know the system and then only with a published implementation, testing and backout plan. If a system needs "tests", you pull it out of production before you start messing with it.
Finally, their "virtualized team" approach (read: our people are scattered all over the place) is moronic -- you see this sort of thing, and without fail it's the result of political pressures rather than sane management. In this case, I'll bet my hat is was a situation where a bunch of middle managers were allowed to maneuver to keep their fingers in the pie when centralization tool place, so instead of having everyone you need on hand and in one group you're busy setting up conference calls.
Plus, now their solution is to bring in a bunch of consultants. Yeah, that always works. Good luck, guys! You're gonna need it.
Every year during my review, I just pray the words "slashdot.org" aren't mentioned.
In my experience with federal govt IT jobs, you usually have to FORCE others to do their jobs if your job depends on someone else doing their job to completion. This involves lots of whistleblowing and reporting them up the chain of command when they're slacking, and generally playing cop over them. You must always save all emails, memos and all forms of correspondence with them, take plenty of detailed notes at all meetings, and I've found that secretly recording your conversations and meetings with a portable voice recorder slipped into your pocket works wonders to help get a project done. Federal agencies have pretty much all de-evolved into a perpetual surveilence society (that's why they're doing it to the citizens now too), and surveillance is a language they understand clearly. You have to think and act like a prosecuting attorney to keep your project on track.
This is funny to me. I was hired by the VA in St. Petersburg, Florida a few years ago when Windows 2003 first came out to train all of the NT administrators on the migration to 2003. Of the 60 or so NT administrators, all but three of them were losing their title and becoming helpdesk for their site and "physical hands" for the few remaining administrators.
A lof of the admins were unhappy about that, as I would have been. I am just curious if the failure to complete the project had to do with the lack of respect for the older employees with NT experience and essentially downgrading those employees.
Vista != Microsoft Windows Vista
Within the Department of Veterans Affairs, they have an computerized medical records system. That medical records system has been called Vista for decades. It's an unfortunate source of confusion that Microsoft chose the same word for their new OS release, but the two have nothing to do with each other.
Does anybody else get the impression that they created an Ethernet loop and couldn't figure it out for a whole day ?
staffers from Hewlett-Packard Co. conducting a review of the center's HP AlphaServer system running on Virtual Memory System and testing its performance.
We hardly knew ye.
--- I do not moderate.
What they were doing was a major change to their IT infrastructure. That's massive. Things happen. The fact that they were down at 17 of 128+3 (131) data centers because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.
Seems to me that things worked otherwise well is a major accomplishment. They are still on the old system and are entering in data back into that system and migrating into the new system. But it seems things went well otherwise.
Anytime you do a major shift like this, it's hard. The users hate it because they can do their job very quickly on the system they are use to, but now have to learn a new system and slow down.
Things happen.
And these are the clowns the dems want to put in charge of healthcare...
1st off... VISTA is not Windows VISTA. It's the "Veterans Health Information Systems and Technology Architecture". Do a google search on that.
.:Thinks:. Too bad they don't know about everything we've short changed to make such an obscene deadline!
VISTA runs on HP's VMS, and on top of that it runs Cache from Intersystems. (And yes it costs the tax payers a lot! But a lot less since we've been centralizing it over the last 3 or 4 years.)
It is a HUGE system.
The centralization that we're currently undergoing is massive, this problem was (IMHO) scape goated to a poor change control process.
I know what was change, I know who changed it, and I know when they changed it. However, this 'melt down' has happened three times... (Not to the same drastic outcome.) It comes down to VMS locking out logons because locks aren't being released properly. (Now you could argue that the reason locks got behind was this change... But I don't think that is the real reason because of our previous problems.)
It's that simple. Ask the VISTA manager over lunch sometime. They weren't afraid of data corruption. They were afraid if they moved the systems, the other system would lock up too with too much user load.
There goes "VISTA". Everyone logged in is fine. Everyone not on... Isn't getting on.
Now comes the bad part... No procedures!
We take 32 medical centers, and throw their IT into a data center. You 'had' clear lines of who owns what, and what happens when they go down. Now you centralize all that... Who raises the flag when something bad happens? Is it the site that has the problem? Is it someone who now controls the system at the data center? Who is responsible for what?
Oh wait... OI&T only has a dozen staff... And almost NONE of those people are technical. Everyones pay was simply moved from one appropriation to another. But what about the IT systems?!?! We moved those too, but didn't hire any permanent staff to take care of it? We just rubber banded a bunch of people together that work across the whole west coast and hand them a pager and say good luck?
Suffice it to say, we have some REALLY REALLY hard working people... And some really bad management. (Congress forcing us to do things on a time table is really annoying. Especially since they expect results, but don't expect any documentation... What do you think is going to get skipped?)
Congress: How is that data center move going!
Howard: We've moved 28 sites!
Congress: Good Job!
Howard:
Then again... Howard doesn't even know everything we skip to get things done.
Bah
Cowboy IT people remain employed because they're cheap!
First thing I learned in the military: your weapon was made by the lowest bidder.
668: Neighbour of the Beast
I work for a company that uses the Intersystems Cache database and I have to say that I imagine that Cache is a large part of the problem. The amount of good documentation for Cache lies between very little and none and my company has been on a nationwide search for people experienced with Cache and they too seem few and far between. Of course, I don't know that Cache really is a "worse" or "better" database that Oracle, SQL Server or MySQL for that matter, however, what I do know that is when it comes to experience, common tasks, documentation, examples and just getting things done, Cache lags far behind the others, not to mention Universities are still teaching relational db theory, not object db theory, at least when I graduated Rutgers a few short years ago. I suspect that given the task of merging databases, even large databases, there are plenty of experienced and knowledge SQL Server, Oracle, mySQL guys out on Monster or some other job site that know how to get the job done, efficiently and correctly, and have done the job a few times before. Based on our current and past searches for people capable of even easier tasks within Cache, there aren't many people out there with any Cache experience, never mind good people with Cache experience, and it's easy to fudge a task when you aren't given much good documentation, examples or experience. In a past career, I worked for a healthcare company that used SQL server for electronic medical records (EMRs) and the system worked rather well. There might have been better ways to design the database, stored procs or application code, however, we never had a problem hiring good staff that understood the database design, SQL queries, T-SQL/stored procs and as i said, I can't say the same about trying to hire good people who know and understand Mumps ("M" the language, not the disease) or Cache ObjectScript or find the Cache tools to be easy and intuitive. Just my $.02, and I don't mean to start a DB debate, just stating that it might just also be time for the VA's to move on from MUMPS/Cache to a more widely used and documented database and programming language, find some new blood.
I can't wait until all of our health care is provided by the government, seeing as how it's being done so well on a smaller scale.
Welcome to the modern enlightened world. You see way too many people feel that patriotism is stupid. If you join the military you are patriotic and so you are stupid. Why should they care?
I have seen people refuse to stand for the National Anthem on Veterans day at an airshow. Did you miss the people complaining about Google have a banner for Veterans Day?
If people will not stand and actively complain about a Google's Veterans Day banner why should they want to fund or fix the VA? That actually costs real money.
Yea we should fix the VA.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
I read the title and immediately thought of this VA IT meltdown.
Blind patriotism is stupid.
I have a security clearance and work with the DoD and IMO really smart people should not join the military. At this point we are so far into overkill mode that cutting the DoD's spending in 1/2 could make little difference to our overall safety if done in a reasonable fashion. (Ok, that level of change would probably create a mess, but slowly trimming the fat is a good idea IMO.) If you really want to make the US a better place go into private industry and start the next Google. Fighting in some pointless war is of limited value.
However, I feel the VA should be better funded and revamped. When we put a person's life in danger we should respect that and pay them back for their service.
If the incident actually occurred it made squat difference to treatment. I'm under fairly constant care at two related VA facilities and my treatment wasn't affected by any such thing. Sounds like it's just IT's problem.
I've gone to VA hospitals since 1989. I got insurance when I started teaching and started going to local doctors and hospitals. Before a year was up I was going back to the VA. Treatment that the VA doesn't provide is treatment the vet didn't request. To be fair, at the VA you need to request harder than elsewhere because elsewhere is going to get money for more treatment and the VA gets nothing different for more or less treatment, except in the largest sense in requesting federal funding after stats show what's been done easily and what took too long to accomplish.
In 1989 it took months to get an appointment for most things. The excuse was "we don't have enough money". Now if it takes a week, you get the same excuse. If I need to see my primary care physician I can get in same day or next day depending on what time of day I call.
I am presently getting the best care I've received to date, this at the hospital the VA said had the lowest marks just 3 years ago.
Sorry you have a problem, IT. You can rest a bit easier knowing it didn't affect the providers.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
They have basically no investment in hiring and personnel systems. I know because I'm a recent MPA grad and have seen their postings for personnel. They're worse than any other federal system (and federal hiring is generally inane).
Two months ago the VA was posting on usajobs and requiring all documents be mailed in -- no fax, no email, no online systems. And some of these positions had a 5 day application window with no leeway for stamped post. Even more bizarre, one of the postings stated flat out that "alternate delivery methods" should be used because the post office wasn't reliable (Read: HAND DELIVERY). It even gave the room number to hand deliver the application to...and this was for an entry level career position (college/MA grads). Clearly they were planning to hire someone internally in that case--at least I hope that corruption was the issue and not incompetence.
With such a high amount of hassle to simply apply for work at the VA, such poorly managed hiring systems, and disinterest in bringing in outsiders, why would anyone competent want to work for the VA? I know I wouldn't want to...I got my MPA to help improve systems, but that requires at least a marginally competent culture and active management...and the VA simply doesn't have either.
So have you written or called your rep about? When voting are you going to look at their voting record on VA funding?
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Urgh!
Walter Reed ARMY Medical Center is Army/DoD health care for active duty personnels.
WRAMC may or may not exhibit similar issues with VA hospital system. That remain to be seen.
However, confusion between the two systems does not help your credibility.
You are right that the Walter Reed scandal was a travesty. However you are missing one key detail. It ISN'T a VA facility! Walter Reed is an army hospital, meaning it is run by the Department of Defense. The VA and DoD are separate departments, each with their own cabinet secretary.
Put another way, would it be reasonable or appropriate to blame NIH (Dept of Health and Human Services) for security breaches at Los Alamos (Dept of Energy)? I mean, they both do basic science research, so they must be the same, right?
There has never been a successful failover test of the RI Department of State's Central Voter Registration System. Most of that is because of the obstinacy of the RI Department of Administration.
And State recently suffered a MAJOR web outage. Press says it was hacked, I know better. I used to manage that web server before I was summarily laid off. The MySQL database would start going haywire because it was an ancient version. All you had to do was kill the MySQL slave and restart MySQL and all would be fine.