Slashdot Mirror

← Back to Stories (view on slashdot.org)

Anatomy of the VA's IT Meltdown

Posted by Zonk on from the use-your-words dept.

Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"

103 of 137 comments (clear)

  1. In other words.... by Like2Byte · · Score: 4, Insightful

    Business as usual for the VA.

    Once again, the VA shows its true colors and mucks up another project funded by taxpayers for the well-being of our nations Veterans. A more screwed up organization one will not find.

    1. Re:In other words.... by Anonymous Coward · · Score: 1, Insightful

      Actually, the VA is lightyears ahead of the private sector when it comes to electronic patient records. Also, I would imagine that if one were to research all of the screw ups in private hospitals and small practices one would find just as many problems as the VA seems to have. It's just the fact that the VA is a large government entity that it becomes an easy target. On the whole, the VA used to be deserving of much of its bad publicity, but in recent years it has made a significant turnaround and should be considered a model for others to follow. I'm not saying that serious problems don't still exist, or that it's a perfectly run organization, but I do think that the media paints a one-sided picture.

    2. Re:In other words.... by LuxMaker · · Score: 1

      And Americans want a nationalized health care system? Examine the VA system and I suspect a nationalized health care system will be very similar.

      --
      I regret that I only have one mod point to give per post.
    3. Re:In other words.... by LurkerXXX · · Score: 2, Insightful

      You think things like this don't happen at private hospitals? I work with one and I can tell you right now they do.

    4. Re:In other words.... by AvitarX · · Score: 2, Insightful

      Or maybe examine medicare.

      I won't say it's perfect, but it has quite low overhead (relative to private insurance) and if there was no debate about who was allowed on and who wasn't it could be streamlined further.

      Very few people want a single source of healthcare providing everything.

      --
      Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
    5. Re:In other words.... by Enry · · Score: 4, Informative

      I can follow up a bit on this, since I worked for the DVA for a few years in the early 90s. Even then, just about all records were online and searchable. A veteran that went from Albany, NY to Tampa, FL and got sick could get his records transferred overnight (electronically) between the two hospitals, and there were ways to get metadata about the veteran immediately, including recent visits at any location and reason for the visit. I imagine that improvements in networks mean that these records can be viewed immediately.

      At the time, there seemed to be a lot of waste (think $10,000 CD burner in 1993ish, optical cards with images and data impressed on them, etc). But they really were trying to be ahead of the game - a friend of mine showed me his green card and it was almost identical to a design I was working with when I was at the DVA. They also had mechanisms for charging back to private insurance companies in the event a veteran was only partially covered for a visit.

      Oh, and just about all the software that was written and in use by those hospitals are in the public domain and downloadable for free - many other hospitals use VistA as their base.

    6. Re:In other words.... by Like2Byte · · Score: 4, Insightful

      The VA is far more than just another hospital. It is supposed to aid US Veterans of all service branchs to see to the needs of them from educational loans, purchasing a home, medical care/assitance and others. See their site: http://va.gov./

      If any one hospital or chain of hospitals peformed as consistantly lousey as the VA has that hospital would have been sued into oblivion decades ago. Hundreds of thousands of vets who've used the VA's services can attest. But, we can't neccessarily sue the VA because they're part of the government. Go to any VA hospital in the US. Odds are that after you pass through the pretty facade they've set up you'll find patient after patient sitting in a wheel chair or bed lined along some wall waiting for some over-worked, over-stressed and under-staffed doctor and not getting the care they deserve.

      The VA needs to take a lesson from the corporate world and change it's face. Rename itself, start fresh. AND START DOING THEIR G-D JOB! That's the best dismal chance they've got to make things right. As it is right now there isn't a Vet in the US or abroad that thinks highly of the VA. And if there is, I'd find 100 that would refute any positive statement made about the VA.

      And, yes - I'm a Vet. My Father is a Vet. My Grandfather is a Vet. My Uncle is a Vet. I don't recall them looking forward to communicating with the VA, either.

      In closing, if the VA *did* do their job the homeless wouldn't consist of 25% US Veterans that couldn't re-adjust to civilian life after witnessing the horrors of war!

      http://www.cnn.com/2007/US/11/08/homeless.veterans/
      http://www.cnn.com/HEALTH/blogs/paging.dr.gupta/2007/05/mia-in-plain-sight.html

    7. Re:In other words.... by leoxx · · Score: 2, Informative

      You should all be so lucky.

    8. Re:In other words.... by Scrameustache · · Score: 1

      And, yes - I'm a Vet. My Father is a Vet. My Grandfather is a Vet. My Uncle is a Vet. You must have some really healthy pets! : )
      --

      You can't take the sky from me...

    9. Re:In other words.... by bockelboy · · Score: 3, Informative

      I beg to differ. If you think the VA is crap, go to a private hospital. The VA consistently ranks better than any hospital system in the US. The following article is 2 years old, but it outlines how it beats the crap out of other hospital systems:

      http://www.washingtonmonthly.com/features/2005/0501.longman.html

      If you think the VA is bad, you can always go to your favorite HMO and have a higher chance of death.

      Did I mention that the VA is a leader in hospital IT infrastructure and is decades ahead of other hospitals?

      http://en.wikipedia.org/wiki/Veterans_Health_Information_Systems_and_Technology_Architecture

      The VA is the largest hospital system in the US and its budget is decreased most years after adjusted for inflation. Given the predicament that Congress puts them in, they've done pretty well.

      However, every single mistake they make is a public headline. Private hospitals have the luxury of being sued and quietly settle for $$$. Instead, the VA has to endure lots of bad publicity.

      If the VA was a corporation, costs would skyrocket and even more corners would be cut. If you want to make it better, how about you ask Congress to provide adequate funding for the avalanche of people they are getting?

    10. Re:In other words.... by Like2Byte · · Score: 1

      HAHAHAHAHAHAHAHAHHAHAOMGTOOFUNNNY!!!!!11!!!!!!!!one!

      May I remind you of Walter Reed Medical Hospital travesty that *recently* made headlines?

      http://www.encyclopedia.com/doc/1G1-161076682.html

      http://akaka.senate.gov/public/index.cfm?FuseAction=newsarticles.home&month=3&year=2007&release_id=1570

      http://www.usatoday.com/news/washington/2007-03-21-va-review_N.htm

    11. Re:In other words.... by moosesocks · · Score: 2, Informative

      The Walter Reed travesty was indeed one that never should have happened.

      HOWEVER, it is a brilliant example in which a public outrage was sparked, and the government was forced to do its job, and did indeed clean things up after the horrible conditions were brought to light.

      If it were a private hospital, I fear that things would have been kept hush-hush for far longer through lawsuits and settlements. Even then, the worst that the government could do to the place would be to either impose fines, or shut them down entirely. Neither of these scenarios would benefit the patient.

      90% of the time, patients at hospitals don't have an active choice in which hospital they arrive at. If you're severely injured or sick, you're not going to drive 2 extra hours to the "better" hospital. You're going to want to get immediate medical attention. In this regards, hospitals are prime candidates for nationalization, as they are hardly ever in competition with each other, which in turn results in all sorts of nasty side effects like inefficiency, negligence, etc.... It really does make sense to hold every hospital to the same exact standards across the nation.

      You're also conveniently ignoring the fact that Walter Reed was a single entity in a very large system. There are always going to be a few outliers. Considering that the VA isn't funded nearly as well as it should be (we cut funding to the VA, and used the cash to fund an illegal war) it's still a pretty darn nice health system. An outrage was sparked because a government-run institution that was part of system in which all of the hospitals should have been more or less equally, had fallen behind. This simply would not have occurred in a private system.

      To use these reasons to argue against a nationalized healthcare system in the US is to be completely ignorant. We're one of the only industrialized nations *without* a nationalized healthcare system open to all. The idea works. Reconciling the general incompetence of the US government is another issue entirely.

      --
      -- If you try to fail and succeed, which have you done? - Uli's moose
    12. Re:In other words.... by Like2Byte · · Score: 1

      Back at ya.

      Read the comments that were attached to that article. They don't give as glowing a review as the article.

      1 Article - followed by many more jaded Veterans or the family members who had to assist their Veteran to get there.

    13. Re:In other words.... by dbIII · · Score: 1

      Go to Canada, Australia, Sweden, even the much maligned NHS in the UK. There's no point arguing that both the private hospitals and VA are no good and leave it at that - others do better with less resources. It costs a lot to run through a lot of burnt out medical staff and the experienced people that could get a lot done don't have much reason to stay. The problem is change requires a lot of resources immediately even if it won't in the long run.

    14. Re:In other words.... by IgnoramusMaximus · · Score: 1

      I normally reply to these things with examples of well (or reasonably well) functioning universal medicare systems in pretty much all industrialized (and other) nations. However when faced with the continuing stream of total and utter fiascos authored by the US federal or state governments, who somehow mastered hereto before unattainable levels of incompetence, graft and general stupidity in anything even remotely relating to common good, I am beginning to lean towards another notion. It is the theory that America is truly unique in the quality of its sociopaths and the American public has grown so narcisstic, anti-social and greedy that it has become incapable of cooperating in anything not involving exploitation, theft, rape, murder or pillaging. Or some combination of thereof.

      A truly unique American achievment which might become the national epitaph. Sort of like the bloodbaths in the Colliseum were for Rome.

    15. Re:In other words.... by JebusIsLord · · Score: 1

      So you're saying that americans just aren't smart enough to have successful public institutions like Europe and Canada? Interesting position...

      --
      Jeremy
    16. Re:In other words.... by IgnoramusMaximus · · Score: 1

      Apparently not. Their public institutions are becoming less "public" as time goes on, serving and defending the newly froming de-facto corporate aristocracy (complete with dynastical "presidencies") and court-jester "parties". Creation of new egalitarian institutions, or even repeairing of the once impressive existing ones, is apparently already out of the question. As are inheretance taxes, the last remaining obstacle between any pretense of meritocracy and the feudal order.

      And so the Republic crumbles to dust, proud, arrogant, jingoistic and belligerent to the end. But despeir not, chances are that before it vanishes into history books it will flare once more, like a star in its final death throes, as it goes out in a blaze of a nation's equivalent to a Supernova: the Imperial Hegemony.

    17. Re:In other words.... by bronsinbound · · Score: 1

      Oh, I don't know about that. Why not look at the House or Senate?! Or the clown in the White House, State Department, ... ad nauseum.

    18. Re:In other words.... by jasontn · · Score: 1

      You can install a completely FOSS stack by running VistA on http://www.geocities.com/kdtop3/GT.M on top of Linux. The only issue is that GUI part, i.e. CPRS only runs on Windows. Modifying it to run on Wine is still incomplete.

    19. Re:In other words.... by leoxx · · Score: 1

      Right, because as we all know here on slashdot, comments posted anonymously on a web page are always factual and accurate.

  2. Oh, great. by Penguinisto · · Score: 2, Funny
    Now MSFt will take this and start trumpeting a victory for Vista.

    (of course, it would be a first for 'em... even if it's the "wrong" Vista we're talking here).

    /P

    --
    Quo usque tandem abutere, Nimbus, patientia nostra?
  3. I see the problem by moogied · · Score: 4, Funny
    Too many discplines combined..

    Anatomy Medical.

    centralize IT systems IT.

    four regional Topographical.

    Fault lines appear Seismology.

    There clearly is just not enough synergy..

    --
    So basically, -1 troll/offtopic is really slashdots way of saying "I hate that you thought of something before me."
    1. Re:I see the problem by protolith · · Score: 1

      So you need a DBA geointerographer

  4. not cancelled by mlong · · Score: 1

    The article said the project was pulled back and will be looked at - that doesn't necessarily mean cancellation

    --
    //m
    1. Re:not cancelled by orclevegam · · Score: 1

      We're sorry for any inconvenience, those responsible have been sacked. We will now continue at great expense in a completely different fashion. Also, my sister was once bitten by a moose.

      --
      Curiosity was framed, Ignorance killed the cat.
  5. Start by banning the name. by LibertineR · · Score: 1

    No organization that I know of has EVER had good luck with the name VISTA.

    1. Re:Start by banning the name. by Slashidiot · · Score: 1

      Personally I'm wishing good luck to this VISTA.

      --
      Tis women makes us love, Tis Love that makes us sad, Tis sadness makes us drink, And drinking makes us mad.
    2. Re:Start by banning the name. by TallMatt · · Score: 1

      I work for a medical imaging company called MIMvista and we are doing quite well. We were around before windows vista came to the party.

  6. Assumption junction, what's your function? by digitaldc · · Score: 3, Insightful

    Volpp assumed that the data center in Sacramento would move into the first level of backup -- switching over to the Denver data center. It didn't happen.

    DOH! Looks like it was all just due to someone's assumption that someone else would do their job.
    From my experience, you can assume things happened, but if you don't verify that they actually happened - you are DOOMED.

    --
    He who knows best knows how little he knows. - Thomas Jefferson
    1. Re:Assumption junction, what's your function? by Qzukk · · Score: 5, Informative

      DOH! Looks like it was all just due to someone's assumption that someone else would do their job.

      DOH! Looks like someone was making assumptions without reading the article. They considered switching to the backup, but since they didn't know whether the problem was on their end or the server's end, they were afraid that switching to the backup data center would destroy that one as well.

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
    2. Re:Assumption junction, what's your function? by digitaldc · · Score: 1

      DOH! Looks like someone read the article.
      And since they didn't know what the actual problem was, they just assumed things and it got hosed. I stand by my original statement.

      --
      He who knows best knows how little he knows. - Thomas Jefferson
    3. Re:Assumption junction, what's your function? by jesdynf · · Score: 1

      Err. Yes. That's what happens when you don't know things and can't feasibly learn them before you have to make a decision.

      Generally, "they're stupid 'cause when u assume lol" is reserved for thoughtlessly destructive acts. The decision not to sync to peer wasn't one -- it was an informed decision to cut their losses and have merely *one* hospital down, rather than risk having N hospitals down.

      --
      Yahoo! Pipes are awesome. How awesome? http://pipes.yahoo.com/jesdynf/slashdot
  7. my 2 cents. by Brigadier · · Score: 4, Insightful



    unfortunately one of the best ways to learn how well your disaster recovery system works is to have a disaster. The problem with scheduled drills is the scenarios themselves are planned out and typically not run system wide ie test the part of the system then that part of the system etc. on RTFA it seems much of the breakdown occurred because too many people assumed. There was also no centralized decision making identities who had access to all the information. All scenarios when view from there individual perspective seemed to have made the right decision. However sometimes when implementing a global recovery plan one system may have to be sacrificed by another.

    1. Re:my 2 cents. by Fulcrum+of+Evil · · Score: 1

      Simple solution there: start with the planned small scale DR, then work up to turning off a datacenter. People get a large window when the drill will happen, but details and precise schedule are not released.

      --
      "We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
  8. awesome! by 192939495969798999 · · Score: 3, Informative

    Awesome, sorry if someone already posted but I just couldn't resist the following quote:

    Instantly, technicians present began to troubleshoot the problem. "There was a lot of attention on the signs and symptoms of the problem and very little attention on what is very often the first step you have in triaging an IT incident, which is, 'What was the last thing that got changed in this environment?'" Raffin said.

    p.s. I am shocked at how many junior cowboy IT people remain employed, given the supposed glut of hire-able and knowledgeable folks.

    --
    stuff |
    1. Re:awesome! by Richard+Steiner · · Score: 2, Insightful

      Many companies don't know enough to fire people who are damaging to their operations.

      --
      Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
      The Theorem Theorem: If If, Then Then.
    2. Re:awesome! by TooMuchToDo · · Score: 2, Insightful

      Mod parent up. As a small business owner, I've found that one reason our clients love us is because we manage their entire environment (be it hosting or internal network) and we provide them with all the documentation. I tell them "If you can't fire us at any time and keep running with no problems, we haven't done our job." Luckily, our clients love us, and we haven't been fired yet (in business 7 years).

    3. Re:awesome! by Herkum01 · · Score: 1

      What was the last thing that got changed in this environment?

      The way most IT networks are run you would be lucky to know if anyone is updating their environment. Businesses don't care about employing good or experienced employees, not that they know how to know they have one, they are concerned with whether they are under-budget so they can get their bonus for the year.

    4. Re:awesome! by Aram+Fingal · · Score: 1

      I think the point of this quote in the article is not that there was something wrong with the technician's response but with IT management's response. It's perfectly logical for individual technicians, who have minimal information, to do whatever troubleshooting they can with what they have. The failure was at a higher level where they should have known about the change made to port configurations.

      I should also point out that you often have to be careful of instantly blaming the last change made before a problem. Anyone with experience knows that, whenever you make a change, especially one that end users know about, you get blamed for the next thing that goes wrong, even if it is completely unrelated to what you did. Obviously, that isn't what happened here since part of the problem was that the people who made the change didn't report it properly. My point is that these systems are complex enough that changes occur frequently enough and problems occur frequently enough that you have a lot of coincidences where a change happens and then there is a problem unrelated to the change. Beware of post hoc ergo propter hoc.

  9. Doesn't VA own slashdot? by Jeremiah+Cornelius · · Score: 1, Funny

    and sourceforge, too?

    --
    "Flyin' in just a sweet place,
    Never been known to fail..."
    1. Re:Doesn't VA own slashdot? by qweqwe321 · · Score: 1

      The Veteran's Administration, not VA Linux.

  10. Zonk, you retard by sootman · · Score: 5, Insightful

    I'm sure I'll get modded to -5, Flamebait, but fucking A, Zonk, Slashdot isn't a newspaper. You don't need to be so economical in your headlines. When I saw the headline, I first thought of VA Linux--you know, the guys who kinda sorta own you. "Medical centers" threw me, so I thought for a second that it might mean the state of Virginia. Then it dawned on me that you probably meant the United States Department of Veterans Affairs. I'm sure I'm not the only one.

    Please, God, isn't there some kind of Editing 101 correspondence-school course we can send all these guys to? I mean, I love Slashdot to death, but please God, can you give the staff just one ounce of basic editorial skills: spelling, grammar, etc? Teach them to write for clarity, not just brevity? Maybe go for broke and touch on dupe-checking, fact-checking, changing links so they point to the original article instead of some guy's AdSense-laden blog page that says nothing more than "here's the story"?

    You're EDITORS, for God's sake (even if in name only), you are indeed allowed to EDIT submissions.

    --
    Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
    1. Re:Zonk, you retard by Ctrl-Z · · Score: 1

      I think you missed the "the" in the headline. "The VA" can't refer to "VA Linux" (an entity which doesn't even exist in that name; even the "VA Software" name was dropped six months ago). Nor does "The VA" refer to the state of Virginia. In either of those cases I would expect "Anatomy of VA's IT Meltdown". What else could "the VA" stand for?

      --
      www.timcoleman.com is a total waste of your time. Never go there.
    2. Re:Zonk, you retard by kent_eh · · Score: 1

      What else could it stand for?
      Who knows. .VA is the TLD for the Vatican.

      Maybe the "the" that you refer to is a typo? who can tell in a /. headline?
      How hard is it to expand an acronym in it's first usage?

      --

      ---
      "I can't complain, but sometimes still do..." Joe Walsh
    3. Re:Zonk, you retard by ch-chuck · · Score: 1

      Flash, Reader Scorches Slashdot Editor - film at 11.

      For a brief second I though it was about a VIA chip that someone overclocked and melted, and they were doing some kind of post mortum on it.

      --
      try { do() || do_not(); } catch (JediException err) { yoda(err); }
    4. Re:Zonk, you retard by RichDice · · Score: 1

      The headline says the VA, obviously referring to the Veterans' Administration.

      Exactly how obvious should that be to the 95.4% of the people in the world who aren't American?

    5. Re:Zonk, you retard by Anonymous+Cowpat · · Score: 1

      and if I, as a Brit, referred to the RBL would you pick up what that expanded to straight away? No, I didn't think so. *
      While I understand that /. is America-centric, which explains why we're even seeing this story, it doesn't seem unreasonable to ask for peculiarly American acronyms to be written out in full for the benefit of the other 5.75 billion people in the world, particularly if a small grammatical error could have resulted in the given acronym actually expanding out to any one of 3 American-centric phrases. There's far too many unknowns in 'the VA' to be able to confidently figure out what's being referred to.
      Even on reading that whole summary, it could still be about a screw up with medical records in Virginia and just submitted with really badly constructed sentences - it also doesn't help that most of the other English-speaking nations have socialised healthcare systems so we won't immediately latch onto the idea of a Veteran's Administration being a major provider of healthcare, and so won't associate it with a story about healthcare provision, even if the acronym is the same.
      The first rule of writing acronyms is to expand each acronym in brackets after the first place where it appears.
      *Royal British Legion

      --
      FGD 135
    6. Re:Zonk, you retard by Skater · · Score: 1

      Somehow, I knew when I clicked on "Read more" I'd get people bitching about the headlines instead of actually discussing the article. Clearly, I'm not new here.

      Get over it. Life's too short to get so upset about this kind of stuff.

    7. Re:Zonk, you retard by Divide+By+Zero · · Score: 1

      You realize that we've been the Department of Veterans Affairs since 1989, right?

      Though, despite the change, "the VA" (the Veterans Affairs?) has managed to stick around, even in official literature. It makes the grammar nazi in me die a little every day, so I choose to take my pent-up frustrations out on you. Sorry.

      Hey, at least I don't work for the Postal Service, right? :)

      --
      Dare to Hope. Prepare to be Disappointed.
    8. Re:Zonk, you retard by sootman · · Score: 1

      I'll admit that I did miss the 'the' the first time. (Wow, I never thought I'd get "the the the" into a grammatically-correct sentence.) But it is well-known that the brain takes shortcuts when reading, and mine glossed over the 'the' and skipped right to the attention-grabbing capital letters.

      Furthermore, few people outside the USA are likely to know what "the VA" is, so a bit of clarification would be handy. Scroll the Slashdot front page right now--this is one of the shortest headlines on the screen. The Web is not a newspaper--we don't need to save every precious inch.

      And I know the name was dropped six months ago but it was around for over ten years before that and it's still in my head. So sue me.

      "The VA" can't refer to "VA Linux" -- sure it could: this is Slashdot. Happens all the time. ;-)

      --
      Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
    9. Re:Zonk, you retard by sootman · · Score: 1

      You realize that we've been the Department of Veterans Affairs since 1989, right? ... Though, despite the change, "the VA" (the Veterans Affairs?) has managed to stick around, even in official literature.

      You know how it is. Names stick. To those who were born at a certain time, the VA will always be the VA. Hell, even their website is still va.gov--despite the name change coming two years before TBL invented the WWW. (Though they might have had the domain name before then.)

      --
      Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
    10. Re:Zonk, you retard by sootman · · Score: 1

      Somehow, I knew when I clicked on "Read more" I'd get people bitching about the headlines instead of actually discussing the article.

      Which is exactly why Slashdot's editors are doing their readership such a huge disservice by not EDITING. I've read PLENTY of threads that were about some minor point instead of being about the story. I've seen stories where literally every +5 comment in the discussion was NOT about the content of the story but rather an error in the reporting or something else tangential--usually the stupid throwaway line the editors tack on to the end of a submission.

      If you want Slashdot discussions to be about the stories, you should be just as mad as I am.

      Life's too short to get so upset about this kind of stuff.

      Slashdot is something I enjoy, and I'm trying to make it better. How is that a waste?

      --
      Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
    11. Re:Zonk, you retard by Divide+By+Zero · · Score: 1

      The domain name thing doesn't bother me - a lot of departments don't have the distinction in their domain name.

      treasury.gov
      state.gov
      ed.gov
      interior.gov
      hhs.gov
      hud.gov

      Thank god they didn't register and promote www.theva.gov (*shudder*)

      --
      Dare to Hope. Prepare to be Disappointed.
    12. Re:Zonk, you retard by Fulcrum+of+Evil · · Score: 1

      if I, as a Brit, referred to the RBL would you pick up what that expanded to straight away?

      Google is your friend

      it doesn't seem unreasonable to ask for peculiarly American acronyms to be written out in full for the benefit of the other 5.75 billion people in the world

      I'd be surprised if the number of connected people was much over a billion.

      --
      "We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
    13. Re:Zonk, you retard by Fulcrum+of+Evil · · Score: 1

      *smacks UbuntuDope*. It's The VA, not VT or VA Tech of VA Linux. This really is perfectly precise. Besides, what effect would Cho's wild ride have on IT systems?

      --
      "We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
    14. Re:Zonk, you retard by Anonymous+Cowpat · · Score: 1

      I figured it out faster than it would have taken to google it, but that five seconds is orders of magnitude longer than the few milliseconds that it would have taken had 'Veterans Administration' just been written in the body of the text. Readability is not improved by having to spend 5 seconds parsing just 2 letters because someone couldn't be bothered to follow the conventional rules of writing things with acronyms. The point still stands.
      A quick google does bring up the "US Department of Veterans Affairs" as the first result. It brings up the Wikipedia article on Virginia as the second result. With little useful context to go on, you still can't be very sure as to what the summary is talking about. The point of a summary is to be able to scan down the page and see what you want to read more about, not to be forced to read more just to find out what the summary actually means.

      Further, even if the number is only a billion connected, rather than 6 billion, still some 75% to 80% of those people will still fall in the category 'not American'.

      --
      FGD 135
    15. Re:Zonk, you retard by ralphdaugherty · · Score: 1

      When I saw the headline, I first thought of VA Linux-

            When I read it, I first thought Veteran's Administration. VA Linux IT meltdown doesn't even make a lot of sense.

            Of course, I've been reading about federal government IT meltdown's for a long time, so I'm conditioned.

        rd

  11. VA Acronym? by bmomjian · · Score: 2, Insightful

    Isn't it obvious that the acronym "VA" isn't good to use in a title? FYI, it stands for "U.S. Veteran's Administration".

    1. Re:VA Acronym? by RobBebop · · Score: 1

      Mod parent up, not off-topic.

      And tag the article "Veteran's Association" because other applicable acronymns for VA include "VA Software" which is the former name of SourceForge Inc (symb: LNUX), who own Slashdot. Also, even after reading the blurb for the article "Virginia" is a possible acronym for VA.

      Sometimes, it doesn't make sense to shorten things with acronyms. Especially within areas where confusion like this exists.

      --
      Support the 30 Hour Work Week!!!
    2. Re:VA Acronym? by Divide+By+Zero · · Score: 1

      I pedanted this earlier, and at the risk of burning some karma, it really doesn't.

      --
      Dare to Hope. Prepare to be Disappointed.
    3. Re:VA Acronym? by idontgno · · Score: 1

      What's even worse? It's not the VA any more. It's the Department of Veterans Affairs. But no one ever calls it the DVA.

      OK, so maybe the hospital part of this Department thingie is the "VA" in question.

      Nope, sorry, wrong again. That's the Veterans Health Administration, or VHA.

      "Do not try to unfubar the VA; that's impossible. Instead only try to realize the truth: There is no VA."

      --
      Welcome to the Panopticon. Used to be a prison, now it's your home.
  12. Why always centralizing? by guruevi · · Score: 3, Insightful

    I wonder why higher management always wants to centralize their resources. The internet protocol and subsequent many IT applications were built to be efficient in small and decentralized environments.

    1) Trying to centralize gives us large expensive computers that are made out of the same components as smaller ones and thus fail just as the smaller ones do, however, ever trying to cram more crap on the same machine will bring down everything at once whenever it fails.
    2) Trying to centralize has the ultimate goal to eliminate jobs but they need those people since they know all the little details and hickups their systems have. If people know a project is going to eliminate their job, they won't be cooperative. IT not being cooperative is very bad in this world where everything is computerized.
    3) Eventually the same number of people is going to have to work in the centralized system just because you also centralize the problems and more problems will bring more people, more people will bring more overhead and inefficiency, more inefficiency will bring more people (at least that's the default in today's business world, throwing more people at an IT problem doesn't make it disappear faster)
    4) More people in a project that was designed to be more cost efficient means the managers will have to cut expenses. Cut expenses brings underpaid people, underpaid people bring less or no experience and higher turnover, higher turnover means more cutting expenses.

    Therefore: keep your local IT guy(s) and infrastructure although you can't squeeze 100% of work/day and it will bring a little more expense. The end-users have a better relationship with the guy(s) and that makes happier people. Centralizing brings more overhead, less customer-interaction with IT and thus more inefficiency throughout the business.

    --
    Custom electronics and digital signage for your business: www.evcircuits.com
    1. Re:Why always centralizing? by bobaferret · · Score: 2, Insightful

      There are certain this that centraalization brings to the table. Such as this this guy just came into the hospital unconcious , and we know that when he was in a VA hospital accross the country last week he was given a drug that would interact badly with what we want to give him right now. Or what is the chnage in his cat scan since last week whe he was someplace else and had one.

      Obviously not all of this data needs to be centralized, but it's existance should be. We don't know to what level the VA was doing this, but I've met a large number of people who work in it's IT branch, and they love what they do, and are very good at it as well. Sometimes things just go wrong, and sometimes things get pushed out there for beuracratic reasons, but most of the time the VA is very IT savy.

    2. Re:Why always centralizing? by TheSkyIsPurple · · Score: 2, Insightful

      1) Trying to centralize gives us large expensive computers that are made out of the same components as smaller ones and thus fail just as the smaller ones do, however, ever trying to cram more crap on the same machine will bring down everything at once whenever it fails.

      If that's how you're doing it, you're doing it wrong.

      On how many smaller systems can you upgrade your disk controller's firmware without having to reboot or even stop access to the disks? Not a problem on a good SAN system.
      And those systems only get economical when your data storage needs get big.

      2) Trying to centralize has the ultimate goal to eliminate jobs but they need those people since they know all the little details and hickups their systems have. If people know a project is going to eliminate their job, they won't be cooperative. IT not being cooperative is very bad in this world where everything is computerized.

      It doesn't always have that ultimate goal, but very often does. And very often, if done correctly, it can achieve that goal.
      Take 8 sites with 2 admins each that are only doing 50% duty running that service. (You need at least 2 so someone gets to have an occasional vacation).
      That's 16 people, doing the workload of 8.
      Bring that down to 1 site, and odds are you could do the exact same job with 8 people (since now there are 7 others to back you up)

      And now you're all on one system, so you don't have lots of little variances, so you can be more efficient, etc...

      Yes, we have lost some of the little details by losing those people, but in general you've got other problems if some information is only known by one person.
      As it turned out, a lot of that "critical" information got passed along to other folks anyway, most of what was left turned out to be unimportant, and that small remaining percentage?
      Well, the rest of us are smart, and the ones with the info weren't idiots... we were able to figure it out.

      3) Eventually the same number of people is going to have to work in the centralized system just because you also centralize the problems and more problems will bring more people, more people will bring more overhead and inefficiency, more inefficiency will bring more people (at least that's the default in today's business world, throwing more people at an IT problem doesn't make it disappear faster)

      Starting with bad assumptions.
      A small focused skilled team can do pretty much anything. =-)
      In fact some would say they're the only ones who do anything.

      One example: We used to repeatedly run into situations where we had the same problem at x sites, so we had at least x people trying to solve it. We didn't realize other's were duplicating our effort, so there was a lot of wasted effort, with solutions from different angles, so the sites ended up getting more and more out of sync in their setups.

      4) More people in a project that was designed to be more cost efficient means the managers will have to cut expenses. Cut expenses brings underpaid people, underpaid people bring less or no experience and higher turnover, higher turnover means more cutting expenses.

      Every centralization project I've been on has had its hiccups, but in the end has resulted in reduced costs overall. We always started off with the people we had, and a contractor or to who was an "expert" in the field we were working in, just to make sure we had an outsider's view. We didn't always believe the contractor, but we'd at least use them for everything they were worth. We then "centralized", and kept most of the folks around to keep everything running everywhere... then the layoffs.

      The main problem we have on from our last centralization is that many in our small team are very shy about sharing issues before they know everything about it. They're afraid of looking bad, because they won't be as valuable. (Hadn't run into that one before)

    3. Re:Why always centralizing? by CKW · · Score: 1

      150 datacenters.

      You try to be the CIO of any organization with 150 datacenters and 150 IT entities, and say confidently that "yes my patient records are safe" or "yes we're using technology that helps our customers receive the absolutely best care".

      One thing I will say - big companies and organizations definitely have a problem with their network groups. Their network groups have such massive complicated networks that are basically all black boxes - when I'm troubleshooting problems with customers I ALWAYS have to ask them to go check and see if their network group hasn't thrown up another firewall or port rule without telling or asking
      anyone.

      Hmmm, I guess that's change control - the exact problem they had in this case. I just can't
      understand how a port change could cause a cascade of problems of increasing severity and not be
      root-cause analyzed.

    4. Re:Why always centralizing? by Divide+By+Zero · · Score: 1

      You're absolutely right. But.

      Standardizing IT in VA would be fantastically helpful (150-something hospitals all running different software is a nightmare), and they thought that this would be an easy way to do it. Easier, certainly, than managing a 150-site rollout and 150 different migrations all with more than a couple nines of uptime.

      The folks in charge of making this decision work at the pleasure of the President, which means they're looking for work in January of '09. They need something on their resume. That's all this is.

      --
      Dare to Hope. Prepare to be Disappointed.
    5. Re:Why always centralizing? by henryhbk · · Score: 2, Informative
      Centralizing is often highly-advantageous in health care, because you need the records RIGHT NOW and the patients are allowed to go to any VA medical center in the country. I had a VA patient who lost his medications from Alaska while on vacation on the east coast, and I was able to retrieve his medication list push to our clinic's pharmacy in minutes, but it took 5 minutes to pull his records. Now I know most folks laugh, that 5 minutes is considered a long time, but anyone who works in a very busy walk-in clinic where you see patients every 15 minutes, realizes that a 5 minute hit is a giant wrench in to the works.

      You also need to realize that these system use a hierarchical database for speed, so "joins" are much more complicated after the fact, than simply centralizing onto regional servers. Also regionalization is how the VA works as well with local integrated delivery networks (sort of hub-and-spoke with clinics, small and large medical centers) where the patients are often going between institutions in the region.

      I recently left the VA system, and have to say their system is better than almost anyone else's, and has a similar failure rate to those at other institutions; as someone pointed out, it is just that this makes bigger press. In general there are very few health systems which have a single pervasive medical record of this scale (heck most places don't have electronic medical records at all) such as Kaiser Permanente here in the US, and the NHS in the UK, so these stories seem all the more spectacular since they are so rare.

    6. Re:Why always centralizing? by Scrameustache · · Score: 1

      I wonder why higher management always wants to centralize their resources. To reduce costs: Rent and payroll in one building VS rent and payroll in many.
      --

      You can't take the sky from me...

    7. Re:Why always centralizing? by gatesvp · · Score: 1

      Here, here, great post.

      But I will pick at one thing, 3 & 4 are not issues with a unionized organization like VA. Given your example the 7 of the 8 leftover people would be "repurposed" to tackle another massive pile of outstanding work and the 8th one would either get promoted to manage the remaining 7 or they would be "trimmed" and someone else would be promoted to manage the 7.

      My experience with government organizations is that they tend to carry a lot of overhead and they can't really get rid of people. Near-perfect job security and above average pay tends to suck in the "average" workers, with the "best" workers carrying the load of 3 people (and likely there b/c they want the pension and don't want to have to compete on the open market).

      As to the layoffs, if you're in a centralization situation and there is an inkling of layoffs on the horizon, it may be best to become a consultant. They'll likely need your knowledge to perform the centralization so you might as well make the big bucks before you leave :) But that's just my $0.02

  13. All your medical records are belong to us by CranberryKing · · Score: 1

    It's another one of these monstrous systems integration projects that will/never work and every hospital/med center is doing them. They want everything to talk to everything and the only reason behind it is really more big brother Total Info Awaren. BS. I worked for a hospital in the 90's. It was started before I got there and still going on years later after I left. I'm sure it's still going on. It is a monstrous bureaucracy that costs millions (billions?) and you can expect problems of this scale to increase as they continue to centralize their vulnerabilities.

  14. Re:I'm in the outsourcing business by Detritus · · Score: 1
    Smithers, release the hounds.

    Do they teach people how to read these days?

    --
    Mea navis aericumbens anguillis abundat
  15. Re:Good thing they weren't running Ubuntu by Anonymous Coward · · Score: 2, Informative

    UbuntuDupe seems to have a major attitude about Ubuntu. For anyone who doesn't know the story here's why. Basically, UbuntuDupe ran into problems installing Ubuntu and, when he asked for help on Ubuntu Forums, immediately started attacking the people that were sincerely trying to help him. Even with his major attitude the Ubuntu folks still tried their best to help him until they just couldn't put up with him any more. Read it for yourself and you'll see UbuntuDupe's Slashdot postings on Ubuntu in a new light.

  16. +1 C'mon Editors by ggvaidya · · Score: 4, Funny

    I had a real fun time parsing this article.

    1. Looks at title: omg! Slashdot's parent company had an IT meltdown! ha-ha! But waitaminute ...
    2. Looks at icon: a ... crown? The Queen? Perhaps they mean *our* overlords, VA Linux? Or is VA Linux a monarchist organisation now?
    3. Looks at summary: and ... medical? Why are th... oh HANG ON WAIT A MINUTE
    4. Looks at icon: I remember that! It means ... government! Crown, government, get it? So, VA Linux screwed up a government's medical system? That makes ...
    5. Looks into the inner recesses of my mind: ... sense, but ... something's out of place, something's ... just ... not ... quite ...
    7. Looks at lightbulb over head: of course! There *is* no VA Linux! It's Sourceforge, Inc now! But that must mean ...
    6. Looks at summary: ... carefully ... the VA, why the VA, shouldn't it be ... Vir..ginia?!

    Gee thanks, Zonk, just what I needed before going to sleep. Now I'll dream of the Queen in Virginia melting down medical computers for Slashdot's open source overlords. Again.

    Last thing I needed ...

  17. They messed up everything they could mess up. by Skyshadow · · Score: 1, Insightful

    Yeah, time to fire your IT organization's management. And a few of their leads, too. And maybe some of the techs.

    Couple of reasons: First, they're running Vista. I'm not trying to be all "You must only run Linux or ur a n00b" here -- you can run Windows servers just fine, but no reasonable IT planner should ever, *ever* consider using an OS that new for a mission-critical enterprise application. If it doesn't have two or three years in the field, don't even consider it.

    Second, their failover plan sucked. Live data syncs are good for physical disasters (fires, earthquakes, zombie attacks) but, as the VA discovered, they leave you shitting your pants when you run into an issue that may or may not be data-related. The solution to this, of course, is to keep a day or week-old copy someplace along with an up-to-date (but not implemented!) transaction log that you can go through and update with once you've sanity-checked it.

    Third, letting the vendor run "tests" on your production system. Nobody, and I mean nobody, should ever get to touch any production system unless they're implementing a specific change that's been tested in an identical environment, passed QA and review by folks who know the system and then only with a published implementation, testing and backout plan. If a system needs "tests", you pull it out of production before you start messing with it.

    Finally, their "virtualized team" approach (read: our people are scattered all over the place) is moronic -- you see this sort of thing, and without fail it's the result of political pressures rather than sane management. In this case, I'll bet my hat is was a situation where a bunch of middle managers were allowed to maneuver to keep their fingers in the pie when centralization tool place, so instead of having everyone you need on hand and in one group you're busy setting up conference calls.

    Plus, now their solution is to bring in a bunch of consultants. Yeah, that always works. Good luck, guys! You're gonna need it.

    --
    Every year during my review, I just pray the words "slashdot.org" aren't mentioned.
    1. Re:They messed up everything they could mess up. by jkroll · · Score: 1

      Who modded this up?

      Yes the article talks about Vista, but Vista the application as in "Veterans Health Information Systems and Technology Architecture". An application and acronym that predates the Microsoft OS of the same name. I know we like to blame MS for everything, but they have no involvement in this problem.

      Two, their failover plan had three levels of planning. That is far better than most of the failure planning I have seen. As a result of this failure, there were degradations in service because the users weren't able to use the computer systems the way they were used to. Patients records weren't lost or destroyed.

      Three, the vendor tests had nothing to do with the problem, it just meant that there were more people around to throw out opinions. In fact, we don't know what the vendor was testing (perhaps passive monitoring), nor if it was even the same hardware involved in the outage, the article just throws it in there. The actual problem was a failure to follow their procedures and documentation when changing a network port.

      Pay a bit more attention to the article if you are going to make a detailed critique of what these people did.

    2. Re:They messed up everything they could mess up. by Nkwe · · Score: 2, Informative
      Um... they are not running the Vista you think they are. From TFA:

      Vista, Veterans Health Information Systems and Technology Architecture, is the VA's system for maintaining electronic health records. It sounds like they are running something much older. Again from TFA:

      According to Director Eric Raffin, members of the technical team were at the site with staffers from Hewlett-Packard Co. conducting a review of the center's HP AlphaServer system running on Virtual Memory System and testing its performance. "Virtual Memory System" on an Alpha would be "VMS" would it not? Note the article only states that some folks were working on VMS at the same time when the Vista system (not the Microsoft OS) went down. It doesn't say that they were the same system, but you should consider that their environment is a bit more older and complicated that you make it out to be.

      The article also states that the cause was a network configuration error. While in a perfect world you would have test and QA systems that are identical to production, it is rarely feasible. The cost to set up large parallel networks with exactly the same configuration in addition to software with the same configuration is generally cost prohibitive. By "same configuration" I mean same IP addresses, port assignments, routing rules, nationwide WAN links, etc.
    3. Re:They messed up everything they could mess up. by Divide+By+Zero · · Score: 1

      Regarding point the first: Windows Vista != Veterans Healthcare Information Services and Technology Architecture (VistA).

      Regarding point the second: Yup. 100% right.

      Regarding point the third: You are more right than you know.

      Regarding point the last: They weren't middle managers.

      --
      Dare to Hope. Prepare to be Disappointed.
    4. Re:They messed up everything they could mess up. by hanshotfirst · · Score: 1

      Mod down. I stopped reading here... "Couple of reasons: First, they're running Vista. I'm not trying to be all "You must only run Linux or ur a n00b" here -- you can run Windows servers just fine, but no reasonable IT planner should ever, *ever* consider using an OS that new for a mission-critical enterprise application. If it doesn't have two or three years in the field, don't even consider it."

      First off, please read close enough to discern which VISTA they are talking about - it's kinda spelled out there.

      Second, this whole "don't touch it - it's new" mindset chafes me to no end. 2-3 years in IT is an EON. For most products that leaves you at 1 or 2 releases behind, with all the security problems and bugs of 2 years ago. (Yep, that product you rely on today has just as many bugs as that new one on the shelf - they're just DIFFERENT bugs.) A product matures by being IN the field, IN use, and ACTIVELY FIXED in response to support calls. It might surprise you that software companies DO actually test their products before releasing them - no amount of testing can hit every configuration of hardware/software/business needs/obscure requirements out there.

      --
      Why, oh why, didn't I take the Blue Pill?
    5. Re:They messed up everything they could mess up. by Skyshadow · · Score: 1

      Vista in this case doesn't have anything to do with Vista the Microsoft OS. I'm really sort of shocked that none of the above posters caught that.

      --
      Every year during my review, I just pray the words "slashdot.org" aren't mentioned.
    6. Re:They messed up everything they could mess up. by hanshotfirst · · Score: 1

      I'll post this publicly, since my comments were public to begin with... not that anyone else is looking this deep at this point.

      I've got a real problem with the way I responded to your post in the first place. It's been bugging me ever since I sent it - and here's why. I railed against your post in reaction to the numerous times I've faced the "wait until x.1" in the corporate world. Even though I get tired of it after a while, I had no reason to unload my personal baggage on your post, making it seem like a very personal attack on you and your comments.

      I apologize, sincerely. It's not the way I carry myself offline. It's not the way I want to treat people online.

      I can tell by your very low ID that you've been around here much longer than I, and I ought to respect that. It not only shows you've been around the block, it shows you've been dealing with technology long enough to have a solid understanding of these kinds of things, and that probably read more articles before I signed up than I've read in total.

      Again, I offer my apologies for over-reacting.

      --
      Why, oh why, didn't I take the Blue Pill?
  18. Disappointing your employees . . . by corifornia2 · · Score: 1, Interesting

    This is funny to me. I was hired by the VA in St. Petersburg, Florida a few years ago when Windows 2003 first came out to train all of the NT administrators on the migration to 2003. Of the 60 or so NT administrators, all but three of them were losing their title and becoming helpdesk for their site and "physical hands" for the few remaining administrators.

    A lof of the admins were unhappy about that, as I would have been. I am just curious if the failure to complete the project had to do with the lack of respect for the older employees with NT experience and essentially downgrading those employees.

    1. Re:Disappointing your employees . . . by TooMuchToDo · · Score: 1

      Better to be downgraded then out of work. At least it gives you time to find a better gig.

  19. The root cause by paffy · · Score: 1

    Does anybody else get the impression that they created an Ethernet loop and couldn't figure it out for a whole day ?

  20. Poor VMS. by juuri · · Score: 2, Insightful

    staffers from Hewlett-Packard Co. conducting a review of the center's HP AlphaServer system running on Virtual Memory System and testing its performance.

    We hardly knew ye.

    --
    --- I do not moderate.
  21. It happens by ACMENEWSLLC · · Score: 3, Insightful

    What they were doing was a major change to their IT infrastructure. That's massive. Things happen. The fact that they were down at 17 of 128+3 (131) data centers because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.

    Seems to me that things worked otherwise well is a major accomplishment. They are still on the old system and are entering in data back into that system and migrating into the new system. But it seems things went well otherwise.

    Anytime you do a major shift like this, it's hard. The users hate it because they can do their job very quickly on the system they are use to, but now have to learn a new system and slow down.

    Things happen.

    1. Re:It happens by Critical+Facilities · · Score: 2, Interesting

      because some IT staffer changed a port # at one of their hub data centers without following proper procedure -- that's minor.
      I don't know if I agree with that. "Change Control" or "Change Management" is a crucial part of any Data Center. The fact that these ports were changed without being properly "run up the flagpole" is a glaring mistake with very unfortunate results. I'll bet anyone swapping ports in the future will ask permission several times over before trying it again.
  22. I work at the heart of this... by Anonymous Coward · · Score: 5, Insightful

    1st off... VISTA is not Windows VISTA. It's the "Veterans Health Information Systems and Technology Architecture". Do a google search on that.

    VISTA runs on HP's VMS, and on top of that it runs Cache from Intersystems. (And yes it costs the tax payers a lot! But a lot less since we've been centralizing it over the last 3 or 4 years.)

    It is a HUGE system.

    The centralization that we're currently undergoing is massive, this problem was (IMHO) scape goated to a poor change control process.

    I know what was change, I know who changed it, and I know when they changed it. However, this 'melt down' has happened three times... (Not to the same drastic outcome.) It comes down to VMS locking out logons because locks aren't being released properly. (Now you could argue that the reason locks got behind was this change... But I don't think that is the real reason because of our previous problems.)

    It's that simple. Ask the VISTA manager over lunch sometime. They weren't afraid of data corruption. They were afraid if they moved the systems, the other system would lock up too with too much user load.

    There goes "VISTA". Everyone logged in is fine. Everyone not on... Isn't getting on.

    Now comes the bad part... No procedures!

    We take 32 medical centers, and throw their IT into a data center. You 'had' clear lines of who owns what, and what happens when they go down. Now you centralize all that... Who raises the flag when something bad happens? Is it the site that has the problem? Is it someone who now controls the system at the data center? Who is responsible for what?

    Oh wait... OI&T only has a dozen staff... And almost NONE of those people are technical. Everyones pay was simply moved from one appropriation to another. But what about the IT systems?!?! We moved those too, but didn't hire any permanent staff to take care of it? We just rubber banded a bunch of people together that work across the whole west coast and hand them a pager and say good luck?

    Suffice it to say, we have some REALLY REALLY hard working people... And some really bad management. (Congress forcing us to do things on a time table is really annoying. Especially since they expect results, but don't expect any documentation... What do you think is going to get skipped?)

    Congress: How is that data center move going!
    Howard: We've moved 28 sites!
    Congress: Good Job!
    Howard: .:Thinks:. Too bad they don't know about everything we've short changed to make such an obscene deadline!

    Then again... Howard doesn't even know everything we skip to get things done.

    Bah

    1. Re:I work at the heart of this... by mikelieman · · Score: 1

      Why the hell is this modded at zero?

      --
      Technology -- No Place For Wimps! Grateful Dead and Jerry Garcia Chatroom -- http://www.wemissjerry.org
    2. Re:I work at the heart of this... by Divide+By+Zero · · Score: 1

      MOD PARENT UP

      This person knows what's going on and is telling you.

      --
      Dare to Hope. Prepare to be Disappointed.
    3. Re:I work at the heart of this... by Anonymous Coward · · Score: 1, Insightful

      And in reply to the article about data corruption.

      We do have snapshots (I believe the brand naming is snap clones) as well as tape, as well as 'MANY' mirrors of the data.

      There was never a data corruption question. VMS shut out the users. Thats it... No VISTA logon because no VMS logon.

      The cluster and consolidation was tested and ran for over a year on 8 sites before all 32 were planned and rolled into one. (But if you read congress transcripts you would also know we're currently on hold.) We actually had this lock problem in pilot roll out. However the problem was never 'fixed' and still exists in the deployed solution. What causes it is a mystery... It can come and go depending on the lock backup.

      The current plan is to break up the cluster. Not to de-centralize, but to bust this puppy into smaller clusters to limit the number of locks on single files/devices. Not only that... But just running back the journal files, startup scripts, etc, takes WAY too long to be agile!

      (The ultimate goal is to go open source... Shhh don't tell anyone we're trying to save you and me money!)

      This problem escalated out of control because of several contributing factors. Lack of communication, lack of expectations, etc...

      Doctors seem to think you can just flip a switch and move to the other data center. This is NOT the case. Running log files, data checks, etc, etc, it takes well over 4 hours just to 'ramp up' to do the move. Then there is changing DNS and ensuring that every one of our over 50k workstations are using all proper DNS entries etc etc etc.

      It's a complex problem, and frankly... I think our VISTA managers are some of the hardest working underpaid Government employees. (Some of them are only GS-9 and yet equally charged with the care of this huge system.)

      The VA exploits its talented employees, so it doesn't have to manage things that well. People care, and will MAKE it work.

      I get worked up... Because this is SOOOO close to home.

    4. Re:I work at the heart of this... by ralphdaugherty · · Score: 1

      The centralization that we're currently undergoing is massive, this problem was (IMHO) scape goated to a poor change control process.

            Thanks for that excellent insight from the inside.

            First of all, the soft computer press quoting of idiot bigwigs and their scapegoating for whatever they're planning and then again the scapegoating when it fails is irritating, but it's the only semi-technical info available. If it weren't for Slashdot where AC's can dish the dirt we would only have incoherent scapegoating filtered through biased (for anything incompetent executives spin) journalism. Thanks to you and Slashdot once again.

            The other point I'd like to make is that a switchover to another system that is kept synched but also live with it's own critical mass is not acceptable, and didn't happen because of it. One was the rightful concern of bringing down a live system with whatever problem brought your own system down, which they pointed out afterwards, and the other is that which you pointed out, probable overload.

            As everyone here probably knows, the synching needs to take place with a spare system for the High Availability being sought. Transactions of course can also go to the other regions as messages and be processed on an as available basis, but the HA system would be kept synched in real time with ability to fail over to it. This should happen without any concern of locking it up as it's not live with another group of people.

            Yes, the HA spare is somewhat expensive, but vendors typically charge much less on licensing and such for an HA system. The VA needs to implement this remotely from their four regional centers as other large organizations that must keep running do.

            They may even get their oft cited ITIL certification if they put a failover system in place that can actually work.

        rd

    5. Re:I work at the heart of this... by Anonymous Coward · · Score: 1, Interesting

      And let us not forget, the Congress that pushed the IT reorganization on DVA following the theft of the researcher's laptop is the same Congress that has not been able to give the DVA a budget on time at the start of the fiscal year for several years running. Running on a continuing resolution where you have no idea of your budget does not help either. For 25 years or more DHCP/Vista was designed and built as a distributed system. Its redundancy prevented widespread disasters such as this. The headlong drive to combine data centers has now given us more single points of failure combined with a complete dependancy on a telecommunications network structure with myriad points of potential failure. I predict that this will not be the last such widespread failure.

  23. Like the budgie! by PHAEDRU5 · · Score: 2, Insightful

    Cowboy IT people remain employed because they're cheap!

    First thing I learned in the military: your weapon was made by the lowest bidder.

    --
    668: Neighbour of the Beast
  24. Cache is part of the problem by FBodyJim · · Score: 2, Interesting

    I work for a company that uses the Intersystems Cache database and I have to say that I imagine that Cache is a large part of the problem. The amount of good documentation for Cache lies between very little and none and my company has been on a nationwide search for people experienced with Cache and they too seem few and far between. Of course, I don't know that Cache really is a "worse" or "better" database that Oracle, SQL Server or MySQL for that matter, however, what I do know that is when it comes to experience, common tasks, documentation, examples and just getting things done, Cache lags far behind the others, not to mention Universities are still teaching relational db theory, not object db theory, at least when I graduated Rutgers a few short years ago. I suspect that given the task of merging databases, even large databases, there are plenty of experienced and knowledge SQL Server, Oracle, mySQL guys out on Monster or some other job site that know how to get the job done, efficiently and correctly, and have done the job a few times before. Based on our current and past searches for people capable of even easier tasks within Cache, there aren't many people out there with any Cache experience, never mind good people with Cache experience, and it's easy to fudge a task when you aren't given much good documentation, examples or experience. In a past career, I worked for a healthcare company that used SQL server for electronic medical records (EMRs) and the system worked rather well. There might have been better ways to design the database, stored procs or application code, however, we never had a problem hiring good staff that understood the database design, SQL queries, T-SQL/stored procs and as i said, I can't say the same about trying to hire good people who know and understand Mumps ("M" the language, not the disease) or Cache ObjectScript or find the Cache tools to be easy and intuitive. Just my $.02, and I don't mean to start a DB debate, just stating that it might just also be time for the VA's to move on from MUMPS/Cache to a more widely used and documented database and programming language, find some new blood.

    1. Re:Cache is part of the problem by mikelieman · · Score: 2, Informative

      I don't know about that. First, the judicious use of whitespace might help people understand you argument better.

      M is what T-SQL/stored procs wants to be when it grows up. I'm pretty sure getting help from Intersystems isn't an issue at the VA.

      This is a Management/Change Management issue. Not a technical issue.

      --
      Technology -- No Place For Wimps! Grateful Dead and Jerry Garcia Chatroom -- http://www.wemissjerry.org
  25. Re:In other words.... People just don't care. by LWATCDR · · Score: 1

    Welcome to the modern enlightened world. You see way too many people feel that patriotism is stupid. If you join the military you are patriotic and so you are stupid. Why should they care?
    I have seen people refuse to stand for the National Anthem on Veterans day at an airshow. Did you miss the people complaining about Google have a banner for Veterans Day?
    If people will not stand and actively complain about a Google's Veterans Day banner why should they want to fund or fix the VA? That actually costs real money.
    Yea we should fix the VA.

    --
    See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
  26. Oh - *that* VA by NaCh0 · · Score: 1

    I read the title and immediately thought of this VA IT meltdown.

  27. What meltdown? by DynaSoar · · Score: 1

    If the incident actually occurred it made squat difference to treatment. I'm under fairly constant care at two related VA facilities and my treatment wasn't affected by any such thing. Sounds like it's just IT's problem.

    I've gone to VA hospitals since 1989. I got insurance when I started teaching and started going to local doctors and hospitals. Before a year was up I was going back to the VA. Treatment that the VA doesn't provide is treatment the vet didn't request. To be fair, at the VA you need to request harder than elsewhere because elsewhere is going to get money for more treatment and the VA gets nothing different for more or less treatment, except in the largest sense in requesting federal funding after stats show what's been done easily and what took too long to accomplish.

    In 1989 it took months to get an appointment for most things. The excuse was "we don't have enough money". Now if it takes a week, you get the same excuse. If I need to see my primary care physician I can get in same day or next day depending on what time of day I call.

    I am presently getting the best care I've received to date, this at the hospital the VA said had the lowest marks just 3 years ago.

    Sorry you have a problem, IT. You can rest a bit easier knowing it didn't affect the providers.

    --
    "I may be synthetic, but I'm not stupid." -- Bishop 341-B
  28. Re:In other words.... People just don't care. by LWATCDR · · Score: 1

    So have you written or called your rep about? When voting are you going to look at their voting record on VA funding?

    --
    See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
  29. Re:Good thing they weren't running Ubuntu by UbuntuDupe · · Score: 1

    But wouldn't you agree the made-up support team's response was about what I got from the forums?

    -Ask to do stuff I already tried? Check.
    -Pretend like a download/burn failure could cause this specific problem? Check.
    -Give inconsistent story about which CD is needed to fix boot errors? Check.
    -Ignore information about error message? Check.
    -Focus on irrelevant Windows usage? Check.
    -Feigning surprise that someone would run Ubuntu in a completely anticipated, common environment? Check.

    I know a lot of what I've said about Ubuntu has cost me a lot of fans and gained me some freaks. (How mature!) Comments on other issues have been well-respected. Even after the revenge modding I still have excellent karma.

    The reason I keep bringing this up is that people make my exact same design criticisms (that I made on Ubuntu) in many, many other contexts, and then get modded to 5. You'd almost think there was a sacred cow here...

    --
    Apology to Ubuntu forum.
  30. DoD not DVA by QuessFan · · Score: 1

    Urgh!

    Walter Reed ARMY Medical Center is Army/DoD health care for active duty personnels.

    WRAMC may or may not exhibit similar issues with VA hospital system. That remain to be seen.

    However, confusion between the two systems does not help your credibility.

  31. But Walter Reed isn't a VA facility by CapsaicinBoy · · Score: 1

    You are right that the Walter Reed scandal was a travesty. However you are missing one key detail. It ISN'T a VA facility! Walter Reed is an army hospital, meaning it is run by the Department of Defense. The VA and DoD are separate departments, each with their own cabinet secretary.

    Put another way, would it be reasonable or appropriate to blame NIH (Dept of Health and Human Services) for security breaches at Los Alamos (Dept of Energy)? I mean, they both do basic science research, so they must be the same, right?

  32. That's ok by kilodelta · · Score: 1

    There has never been a successful failover test of the RI Department of State's Central Voter Registration System. Most of that is because of the obstinacy of the RI Department of Administration.

    And State recently suffered a MAJOR web outage. Press says it was hacked, I know better. I used to manage that web server before I was summarily laid off. The MySQL database would start going haywire because it was an ancient version. All you had to do was kill the MySQL slave and restart MySQL and all would be fine.