Slashdot Mirror

← Back to Stories (view on slashdot.org)

Big HMO Jolted By Email, System Failures

Posted by Zonk on from the so-proud-to-have-epic-systems-in-madison dept.

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

13 of 171 comments (clear)

  1. Smoke, meet fire... by Anonymous Coward · · Score: 5, Informative

    Here's the first e-mail. Their CEO sent out an e-mail response to all their employees the next day (the same day it looks like the CIO suddenly resigned effective immediately).

    I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!

    ac

    1. Re:Smoke, meet fire... by arth1 · · Score: 3, Informative

      To put the 99.2% figure in perspective, it amounts to almost 3 full 24-hour days of downtime a year.

    2. Re:Smoke, meet fire... by networkBoy · · Score: 4, Informative

      but say you can't get antibiotics for a day to treat an infection due to system downtime That would not happen. As a kaiser member I can tell you three things:
      • The patient owns their care. If you don't keep track of things then you will not get the care you need. (this is not malice, but rather I suspect kaiser is a victim of their size)
      • The doctors and nursing staff want to do their best for the most part, but are overworked, again leading to "you own your own care"
      • even when there is an outage, if an ER doc wants meds for a patient, they can get them. The computer system does all the record keeping, the humans still do the medicine, they have paper forms for everything, and when the computer is being an ass, they simply fill out the form like the "old days"

      In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
      -nB
      --
      whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
    3. Re:Smoke, meet fire... by R2.0 · · Score: 3, Informative

      What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out response and replacement intervals.

      For instance, we stated having failures of refrigeration units in a new facility I helped build. Everything was going fine, until we stated getting calls regarding multiple equipment failures and risk to the product inside the unit? We were perplexed - we had built in multiple levels of 100% redundancy (double sized, 2 refrigeration systems per unit, and twice as many units as required for the plant capacity). It turns out that various components had been failing for weeks, but the Facilities department was ignoring them because the backup systems had taken over immediately and there was no imminent risk. It was when the backup systems started failing that we got the frantic calls. There was a systemic problem with the fan motors (didn't use low temperature grease), which the manufacturer fixed promptly, but now the replacements were on an emergency basis instead of planned. When asked why they didn't report the initial failures, the maintenance staff just said "we always run on backups until we can get around to it - what's the use of redundancy if you can't utilize it?"

      Like Ron White said, you can't fix stupid.

      --
      "As God is my witness, I thought turkeys could fly." A. Carlson
  2. Epic Systems? No wonder... by xxxJonBoyxxx · · Score: 4, Funny

    In a blistering 2,000-word treatise, Mr. Deal wrote: "We're spending recklessly, to the tune of over $1.5 billion in waste every year, primarily on HealthConnect, but also on other inefficient and ineffective information technology projects."


    C'mon, it really can't be that bad, can it?

    Kaiser refutes Mr. Deal's assessment of its custom software system, developed by Epic Systems Corp.


    Oh, Epic Systems? No wonder. Dude, you're f****ed.
  3. Long-term Kaiser patient disputes wild claims by msblack · · Score: 4, Informative

    This 45-year old lifetime Kaiser patient believes the media frenzy has blown this out of proportion. Kaiser Permanente undertook the monumental task of converting its patient records system into an electronic information system. Converting the hundreds of paper records for each of millions of patients is truly a monumental task and some problems will occur.

    Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.

    I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.

    --
    signature pending slashdot approval
    1. Re:Long-term Kaiser patient disputes wild claims by Mumpsman · · Score: 3, Interesting

      As someone who was up until 1:30 this morning frantically loading RAs into Chronicles in preparation for my Sunday go-live, I really needed to read your comment today. Dealing with Epics goat cluster of poor design decisions can easily lead an IS shop into a downward spiral of negativity and depression, loosing sight of the real purpose - patient care.

      Does the "pain" I feel really matter when there are actual sick people up in that building relying on providers who need up-to-date and accurate information to deliver care? Perspective is important. Either that or I am about to lose my mind.

      --
      No battles to the death are recalled. Mumpsman can hit to attack and cause brainsmashing.
  4. Re:Epic Systems? No wonder... by Mumpsman · · Score: 4, Informative

    Oh, Epic Systems? No wonder. Dude, you're f****ed.

    *sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.

    Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.

    --
    No battles to the death are recalled. Mumpsman can hit to attack and cause brainsmashing.
  5. A previous article... by Mad_Rain · · Score: 4, Interesting

    There is a Computerworld article from the previous slashdot story that seems pretty helpful in understanding the meltdown of their electronic medical records systems. They say that they are running the world's largest Citrix server system, and it does not scale well for their purposes.

    As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).

    --
    "What do you think?" "I think 'What, do you think?!'"
  6. This all could have been avoided... by eln · · Score: 3, Funny

    If only these guys had used Hyperion to manage their data, they wouldn't be having this issue. Hyperion can effortlessly manage up to 12 GIGABYTES of data, and all you have to do is partition it into 3 different pieces.

    Hyperion: If it's good enough for Google, it's good enough for you.

  7. Percentages and Marketing speak by WarlockD · · Score: 3, Informative

    Dell offers contracts for 99.9% system availability. This means that if you pay the service contract, your within 30 miles of a depot, they guarantee the system won't go down because of hardware for 8.76 hours out of the year. In training we were calculating all that. Technically, it calculates to two "no post" service calls out of the year. So it looks really good when you market 99.9% reliability.

    It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.

    Hours in a year 8760
    99% = 87.6 hours
    99.2% = 70.08 hours
    99.7% = 26.28 hours

    If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.

  8. Windows system doesn't scale. What a shock. by ericferris · · Score: 5, Insightful

    From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.

    "We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."

    So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.

    The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.

    The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.

    Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.

    --
    Fantasy: http://ferrisfantasy.blogspot.com/
  9. Re:What's an HMO? by binaryspiral · · Score: 3, Funny

    For the benefit of those of us who aren't Americans, why not say what an HMO is?

    If only there was a way for someone to find information on the internet. Like a centralized tool or website that indexes other websites and allows people to search for it when they don't understand.

    Or wait, even better - how about an encyclopedia like website that could contain vast amounts of knowledge.

    Ah, that would be grand.