Slashdot Mirror


Big HMO Jolted By Email, System Failures

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

13 of 171 comments (clear)

  1. Smoke, meet fire... by Anonymous Coward · · Score: 5, Informative

    Here's the first e-mail. Their CEO sent out an e-mail response to all their employees the next day (the same day it looks like the CIO suddenly resigned effective immediately).

    I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!

    ac

    1. Re:Smoke, meet fire... by arth1 · · Score: 3, Informative

      To put the 99.2% figure in perspective, it amounts to almost 3 full 24-hour days of downtime a year.

    2. Re:Smoke, meet fire... by networkBoy · · Score: 4, Informative

      but say you can't get antibiotics for a day to treat an infection due to system downtime That would not happen. As a kaiser member I can tell you three things:
      • The patient owns their care. If you don't keep track of things then you will not get the care you need. (this is not malice, but rather I suspect kaiser is a victim of their size)
      • The doctors and nursing staff want to do their best for the most part, but are overworked, again leading to "you own your own care"
      • even when there is an outage, if an ER doc wants meds for a patient, they can get them. The computer system does all the record keeping, the humans still do the medicine, they have paper forms for everything, and when the computer is being an ass, they simply fill out the form like the "old days"

      In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
      -nB
      --
      whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
    3. Re:Smoke, meet fire... by aaarrrgggh · · Score: 2, Informative

      I think in 2006 they had 2-3 major incidents in each of their two data centers. Power might have only been out for around 15 minutes each, but system recovery took well over 8 hours for many of the problems. There were also several planned shutdowns for construction activities, which accounted for another 4-5 6 hour scheduled system outages. ...And all the while, IBM is telling them they just need an N+1 UPS system...

      Looks like things are getting better now; they have brought in several IT and facilities people from the banking world where there is a better sense of uptime. Unfortunately, the fundamental decision early on in system architecture will make it hard for them to get even 3 9's from Health Connect. They did open a new data center in Napa, but I understand that is an N+1 UPS as well.

    4. Re:Smoke, meet fire... by R2.0 · · Score: 3, Informative

      What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out response and replacement intervals.

      For instance, we stated having failures of refrigeration units in a new facility I helped build. Everything was going fine, until we stated getting calls regarding multiple equipment failures and risk to the product inside the unit? We were perplexed - we had built in multiple levels of 100% redundancy (double sized, 2 refrigeration systems per unit, and twice as many units as required for the plant capacity). It turns out that various components had been failing for weeks, but the Facilities department was ignoring them because the backup systems had taken over immediately and there was no imminent risk. It was when the backup systems started failing that we got the frantic calls. There was a systemic problem with the fan motors (didn't use low temperature grease), which the manufacturer fixed promptly, but now the replacements were on an emergency basis instead of planned. When asked why they didn't report the initial failures, the maintenance staff just said "we always run on backups until we can get around to it - what's the use of redundancy if you can't utilize it?"

      Like Ron White said, you can't fix stupid.

      --
      "As God is my witness, I thought turkeys could fly." A. Carlson
  2. Long-term Kaiser patient disputes wild claims by msblack · · Score: 4, Informative

    This 45-year old lifetime Kaiser patient believes the media frenzy has blown this out of proportion. Kaiser Permanente undertook the monumental task of converting its patient records system into an electronic information system. Converting the hundreds of paper records for each of millions of patients is truly a monumental task and some problems will occur.

    Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.

    I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.

    --
    signature pending slashdot approval
    1. Re:Long-term Kaiser patient disputes wild claims by Anonymous Coward · · Score: 1, Informative

      As a KP-IT employee it's good to know your visits were made better - not worse - by IT. But....not a single outpatient facility in California has switched to e-prescribing yet....So are you outside California or just our PR department?

  3. Re:Epic Systems? No wonder... by Mumpsman · · Score: 4, Informative

    Oh, Epic Systems? No wonder. Dude, you're f****ed.

    *sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.

    Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.

    --
    No battles to the death are recalled. Mumpsman can hit to attack and cause brainsmashing.
  4. Re:Typical of medical and insurance businesses. by Enry · · Score: 2, Informative

    What parent said. WYSE terminals aren't that bad, given that a lot of the medical data industry got it start from what is now known as VistA. By some strange coincidence, the name of its predecessor was DHCP. Anyway, it worked on dirt-old VAX systems and output to text-only terminals over serial lines. Worked just fine.

  5. Percentages and Marketing speak by WarlockD · · Score: 3, Informative

    Dell offers contracts for 99.9% system availability. This means that if you pay the service contract, your within 30 miles of a depot, they guarantee the system won't go down because of hardware for 8.76 hours out of the year. In training we were calculating all that. Technically, it calculates to two "no post" service calls out of the year. So it looks really good when you market 99.9% reliability.

    It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.

    Hours in a year 8760
    99% = 87.6 hours
    99.2% = 70.08 hours
    99.7% = 26.28 hours

    If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.

  6. Kaiser Permanente has significant problems by Pedrito · · Score: 2, Informative

    Having been, unfortunately, a former user of Kaiser-Permanente's services, I can say with confidence that their problems span way beyond software problems. 7 years ago, I was misdiagnosed with asthma by 3 Kaiser doctors. A 4th finally agreed that I didn't have asthma (and I don't). My actual problem was panic attacks caused by excessive adrenaline production, which no Kaiser doctor was able to determine. I had to go outside of Kaiser to find a doctor competent enough to determine the actual problem and treat me appropriately (with beta-blockers for the adrenaline instead of mind-numbing drugs like Paxil and Valium).

    Add to this the reports of KP dumping homeless patients on the streets. There was the kidney transplant scandal. The patient information becoming available online scandal. And all of this in just the past 2 years. Kaiser is clearly flawed from top to bottom. They are the poster-child for real health care reform in this country.

  7. Kaiser sub-standard care. by TheGeneration · · Score: 2, Informative

    My father had a heart attack about a month ago. He is a member of Kaiser. Kaiser has it's own hospitals, doctors, and entire network that you must use. His heart attack resulted in him needing to undergo heart surgery. (The advantage to this sort of system is that my father's heart surgeon performed over 1000 bypasses every year and as a result is in the top tier of experts.)

    For me the biggest issue came when it was time for discharge. The handed me a hand written sheet with about 12 different medications scribbled in "Doctorese" cursive writing. I couldn't read half of the sheet, and only the briefest of notes for each medication was listed. I was shocked that I didn't receive a computer printout with the medication instructions, especially for medications needed to be taken in the case of quadruple bypass heart surgery.

    After many hours of sorting through the medications, trying to figure out if he had already been given a dose at the hospital or not, etc... I realized that when I had visited a non-Kaiser emergency room for a sprained ankle I got a multi-page printout with very thorough instructions for the medication and aftercare. On another visit when I had 107 temperature it was the same thing. I was very certain of the doctors orders for post-visit care.

    With Kaiser though I was left confused with short hand written notes about a dozen medications. Confusion is not a state I wanted to be in when I'm charged with the care of one of the most important people in the world (to me) who has a life threatening condition!

    --


    The Generation
    I'd say something witty here, but I'm not that bright.
  8. Re:What's an HMO? by Anonymous Coward · · Score: 1, Informative

    Most Americans are probably unaware or unsure what HMO stands for, but we all know it is: a type of health care service; restricts members to a list of participating doctors/medical practices; usually requires a co-pay; generally requires a referral from a "primary care provider" to see a specialist; and while being far from ideal it is the only affordable option for health care thanks to Hillary Clinton's closed door policy drafting sessions with the big health care providers during her husbands presidential administration.