Big HMO Jolted By Email, System Failures

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

Comments

Smoke, meet fire...

[Anonymous Coward]

Here's the first e-mail. Their CEO sent out an e-mail response to all their employees the next day (the same day it looks like the CIO suddenly resigned effective immediately).

I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!

ac

Re:Smoke, meet fire...

[arth1]

To put the 99.2% figure in perspective, it amounts to almost 3 full 24-hour days of downtime a year.

Re:Smoke, meet fire...

[volsung]

I'd like to see more places quote availability numbers as a percentage AND the longest downtime interval in the previous year. 12 minutes of downtime every night at midnight sucks, but 3 solid days of downtime is a disaster.

Re:Smoke, meet fire...

[networkBoy]

but say you can't get antibiotics for a day to treat an infection due to system downtime That would not happen. As a kaiser member I can tell you three things:

• The patient owns their care. If you don't keep track of things then you will not get the care you need. (this is not malice, but rather I suspect kaiser is a victim of their size)
• The doctors and nursing staff want to do their best for the most part, but are overworked, again leading to "you own your own care"
• even when there is an outage, if an ER doc wants meds for a patient, they can get them. The computer system does all the record keeping, the humans still do the medicine, they have paper forms for everything, and when the computer is being an ass, they simply fill out the form like the "old days"

In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
-nB

Re:Smoke, meet fire...

[aaarrrgggh]

I think in 2006 they had 2-3 major incidents in each of their two data centers. Power might have only been out for around 15 minutes each, but system recovery took well over 8 hours for many of the problems. There were also several planned shutdowns for construction activities, which accounted for another 4-5 6 hour scheduled system outages. ...And all the while, IBM is telling them they just need an N+1 UPS system...

Looks like things are getting better now; they have brought in several IT and facilities people from the banking world where there is a better sense of uptime. Unfortunately, the fundamental decision early on in system architecture will make it hard for them to get even 3 9's from Health Connect. They did open a new data center in Napa, but I understand that is an N+1 UPS as well.

Re:Smoke, meet fire...

[azrider]

If you and vendor agreed on downtime before it actually occuring, it is no longer counted against system availability numbers.

From the vendor perspective (on-site maintenance manager for a multi-national bank), we insisted that certain downtime be scheduled (IBM 3800's are *very* picky about when they are fed parts). As a result, during the 18 months I was in charge of one of the 3 data centers, we only dropped below 7/9's one week. Neither the vendor (if ethical) nor the customer wants to see any less. We also insisted that if a scheduled down time period was delayed or cancelled, this would not count against the reports (thankfully, the DC management was clueful enough that they never asked). If KP did not pay enough attention to the criticality of their environment, that signals a major problem at all levels of IT management. In another case (small hospital chain), we had an ongoing problem with the UPS/Generator equipment (switch to backup w/o problem, switch back to mains - down). The DP Manager and I (as the tech specialist assigned to maintain the equipment) spent an entire week onsite just in case - no emergency callouts, no searching to find who to call. Why KP had no clue as to how this is done is troubling.

Re:Smoke, meet fire...

[R2.0]

What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out response and replacement intervals.

For instance, we stated having failures of refrigeration units in a new facility I helped build. Everything was going fine, until we stated getting calls regarding multiple equipment failures and risk to the product inside the unit? We were perplexed - we had built in multiple levels of 100% redundancy (double sized, 2 refrigeration systems per unit, and twice as many units as required for the plant capacity). It turns out that various components had been failing for weeks, but the Facilities department was ignoring them because the backup systems had taken over immediately and there was no imminent risk. It was when the backup systems started failing that we got the frantic calls. There was a systemic problem with the fan motors (didn't use low temperature grease), which the manufacturer fixed promptly, but now the replacements were on an emergency basis instead of planned. When asked why they didn't report the initial failures, the maintenance staff just said "we always run on backups until we can get around to it - what's the use of redundancy if you can't utilize it?"

Like Ron White said, you can't fix stupid.

Epic Systems? No wonder...

[xxxJonBoyxxx]

In a blistering 2,000-word treatise, Mr. Deal wrote: "We're spending recklessly, to the tune of over $1.5 billion in waste every year, primarily on HealthConnect, but also on other inefficient and ineffective information technology projects."

C'mon, it really can't be that bad, can it?

Kaiser refutes Mr. Deal's assessment of its custom software system, developed by Epic Systems Corp.

Oh, Epic Systems? No wonder. Dude, you're f****ed.

Did they perhaps build it in MUMPS?

[Yoozer]

More horrors than you want to imagine.

$4billion for a failing healthcare system?

[simong]

They're just not trying hard enough

Re:Typical of medical and insurance businesses.

[Applekid]

Considering how maintaining accuracy of patient history, treatment records, billing records, allergies, lab reports, and others is paramount I'd say if it ain't broke don't fix it. Could you imagine the change controls in an environment like that? "Potential impact: PEOPLE WILL DIE DIE DIE."

So what?

[0123456]

From what I understand the British government have blown about $25,000,000,000 on a computer system for the National Health Service which doesn't work worth a damn and which the doctors didn't want in the first place.

Private incompetence pales in comparison to government incompetence.

Re:So what?

[sfjoe]

Private incompetence pales in comparison to government incompetence.

Actually, the oposite is true. For example, the US Medicare system spends about 98% of its funding on patient care. Even the largest and most efficient private insurers only can manage about 90%. The radical right-wing wants to preserve the profits of private insurers and so they spend a great deal of their propaganda efforts highlighting mistakes in the public sector so it appears that government is less efficient than it actually is.

Re:So what?

[swillden]

For example, the US Medicare system spends about 98% of its funding on patient care.

Administrative overhead is just about the worst possible measure of efficiency. What you want to measure is amount of needed services delivered for the cost. Medicare's low administrative overhead means that it doesn't have the administrative capacity to avoid paying for unnecessary procedures or to catch providers upcoding visits or procedures or outright inventing them. Even ignoring the question of how much of the services Medicare pays for are actually necessary, Medicare has a huge rate of outright fraud -- around 10% by most estimates, though some industry analysts think it's much higher.

Medicare needs to spend more of its budget on administration, so that it can cut down on the fraud and reduce the amount of unnecessary care that it's paying for. It's entirely likely that if Medicare were to spend 8% of its budget on administration, rather than 2%, its budget could be cut by 15-20% without affecting the quality of care delivered.

Long-term Kaiser patient disputes wild claims

[msblack]

This 45-year old lifetime Kaiser patient believes the media frenzy has blown this out of proportion. Kaiser Permanente undertook the monumental task of converting its patient records system into an electronic information system. Converting the hundreds of paper records for each of millions of patients is truly a monumental task and some problems will occur.

Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.

I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.

Re:Long-term Kaiser patient disputes wild claims

[Mumpsman]

As someone who was up until 1:30 this morning frantically loading RAs into Chronicles in preparation for my Sunday go-live, I really needed to read your comment today. Dealing with Epics goat cluster of poor design decisions can easily lead an IS shop into a downward spiral of negativity and depression, loosing sight of the real purpose - patient care.

Does the "pain" I feel really matter when there are actual sick people up in that building relying on providers who need up-to-date and accurate information to deliver care? Perspective is important. Either that or I am about to lose my mind.

Re:Long-term Kaiser patient disputes wild claims

[juuri]

As a 45 year Kaiser member you should. What they have done and are still tuning is nothing special. Other HMO/MMOs have already done such and many did this years ago for a far cheaper price. Yes, what you saw was convenience yesterday and that is an awesome selling point. What you didn't see what the horrendous roll-out time for this project, the multiple attempts at re-inventing the wheel and the wasted excess in monies spent which is passed on.

It really pains me every-time I see one of these large organizations who outsources these huge projects onto contractors like this. In many cases it would be *just* as fast and more importantly *cheaper* to create a new internal division or ramp up their own IT staff to complete these projects. Outsourcing projects of epic scale is lazy, wasteful and usually involves something shady. I've seen it time and time again in the IT portion of multiple industries and you would think by now, more people would have caught on to what a sham it is.

Re:Typical of medical and insurance businesses.

[prgrmr]

when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.

There are several legacy, "green screen" apps in the insurance industry still kicking around out there. Some of these were initially developed as far back as the late 1970s. They are capable of high transaction rates in comparison to java-driven, web-based, "thin client" applications. Many of today's OLTP applictions would be better suited for ASCI-based interfaces than the GUI-based interfaces, for no other reason than eliminating the constant transition from keyboard to mouse and back.

Re:Epic Systems? No wonder...

[Mumpsman]

Oh, Epic Systems? No wonder. Dude, you're f****ed.

*sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.

Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.

Re:As someone going to a Kaiser appt. today...

[CmdrGravy]

You actually have healthcare in the US ? I didn't realise that, the impression I got is that when you get ill in the States you have to go off to the market rather than a clinic or hospital like you do elsewhere. I guess once you get to market you can swap your cow or goat, or whatever, for advice from the wise women who live there ?

A previous article...

[Mad_Rain]

There is a Computerworld article from the previous slashdot story that seems pretty helpful in understanding the meltdown of their electronic medical records systems. They say that they are running the world's largest Citrix server system, and it does not scale well for their purposes.

As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).

Re:Typical of medical and insurance businesses.

[Enry]

What parent said. WYSE terminals aren't that bad, given that a lot of the medical data industry got it start from what is now known as VistA. By some strange coincidence, the name of its predecessor was DHCP. Anyway, it worked on dirt-old VAX systems and output to text-only terminals over serial lines. Worked just fine.

This all could have been avoided...

[eln]

If only these guys had used Hyperion to manage their data, they wouldn't be having this issue. Hyperion can effortlessly manage up to 12 GIGABYTES of data, and all you have to do is partition it into 3 different pieces.

Hyperion: If it's good enough for Google, it's good enough for you.

Re:Typical of medical and insurance businesses.

[Dr. Manhattan]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices.

I was involved in a project a few years back where a hospital's IT system ran on a bunch of OpenVMS systems, because the main application only ran on OpenVMS. (Which I concede is a capable operating system (logicals are even more powerful than symlinks, etc. etc.) but was outdated even in 2001.) They were also - by a factor of four or so - the largest deployment of the software, so they ran into all the bugs. Downtime was a major problem.

The local admins had some scripts for diagnosing problems, but they only tended to find out about issues once complaints of frozen screens started coming in. I had to develop something that allowed our product to talk to OpenVMS machines and try to detect problems before they snowballed. We succeeded, and downtime was dramatically reduced.

It worked out nicely for us, but even at the time I wondered if such a band-aid should really have been necessary.

Percentages and Marketing speak

[WarlockD]

Dell offers contracts for 99.9% system availability. This means that if you pay the service contract, your within 30 miles of a depot, they guarantee the system won't go down because of hardware for 8.76 hours out of the year. In training we were calculating all that. Technically, it calculates to two "no post" service calls out of the year. So it looks really good when you market 99.9% reliability.

It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.

Hours in a year 8760
99% = 87.6 hours
99.2% = 70.08 hours
99.7% = 26.28 hours

If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.

Re:Percentages and Marketing speak

[NeutronCowboy]

You know what's sad? I started my IT career with Compaq at it's (nee DEC) non-stop computing department. 99.99999% uptime for these suckers. Redundant everything, hotswapping of HDs, all in 1999. Can't believe people are touting 99.9% hardware availability as a success. Not when 99.8% availability for critical apps is becoming a de-facto minimum....

Windows system doesn't scale. What a shock.

[ericferris]

From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.

"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."

So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.

The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.

The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.

Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.

Kaiser Permanente has significant problems

[Pedrito]

Having been, unfortunately, a former user of Kaiser-Permanente's services, I can say with confidence that their problems span way beyond software problems. 7 years ago, I was misdiagnosed with asthma by 3 Kaiser doctors. A 4th finally agreed that I didn't have asthma (and I don't). My actual problem was panic attacks caused by excessive adrenaline production, which no Kaiser doctor was able to determine. I had to go outside of Kaiser to find a doctor competent enough to determine the actual problem and treat me appropriately (with beta-blockers for the adrenaline instead of mind-numbing drugs like Paxil and Valium).

Add to this the reports of KP dumping homeless patients on the streets. There was the kidney transplant scandal. The patient information becoming available online scandal. And all of this in just the past 2 years. Kaiser is clearly flawed from top to bottom. They are the poster-child for real health care reform in this country.

Epic Systems

[DrDitto]

Epic Systems-- selling $4b systems based on clunky, non-relational databases you've never heard of.

Re:Typical of medical and insurance businesses.

[NTT]

Hospitals generally don't keep old computer equipment because they are cheap bastards. Because of the patient safety issues, and related federal/state regulations, most medical related software have *extremely* narrow specs. They keep the old hardware around because the software provider will only certify the product's reliability on a specific class of computer. And this carries over to other responsibilities seemingly unrelated to direct patient care. I once had to maintain a quite old hospital food inventory system because it was the only thing that could talk to the billing system. The billing system was doing it's job, so there was no pressure to upgrade that.

Kaiser sub-standard care.

[TheGeneration]

My father had a heart attack about a month ago. He is a member of Kaiser. Kaiser has it's own hospitals, doctors, and entire network that you must use. His heart attack resulted in him needing to undergo heart surgery. (The advantage to this sort of system is that my father's heart surgeon performed over 1000 bypasses every year and as a result is in the top tier of experts.)

For me the biggest issue came when it was time for discharge. The handed me a hand written sheet with about 12 different medications scribbled in "Doctorese" cursive writing. I couldn't read half of the sheet, and only the briefest of notes for each medication was listed. I was shocked that I didn't receive a computer printout with the medication instructions, especially for medications needed to be taken in the case of quadruple bypass heart surgery.

After many hours of sorting through the medications, trying to figure out if he had already been given a dose at the hospital or not, etc... I realized that when I had visited a non-Kaiser emergency room for a sprained ankle I got a multi-page printout with very thorough instructions for the medication and aftercare. On another visit when I had 107 temperature it was the same thing. I was very certain of the doctors orders for post-visit care.

With Kaiser though I was left confused with short hand written notes about a dozen medications. Confusion is not a state I wanted to be in when I'm charged with the care of one of the most important people in the world (to me) who has a life threatening condition!

Re:What's an HMO?

[binaryspiral]

For the benefit of those of us who aren't Americans, why not say what an HMO is?

If only there was a way for someone to find information on the internet. Like a centralized tool or website that indexes other websites and allows people to search for it when they don't understand.

Or wait, even better - how about an encyclopedia like website that could contain vast amounts of knowledge.

Ah, that would be grand.

Re:Typical of medical and insurance businesses.

[gstoddart]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever.

That's not being cheap. That's being a very conservative organization who can't really gamble on things suddenly not working.

I have seen numerous projects where a company with some older machines (say, mainframes) is being sold a piece of software to replace their 20 year old system -- and replace it with a Windows box or something.

The new system invariably does only about half of what the other system does. Invariably manages to make the process more cumbersome. And, in the end, generally pisses off the user community.

We all like to think in this industry that our new hotness is way better than the old and dusty. The actual case is, many of those old-skool apps which have been around for a while are less visually pleasing, but way more useful than the new hotness.

I'm not saying all modern software is crap, or that you don't eventually need to replace legacy systems. You just frequently end up with something which is not quite what the customer expected it to do -- sometimes, just because the old app has had exceptions built into it for all of the use cases that the analysts never really get to discover until you realize you can't support them.

It's tough to replace a highly specialized app which has been doing it's job for quite a while.

Cheers

Re:Can you say "will never work in IT again" ?

[Wolvie MkM]

Yea... How selfish of him doing the right thing trying to bring to light the potential endangerment of thousands of lives... Big trouble maker.

The world needs more people with spines like this kid, good for him.

Re:Typical of medical and insurance businesses.

[sphealey]

> If you want to see the oldest computer gear simply go to a
> hospital, insurance company or doctors offices. These places
> hate to spend money on IT and let old gear sit in place for
> almost ever.

Except that this article shows why that might be: true critical systems need to be reliable and understandable. Systems that have been in production for many many years often meet those requirements. That is why you often see 3270 green screen applications in large medical offices, and DOS or (old, pre-lawsuit) SCO Unix(tm) or MS-DOS applications in small offices: not because no one wants shiny new toys but because they have been beaten to death for 20 years (or more) and they either work or have known, predictable bugs.

Compare that to the referenced "upgrade" project. The same thing happened to a former employer of mine when they tried to "big bang" their mainframe apps to Windows client/server for Y2K: they couldn't invoice for 7 months. Good thing they had a _large_ line of credit...

The 3rd-to-last issue of the original print Byte had an excellent article about this. They talked to IBM's Director of the OS/nnnn operating systems; he explained how the code in the CPU scheduler had been refined from 1960 to 1980 - at which point it was locked, and will probably never be changed again in human history. Now, that code is written in S/360 machine language and should no doubt be considered "obsolete" - but they aren't going to "upgrade" it.

sPh

Re:Epic Systems? No wonder...

[drgonzo59]

Exactly. Just do a search for Epic Systems employment and you'll find plenty info Epic. I am sure your "sigh" means that you already do know. I wouldn't wish even my enemy to work for Epic Systems. That place is so messed up, it is scary. Their CEO and founder Judy Faulkner is crazy, she decorated the place with bright yellow and blue walls, crazy wierd "art" pieces and she never shows up to interact with anyone and just hides for days on end. That is still the best part. They hire naive fresh grads, promise them work in an "exciting upstart company" (even though it was started in the early 80's) and then gives them a fixed salary and makes them work 60 hour weeks. And they really love to hire H1B workers, they pay for their visas but them make them slave for the company for 5 years minimum. And MUMPS, they use MUMPS! Oh, my God...(I wanted to apply to work there, actually saw their job add on Slashdot and thought I'd give it a try) when I read the description of MUMPS it made Cobol look new and exciting.

I don't know how Epic ever managed to scam Kaiser to secure a contract with them, I am sure Kaiser learned their lesson after this...