Big HMO Jolted By Email, System Failures

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

Comments

Typical of medical and insurance businesses.

[Lumpy]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever. A local hospital here has several Windows 98 pc's still in use. Most doctors offices have antiquated IT gear (Dentists as well) and when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.

Re:Typical of medical and insurance businesses.

[Dr. Manhattan]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices.

I was involved in a project a few years back where a hospital's IT system ran on a bunch of OpenVMS systems, because the main application only ran on OpenVMS. (Which I concede is a capable operating system (logicals are even more powerful than symlinks, etc. etc.) but was outdated even in 2001.) They were also - by a factor of four or so - the largest deployment of the software, so they ran into all the bugs. Downtime was a major problem.

The local admins had some scripts for diagnosing problems, but they only tended to find out about issues once complaints of frozen screens started coming in. I had to develop something that allowed our product to talk to OpenVMS machines and try to detect problems before they snowballed. We succeeded, and downtime was dramatically reduced.

It worked out nicely for us, but even at the time I wondered if such a band-aid should really have been necessary.

Re:Typical of medical and insurance businesses.

[sphealey]

> If you want to see the oldest computer gear simply go to a
> hospital, insurance company or doctors offices. These places
> hate to spend money on IT and let old gear sit in place for
> almost ever.

Except that this article shows why that might be: true critical systems need to be reliable and understandable. Systems that have been in production for many many years often meet those requirements. That is why you often see 3270 green screen applications in large medical offices, and DOS or (old, pre-lawsuit) SCO Unix(tm) or MS-DOS applications in small offices: not because no one wants shiny new toys but because they have been beaten to death for 20 years (or more) and they either work or have known, predictable bugs.

Compare that to the referenced "upgrade" project. The same thing happened to a former employer of mine when they tried to "big bang" their mainframe apps to Windows client/server for Y2K: they couldn't invoice for 7 months. Good thing they had a _large_ line of credit...

The 3rd-to-last issue of the original print Byte had an excellent article about this. They talked to IBM's Director of the OS/nnnn operating systems; he explained how the code in the CPU scheduler had been refined from 1960 to 1980 - at which point it was locked, and will probably never be changed again in human history. Now, that code is written in S/360 machine language and should no doubt be considered "obsolete" - but they aren't going to "upgrade" it.

sPh

As someone going to a Kaiser appt. today...

[EccentricAnomaly]

The problems with the new Kaiser software are obvious to anyone who's been to Kaiser recently or spoken to a doctor or nurses who work there. Test results disappear, appointments disappear.... sometimes the people on the phone can't schedule appointments at all and tell people to call back later.

A previous article...

[Mad_Rain]

There is a Computerworld article from the previous slashdot story that seems pretty helpful in understanding the meltdown of their electronic medical records systems. They say that they are running the world's largest Citrix server system, and it does not scale well for their purposes.

As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).

Re:Long-term Kaiser patient disputes wild claims

[Mumpsman]

As someone who was up until 1:30 this morning frantically loading RAs into Chronicles in preparation for my Sunday go-live, I really needed to read your comment today. Dealing with Epics goat cluster of poor design decisions can easily lead an IS shop into a downward spiral of negativity and depression, loosing sight of the real purpose - patient care.

Does the "pain" I feel really matter when there are actual sick people up in that building relying on providers who need up-to-date and accurate information to deliver care? Perspective is important. Either that or I am about to lose my mind.

Re:Smoke, meet fire...

[azrider]

If you and vendor agreed on downtime before it actually occuring, it is no longer counted against system availability numbers.

From the vendor perspective (on-site maintenance manager for a multi-national bank), we insisted that certain downtime be scheduled (IBM 3800's are *very* picky about when they are fed parts). As a result, during the 18 months I was in charge of one of the 3 data centers, we only dropped below 7/9's one week. Neither the vendor (if ethical) nor the customer wants to see any less. We also insisted that if a scheduled down time period was delayed or cancelled, this would not count against the reports (thankfully, the DC management was clueful enough that they never asked). If KP did not pay enough attention to the criticality of their environment, that signals a major problem at all levels of IT management. In another case (small hospital chain), we had an ongoing problem with the UPS/Generator equipment (switch to backup w/o problem, switch back to mains - down). The DP Manager and I (as the tech specialist assigned to maintain the equipment) spent an entire week onsite just in case - no emergency callouts, no searching to find who to call. Why KP had no clue as to how this is done is troubling.

Re:Epic Systems? No wonder...

[drgonzo59]

Exactly. Just do a search for Epic Systems employment and you'll find plenty info Epic. I am sure your "sigh" means that you already do know. I wouldn't wish even my enemy to work for Epic Systems. That place is so messed up, it is scary. Their CEO and founder Judy Faulkner is crazy, she decorated the place with bright yellow and blue walls, crazy wierd "art" pieces and she never shows up to interact with anyone and just hides for days on end. That is still the best part. They hire naive fresh grads, promise them work in an "exciting upstart company" (even though it was started in the early 80's) and then gives them a fixed salary and makes them work 60 hour weeks. And they really love to hire H1B workers, they pay for their visas but them make them slave for the company for 5 years minimum. And MUMPS, they use MUMPS! Oh, my God...(I wanted to apply to work there, actually saw their job add on Slashdot and thought I'd give it a try) when I read the description of MUMPS it made Cobol look new and exciting.

I don't know how Epic ever managed to scam Kaiser to secure a contract with them, I am sure Kaiser learned their lesson after this...

Re:So what?

[swillden]

For example, the US Medicare system spends about 98% of its funding on patient care.

Administrative overhead is just about the worst possible measure of efficiency. What you want to measure is amount of needed services delivered for the cost. Medicare's low administrative overhead means that it doesn't have the administrative capacity to avoid paying for unnecessary procedures or to catch providers upcoding visits or procedures or outright inventing them. Even ignoring the question of how much of the services Medicare pays for are actually necessary, Medicare has a huge rate of outright fraud -- around 10% by most estimates, though some industry analysts think it's much higher.

Medicare needs to spend more of its budget on administration, so that it can cut down on the fraud and reduce the amount of unnecessary care that it's paying for. It's entirely likely that if Medicare were to spend 8% of its budget on administration, rather than 2%, its budget could be cut by 15-20% without affecting the quality of care delivered.