Slashdot Mirror
Big HMO Jolted By Email, System Failures
JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."
Considering how maintaining accuracy of patient history, treatment records, billing records, allergies, lab reports, and others is paramount I'd say if it ain't broke don't fix it. Could you imagine the change controls in an environment like that? "Potential impact: PEOPLE WILL DIE DIE DIE."
More Twoson than Cupertino
From what I understand the British government have blown about $25,000,000,000 on a computer system for the National Health Service which doesn't work worth a damn and which the doctors didn't want in the first place.
Private incompetence pales in comparison to government incompetence.
when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.
There are several legacy, "green screen" apps in the insurance industry still kicking around out there. Some of these were initially developed as far back as the late 1970s. They are capable of high transaction rates in comparison to java-driven, web-based, "thin client" applications. Many of today's OLTP applictions would be better suited for ASCI-based interfaces than the GUI-based interfaces, for no other reason than eliminating the constant transition from keyboard to mouse and back.
I'd like to see more places quote availability numbers as a percentage AND the longest downtime interval in the previous year. 12 minutes of downtime every night at midnight sucks, but 3 solid days of downtime is a disaster.
From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.
"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."
So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.
The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.
The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.
Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.
Fantasy: http://ferrisfantasy.blogspot.com/
Hospitals generally don't keep old computer equipment because they are cheap bastards. Because of the patient safety issues, and related federal/state regulations, most medical related software have *extremely* narrow specs. They keep the old hardware around because the software provider will only certify the product's reliability on a specific class of computer. And this carries over to other responsibilities seemingly unrelated to direct patient care. I once had to maintain a quite old hospital food inventory system because it was the only thing that could talk to the billing system. The billing system was doing it's job, so there was no pressure to upgrade that.
As a 45 year Kaiser member you should. What they have done and are still tuning is nothing special. Other HMO/MMOs have already done such and many did this years ago for a far cheaper price. Yes, what you saw was convenience yesterday and that is an awesome selling point. What you didn't see what the horrendous roll-out time for this project, the multiple attempts at re-inventing the wheel and the wasted excess in monies spent which is passed on.
It really pains me every-time I see one of these large organizations who outsources these huge projects onto contractors like this. In many cases it would be *just* as fast and more importantly *cheaper* to create a new internal division or ramp up their own IT staff to complete these projects. Outsourcing projects of epic scale is lazy, wasteful and usually involves something shady. I've seen it time and time again in the IT portion of multiple industries and you would think by now, more people would have caught on to what a sham it is.
--- I do not moderate.
she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.
...squandering all their money on the "latest, greatest" bleeding edge, blaze-the-pioneer-trail technology that keeps them on a perpetual upgrade and replace treadmill that only does good to keep the corporate appetite of their IT vendors satisfied.
An enterprise-level, mission-critical system for a huge company should only be implemented if it has an expected useful service life of no less than 15 years from go-live until replacement. And at 5 years prior to that replacement date, the wheels should be set into motion for the next new replacement system becuase it really does take that long to make a full transition.
"Big Business" is sick and tired of the IT industry forcing premature obsolescence down their throats. Expect things to come back round full-circle to the way things used to be when IBM was the king, systems were expensive as hell, tech staff was expensive as hell, but the systems and the people who ran them would stick around for longer than a flash-in-the-pan moment.
You know what's sad? I started my IT career with Compaq at it's (nee DEC) non-stop computing department. 99.99999% uptime for these suckers. Redundant everything, hotswapping of HDs, all in 1999. Can't believe people are touting 99.9% hardware availability as a success. Not when 99.8% availability for critical apps is becoming a de-facto minimum....
Those who can, do. Those who can't, sue.
That's not being cheap. That's being a very conservative organization who can't really gamble on things suddenly not working.
I have seen numerous projects where a company with some older machines (say, mainframes) is being sold a piece of software to replace their 20 year old system -- and replace it with a Windows box or something.
The new system invariably does only about half of what the other system does. Invariably manages to make the process more cumbersome. And, in the end, generally pisses off the user community.
We all like to think in this industry that our new hotness is way better than the old and dusty. The actual case is, many of those old-skool apps which have been around for a while are less visually pleasing, but way more useful than the new hotness.
I'm not saying all modern software is crap, or that you don't eventually need to replace legacy systems. You just frequently end up with something which is not quite what the customer expected it to do -- sometimes, just because the old app has had exceptions built into it for all of the use cases that the analysts never really get to discover until you realize you can't support them.
It's tough to replace a highly specialized app which has been doing it's job for quite a while.
Cheers
Lost at C:>. Found at C.
As the saying goes, sufficiently advanced incompetence is indistinguishable from malice. Although, as a former kaiser member myself, I agree that they aren't that bad.
Yea... How selfish of him doing the right thing trying to bring to light the potential endangerment of thousands of lives... Big trouble maker.
The world needs more people with spines like this kid, good for him.
I Like Pie...