Big HMO Jolted By Email, System Failures

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

Comments

Smoke, meet fire...

[Anonymous Coward]

Here's the first e-mail. Their CEO sent out an e-mail response to all their employees the next day (the same day it looks like the CIO suddenly resigned effective immediately).

I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!

ac

Re:Smoke, meet fire...

[arth1]

To put the 99.2% figure in perspective, it amounts to almost 3 full 24-hour days of downtime a year.

Re:Smoke, meet fire...

[volsung]

I'd like to see more places quote availability numbers as a percentage AND the longest downtime interval in the previous year. 12 minutes of downtime every night at midnight sucks, but 3 solid days of downtime is a disaster.

Re:Smoke, meet fire...

[cuncator]

No joke! 99.7% isn't that great either for that matter. What ever happened to five or six 9s? Especially for something as critical as health care I would think even four 9s would be the least they'd aim for. Nobody's seriously injured if homestarrunner.com has 8 hours of downtime with four 9s (99.99% uptime), but say you can't get antibiotics for a day to treat an infection due to system downtime. That could be the difference between keeping and losing a limb.

Re:Smoke, meet fire...

[Reapman]

The "five 9's" and all that is a joke anyways. We have that with our vendor... so that means we can have an office down for a solid week and they still met their goals. Just because a contract is "xyz 9's" doesn't mean jake without more info.

Re:Smoke, meet fire...

[sgtrock]

Then you defined your contract terms wrong, didn't you?

Re:Smoke, meet fire...

[networkBoy]

but say you can't get antibiotics for a day to treat an infection due to system downtime That would not happen. As a kaiser member I can tell you three things:

• The patient owns their care. If you don't keep track of things then you will not get the care you need. (this is not malice, but rather I suspect kaiser is a victim of their size)
• The doctors and nursing staff want to do their best for the most part, but are overworked, again leading to "you own your own care"
• even when there is an outage, if an ER doc wants meds for a patient, they can get them. The computer system does all the record keeping, the humans still do the medicine, they have paper forms for everything, and when the computer is being an ass, they simply fill out the form like the "old days"

In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
-nB

Re:Smoke, meet fire...

[aaarrrgggh]

I think in 2006 they had 2-3 major incidents in each of their two data centers. Power might have only been out for around 15 minutes each, but system recovery took well over 8 hours for many of the problems. There were also several planned shutdowns for construction activities, which accounted for another 4-5 6 hour scheduled system outages. ...And all the while, IBM is telling them they just need an N+1 UPS system...

Looks like things are getting better now; they have brought in several IT and facilities people from the banking world where there is a better sense of uptime. Unfortunately, the fundamental decision early on in system architecture will make it hard for them to get even 3 9's from Health Connect. They did open a new data center in Napa, but I understand that is an N+1 UPS as well.

Re:Smoke, meet fire...

[644bd346996]

As the saying goes, sufficiently advanced incompetence is indistinguishable from malice. Although, as a former kaiser member myself, I agree that they aren't that bad.

Re:Smoke, meet fire...

[a_ghostwheel]

Contract like this usually includes wording along the lines "99.99% not including scheduled downtime for maintenance and other activities" - keyword being "scheduled". If you and vendor agreed on downtime before it actually occuring, it is no longer counted against system availability numbers. After that it is all about political games - like vendor calling and pushing you to schedule downtime to apply some sort of a systemwide patch that they consider necessary.

Re:Smoke, meet fire...

[azrider]

If you and vendor agreed on downtime before it actually occuring, it is no longer counted against system availability numbers.

From the vendor perspective (on-site maintenance manager for a multi-national bank), we insisted that certain downtime be scheduled (IBM 3800's are *very* picky about when they are fed parts). As a result, during the 18 months I was in charge of one of the 3 data centers, we only dropped below 7/9's one week. Neither the vendor (if ethical) nor the customer wants to see any less. We also insisted that if a scheduled down time period was delayed or cancelled, this would not count against the reports (thankfully, the DC management was clueful enough that they never asked). If KP did not pay enough attention to the criticality of their environment, that signals a major problem at all levels of IT management. In another case (small hospital chain), we had an ongoing problem with the UPS/Generator equipment (switch to backup w/o problem, switch back to mains - down). The DP Manager and I (as the tech specialist assigned to maintain the equipment) spent an entire week onsite just in case - no emergency callouts, no searching to find who to call. Why KP had no clue as to how this is done is troubling.

Re:Smoke, meet fire...

[EvilTwinSkippy]

My guess: They hired some bright young faces from right out of school.

The last 3 jobs I applied for did! (Scary: 2 were government, and one was a company that designed databases for container shipping.)

Re:Smoke, meet fire...

[bsane]

The patient owns their care. If you don't keep track of things then you will not get the care you need.

Now I realize that this is true to _some_ extent everywhere, just because of reality, I really hope that this isn't part of the culture or procedure at Kaiser facilities.

Many people recieving health care do not have the capability to 'own' their profesional care. Its great that some people (and/or thier families) can do this, but it would be a shame if anyone who can't is simply thrown to the wolves.

Re:Smoke, meet fire...

[eyeye]

I've been reading up on this and it is disgusting, it is a not for profit organisation but cronyism is siphoning millions or billions out of it. Someone with a wikepedia account should note this controversy on the Kaiser Permanente page.

Re:Smoke, meet fire...

[networkBoy]

Responding to you and the AC above who claims to be a doctor, but fails to read what they are responding to...

Owning your own care, while not a culture or expectation at kaiser, is a fact of life. The staff is overworked, and the number one gripe I hear from staff is about the computer system. They find out where I work, and I get unloaded on...
It's simply that the staff has more to do than they can realistically cover. My kids' pediatrician is great, but if you don't get a morning appointment you will be waiting well beyond your appointment time as the appointments are scheduled too close to each other and schedule slippage takes its toll.
-nB

Re:Smoke, meet fire...

[sgt_doom]

As a former client of Kaiser, this is the absolute worst, most incompetent outfit which has ever existed.

The patient owns their care.

Hell, the patient has not only do everything themselves, but to do the same thing over and over and over and over and over...starting to get the picture here? Because those clowns lose all medical records, all diagnostic data, all everything, over and over and over....well, I'm sure you're getting the picture by this time......

Re:Smoke, meet fire...

[R2.0]

What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out response and replacement intervals.

For instance, we stated having failures of refrigeration units in a new facility I helped build. Everything was going fine, until we stated getting calls regarding multiple equipment failures and risk to the product inside the unit? We were perplexed - we had built in multiple levels of 100% redundancy (double sized, 2 refrigeration systems per unit, and twice as many units as required for the plant capacity). It turns out that various components had been failing for weeks, but the Facilities department was ignoring them because the backup systems had taken over immediately and there was no imminent risk. It was when the backup systems started failing that we got the frantic calls. There was a systemic problem with the fan motors (didn't use low temperature grease), which the manufacturer fixed promptly, but now the replacements were on an emergency basis instead of planned. When asked why they didn't report the initial failures, the maintenance staff just said "we always run on backups until we can get around to it - what's the use of redundancy if you can't utilize it?"

Like Ron White said, you can't fix stupid.

Re:Smoke, meet fire...

[R2.0]

Funny, I switched *to* KP due to the actions of my long-time insurer, BC/BS. Why? They went through a cost cutting splurge and consolidated claims processing centers. As a result, claims submitted under one version of the plan would be processed (and subsequently denied) under another plan. I went through 2 years of claims volleyball:
- BCBS says provider screwed up and needs to resubmit
- Provider provides evidence their paperwork is correct, and they are not going to resubmit
- Patient resubmits, and is told by BCBS that only the Provider is allowed to submit
- Repeat.

I had to give up a long-term relationship with a health care provider because of this. The quality of care was great, but I was effectively denied access to it because BCBS wouldn't honor their contract.

KP can be a pain in the ass, but I've never been told not to come back until I had my insurance sorted out.

Re:Smoke, meet fire...

[Reapman]

Like I had any say on this contract haha, not saying I know how to write em better then our yahoo's did, but the vendor definitely won this contract.

Manager's seem to love (or did) that five 9's lingo... i'm trying to point out that by itself means nothing. It's like saying my OS is rock solid.. without mentioning it's also not running anything at all.

Re:Smoke, meet fire...

[aaarrrgggh]

What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out response and replacement intervals.

N+1 Redundancy is appropriate for systems where individual pieces of equipment require significant maintenance, like a pump or chiller. The redundancy is provided for the purposes of concurrent maintainability. N+1 Redundancy does not do much for improving overall reliability.

To improve reliability, you must introduce fault tolerance. Since an N+1 system always has single points of failure, it cannot tolerate a complete range of faults to a system.

In your example, redundant CRAC units are useful in allowing maintenance of a single unit, or supporting a failure of a single unit. Maintenance wasn't "stupid"; they reasonably counted on the system redundancy. Unfortunately, the scenario you describe should require escalation by maintenance to identify hightened risk for business units. Some contingency planning, like move-n-cool's could have been provisioned if deemed necessary.

The point I originally was trying to make was that when you have just IT people looking at a problem, they often fail to realize operational issues. IBM has often made this mistake with their consulting clients. They are just interested in selling more hardware and consulting services.

Typical of medical and insurance businesses.

[Lumpy]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever. A local hospital here has several Windows 98 pc's still in use. Most doctors offices have antiquated IT gear (Dentists as well) and when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.

Re:Typical of medical and insurance businesses.

[InsaneProcessor]

You must be talking about facilities that are primarily HMOs. Anyone in an HMO is either dirt poor and has no choice or a total idiot I was in one once and opted to buy my own medical insurance instead..

The doctors, dentist and hospital I use have the newest data base systems and computers. Every time I am in the hospital, I see more new equipment.

Re:Typical of medical and insurance businesses.

[lucabrasi999]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices

While the use of old technology at a physicians office may be true, I wouldn't generalize hospitals in that category. I have consulted at two different hospital systems in the past six years. At each location, the hospital was pushing to the forefront of technology. My last hospital was in the early stages of virtualizing all of their hardware environment and was using the most recent UNIX hardware available.

Just like all organizations, some are technologically advanced, some are behind the times.

Re:Typical of medical and insurance businesses.

[Applekid]

Considering how maintaining accuracy of patient history, treatment records, billing records, allergies, lab reports, and others is paramount I'd say if it ain't broke don't fix it. Could you imagine the change controls in an environment like that? "Potential impact: PEOPLE WILL DIE DIE DIE."

Re:Typical of medical and insurance businesses.

[smooth wombat]

Anyone in an HMO is either dirt poor and has no choice or a total idiot

Or option 3, has no choice but to be in an HMO because that's all their employer offers.

I'm in that situation and almost all medical expenses are taken care of. Then again, I'm a basic user. Twice a year dentist check is all I need.

Granted, I wish I could opt out of paying for any medical and get the money in hand and pay my own, rare, expenses but I don't have that option. It would be cheaper for me to get the money and pay when I go to a visit than it is to piss away money every paycheck and not use it.

While HMOs have a bad reputation, if you're a basic user, you can generally find decent medical care.

Re:Typical of medical and insurance businesses.

[prgrmr]

when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.

There are several legacy, "green screen" apps in the insurance industry still kicking around out there. Some of these were initially developed as far back as the late 1970s. They are capable of high transaction rates in comparison to java-driven, web-based, "thin client" applications. Many of today's OLTP applictions would be better suited for ASCI-based interfaces than the GUI-based interfaces, for no other reason than eliminating the constant transition from keyboard to mouse and back.

Re:Typical of medical and insurance businesses.

[Enry]

What parent said. WYSE terminals aren't that bad, given that a lot of the medical data industry got it start from what is now known as VistA. By some strange coincidence, the name of its predecessor was DHCP. Anyway, it worked on dirt-old VAX systems and output to text-only terminals over serial lines. Worked just fine.

Re:Typical of medical and insurance businesses.

[Dr. Manhattan]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices.

I was involved in a project a few years back where a hospital's IT system ran on a bunch of OpenVMS systems, because the main application only ran on OpenVMS. (Which I concede is a capable operating system (logicals are even more powerful than symlinks, etc. etc.) but was outdated even in 2001.) They were also - by a factor of four or so - the largest deployment of the software, so they ran into all the bugs. Downtime was a major problem.

The local admins had some scripts for diagnosing problems, but they only tended to find out about issues once complaints of frozen screens started coming in. I had to develop something that allowed our product to talk to OpenVMS machines and try to detect problems before they snowballed. We succeeded, and downtime was dramatically reduced.

It worked out nicely for us, but even at the time I wondered if such a band-aid should really have been necessary.

Re:Typical of medical and insurance businesses.

[NTT]

Hospitals generally don't keep old computer equipment because they are cheap bastards. Because of the patient safety issues, and related federal/state regulations, most medical related software have *extremely* narrow specs. They keep the old hardware around because the software provider will only certify the product's reliability on a specific class of computer. And this carries over to other responsibilities seemingly unrelated to direct patient care. I once had to maintain a quite old hospital food inventory system because it was the only thing that could talk to the billing system. The billing system was doing it's job, so there was no pressure to upgrade that.

Re:Typical of medical and insurance businesses.

[Anonymous Coward]

While HMOs have a bad reputation, if you're a basic user, you can generally find decent medical care.

If you aren't sick, an HMO is great. I was in my 20s in the 1980s when HMOs started. I developed a chronic cough. After dozens of visits I was never diagnosed with asthma. On my next insurance the first I saw doctor told me "You have asthma" and started to treat it. Treatment was expensive, but worked. Later I learned that untreated asthma causes lung damage. I hope some of those HMO doctors end up in Hell. I wasn't paying all the money to be lied to. BTW, I'm still on an "HMO", but there are many kinds. At the first one, the doctors were all working for the HMO and probably got rewarded for missing expensive diagnoses.

Re:Typical of medical and insurance businesses.

[gstoddart]

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever.

That's not being cheap. That's being a very conservative organization who can't really gamble on things suddenly not working.

I have seen numerous projects where a company with some older machines (say, mainframes) is being sold a piece of software to replace their 20 year old system -- and replace it with a Windows box or something.

The new system invariably does only about half of what the other system does. Invariably manages to make the process more cumbersome. And, in the end, generally pisses off the user community.

We all like to think in this industry that our new hotness is way better than the old and dusty. The actual case is, many of those old-skool apps which have been around for a while are less visually pleasing, but way more useful than the new hotness.

I'm not saying all modern software is crap, or that you don't eventually need to replace legacy systems. You just frequently end up with something which is not quite what the customer expected it to do -- sometimes, just because the old app has had exceptions built into it for all of the use cases that the analysts never really get to discover until you realize you can't support them.

It's tough to replace a highly specialized app which has been doing it's job for quite a while.

Cheers

Re:Typical of medical and insurance businesses.

[Penguinisto]

Nononono.... Auto dealerships.

I had personally, in 1999, had to go in and call Texas Instruments specifically to special-order a logic board for an ailing TI-931 terminal (early 80's model). I'm betting it's still there, right next to the old COMBAT 300-baud modem they used to call in their accounting sheets to Ford Motor Company every night.

(Then again, I was living in my hometown in Northwest Arkansas at the time, so draw your conclusions from that too...)

/P

Re:Typical of medical and insurance businesses.

[networkBoy]

I worked for a steam cleaning company on their billing computer. Small company, 10 employees tops.
Eventually the printer died. The app (dos, running on a 486) was hard coded for only three possible printers.
Guess how many I was able to find? none. not on ebay, not supported with emulation. none. I contacted the person who wrote the app and asked if I could have a copy of the source, to update the printer driver. I offered to sign an NDA and a non-redistribution agreement, as well as giving the source changes back. Her answer was: "That'd be an awesome deal if I still had the source".

Off to dell, bought a new system, this time with a backup to tarball (rar actually, being this is windows) feature to an external hard drive, new application, and a monster bitch of a perl script to read the (proprietary, undocumented, uncommented) database from the old app to a nice easy xls file, which the new app understood.

Moral of the story:
If it ain't broke, don't fix it, however, if it doesn't break soon enough you're in a world of hurt once it does.
I have a feeling we're going to be seeing more of this on a grander scale as *very* old business apps start failing for obscure reasons and can not be ported because of lost source.
-nB

Re:Typical of medical and insurance businesses.

[InsaneProcessor]

This was the point of my earlier post exactly! HMOs are of no value if you are really sick. The doctors are typically those that cannot sustain a real practice or are (as it was when I was briefly in the HMO) paid $10 per patient per month whether he sees you or not. They have no incentive to work for the patient and therefore do not. They are miscreants of the HMO and have no interest in the patient.

HMO was all my employer offered so I said "Hell NO" and bought my own coverage. You always have that option unless you are dirt poor.

Re:Typical of medical and insurance businesses.

[Cramer]

Lost source isn't the only problem... Source that no one (alive) can understand is also a growing problem. There are alot of old, obscure programming systems out there. Nobody teaches COBOL anymore :-)

Re:Typical of medical and insurance businesses.

[sphealey]

> If you want to see the oldest computer gear simply go to a
> hospital, insurance company or doctors offices. These places
> hate to spend money on IT and let old gear sit in place for
> almost ever.

Except that this article shows why that might be: true critical systems need to be reliable and understandable. Systems that have been in production for many many years often meet those requirements. That is why you often see 3270 green screen applications in large medical offices, and DOS or (old, pre-lawsuit) SCO Unix(tm) or MS-DOS applications in small offices: not because no one wants shiny new toys but because they have been beaten to death for 20 years (or more) and they either work or have known, predictable bugs.

Compare that to the referenced "upgrade" project. The same thing happened to a former employer of mine when they tried to "big bang" their mainframe apps to Windows client/server for Y2K: they couldn't invoice for 7 months. Good thing they had a _large_ line of credit...

The 3rd-to-last issue of the original print Byte had an excellent article about this. They talked to IBM's Director of the OS/nnnn operating systems; he explained how the code in the CPU scheduler had been refined from 1960 to 1980 - at which point it was locked, and will probably never be changed again in human history. Now, that code is written in S/360 machine language and should no doubt be considered "obsolete" - but they aren't going to "upgrade" it.

sPh

Re:Typical of medical and insurance businesses.

[LunaticTippy]

I saved a dozen Televideo terminals from the dumpster in the early 90s when my company upgraded. I used them to dial in to my shell account. When I moved into a smaller house in 99 or 00 and had to sell them I was shocked to find many eager buyers. I could get a hundred dollars easy.

Re:Typical of medical and insurance businesses.

[AgentPaper]

That's primarily for two reasons: 1) healthcare providers absolutely MUST retain backwards compatibility with whatever archaic, proprietary records systems they bought back in the 70's and 80's, and 2) they can't or won't afford anything better. As an example, the hospital my dad works at offers cutting-edge (pardon the pun) patient care across dozens of specialties, but they're still using a patient records system called HDS Ulticare that they purchased in... wait for it... 1983. This facility's IT complement is fairly modern by healthcare standards, too; I can think of a few places, respected teaching hospitals at that, that are still using the original 1960s-1970s POMR and Regenstreif punch-card systems.

Meanwhile, my dad's office ran all Windows 95 machines until about a month ago, and one machine in the set had to be run in straight MS-DOS 6.2 (?!?) in order to run the hospital's surgical scheduling software, which will not tolerate any form of emulation whatsoever. They're slightly better off now, with four XP machines and a Linux server, but they still have to keep that one DOS box until the hospital upgrades, which will probably be around the 37th of Janunever. To add insult to injury, the other surgeon in his practice pitched a fit about the expense of new computers, and he only agreed to upgrade under the condition that half the bill (it was about $7,000 total) come out of Dad's pocket. The two of them, and by extension nearly all the physicians and surgeons I know, will spare no expense for better patient care (learning new procedures, etc), but they see IT as a poor investment at best and a waste of money in most cases.

Re:Typical of medical and insurance businesses.

[Anonymous Coward]

A Wyse terminal is just a terminal; it's the host system that counts. You could be connecting to a host that's brand new or 20 years old. There really isn't any reason to replace the terminal unless it's broken.

That said...physicians are known for being cheap. However, larger health organizations are spending on IT like there's no tomorrow.

Epic Systems? No wonder...

[xxxJonBoyxxx]

In a blistering 2,000-word treatise, Mr. Deal wrote: "We're spending recklessly, to the tune of over $1.5 billion in waste every year, primarily on HealthConnect, but also on other inefficient and ineffective information technology projects."

C'mon, it really can't be that bad, can it?

Kaiser refutes Mr. Deal's assessment of its custom software system, developed by Epic Systems Corp.

Oh, Epic Systems? No wonder. Dude, you're f****ed.

Did they perhaps build it in MUMPS?

[Yoozer]

More horrors than you want to imagine.

Re:Did they perhaps build it in MUMPS?

[berbo]

from the link:

CASE SENSITIVITY: Commands and intrinsic functions are case-insensitive. Variable names and labels are case-sensitive.

OMFG. And it gets worse.

they didn't just use MUMPS, they used it cleverly. Partially as a result of MUMPS' limitation and partially because they had all gone insane.

Re:Did they perhaps build it in MUMPS?

[morgan_greywolf]

Yes. Epic's software is built using MUMPS.

Re:Did they perhaps build it in MUMPS?

[flink]

...developers needed to use the internally-created text editor. It actually made vi look user-friendly.

Hehe, that sounds suspiciously like my first job. I actually worked with someone who helped write that editor. I left M behind years ago, but I think it's made me a stronger person. Heres to anyone who still has nightmares about encountering
X @X
in the middle of a routine!

$4billion for a failing healthcare system?

[simong]

They're just not trying hard enough

Re:Epic Systems? No wonder...

[sammy baby]

Man. And you thought it was just DNF.

As someone going to a Kaiser appt. today...

[EccentricAnomaly]

The problems with the new Kaiser software are obvious to anyone who's been to Kaiser recently or spoken to a doctor or nurses who work there. Test results disappear, appointments disappear.... sometimes the people on the phone can't schedule appointments at all and tell people to call back later.

Re:As someone going to a Kaiser appt. today...

[lucabrasi999]

Test results disappear, appointments disappear.... sometimes the people on the phone can't schedule appointments at all and tell people to call back later.

I experience poor service whenever I deal with any organization in Health Care. Of course, where I live, Kaiser isn't a significant presence, so I can't blame them or their new software. I just blame it on the shitty system of Health Care we have here in the US.

Re:As someone going to a Kaiser appt. today...

[CmdrGravy]

You actually have healthcare in the US ? I didn't realise that, the impression I got is that when you get ill in the States you have to go off to the market rather than a clinic or hospital like you do elsewhere. I guess once you get to market you can swap your cow or goat, or whatever, for advice from the wise women who live there ?

Re:As someone going to a Kaiser appt. today...

[EvilTwinSkippy]

Puleeezzze.

Ever since the '40s we've been trading in zebra finches. Much easier to carry, and you can swap 2 finches for a goat and 10 finches for a Cow at any local farmer's market.

What's an HMO?

[ebcdic]

For the benefit of those of us who aren't Americans, why not say what an HMO is?

Re:What's an HMO?

[CheeseTroll]

Health Maintenance Organization

Re:What's an HMO?

[binaryspiral]

For the benefit of those of us who aren't Americans, why not say what an HMO is?

If only there was a way for someone to find information on the internet. Like a centralized tool or website that indexes other websites and allows people to search for it when they don't understand.

Or wait, even better - how about an encyclopedia like website that could contain vast amounts of knowledge.

Ah, that would be grand.

Re:What's an HMO?

[LordSnooty]

Journalism rule no.1: Know your audience. Ensure your work can be understood by your audience. Slashdot is read by people outside the USA.

Re:What's an HMO?

[Anonymous Coward]

Its a private version of a national health system (doctors on salaries, what doc you can see depends on where you live, etc), only we have multiple of them to choose from.

Re:What's an HMO?

[Anonymous Coward]

Most Americans are probably unaware or unsure what HMO stands for, but we all know it is: a type of health care service; restricts members to a list of participating doctors/medical practices; usually requires a co-pay; generally requires a referral from a "primary care provider" to see a specialist; and while being far from ideal it is the only affordable option for health care thanks to Hillary Clinton's closed door policy drafting sessions with the big health care providers during her husbands presidential administration.

Re:What's an HMO?

[nasch]

Slashdot is read by people outside the USA. Firstly, Slashdot is intentionally US-centric. If people from France or Japan or wherever don't like that, they should look for other news websites, because this one is apparently not for them. Secondly, the GP's point is that Google and Wikipedia are also read by people outside the USA, and in this case should have been. Here are the results from a Google search on HMO, the first of which is the Wikipedia page on it, with "Health Maintenance Organization" spelled out right on the search results page.

Re:What's an HMO?

[hereticia]

For the benefit of those of us who aren't Americans, why not say what an HMO is? Hibbert Moneymaking Organization, of course.

Re:What's an HMO?

[avdp]

Doctors that take HMO patients are not on salary. But they have to accept the fee structure the HMO setup (so $45/visit, instead of $150/visit kind of thing). That's why some physicians will take some insurance and not others.

Re:What's an HMO?

[Anonymous Coward]

What you desribe is what I know as a PPO; Blue Cross offers good ones. Where you have a netowrk of private doctors to choose from. But many HMOs, and definately Kaiser, employ docs directly and others pay a capped amount.

If you have Kaiser, you go to a Kaiser hospital or Kaiser health care center where all the staff including the docs are Kaiser employees, not a private practioner who happens to accept Kaiser. At least here in the west coast (Cal) thats how they work.

Re:What's an HMO?

[squiggleslash]

If you're British, you may be aware of an organization called the National Health Service, a group that provides (some) free health services via a compulsory overburdened insurance scheme although its competence and ability to deliver is often questioned.

Well, in America, such a concept is thought of as communist. It's considered important that things like incompetently provided healthcare are provided by free market means. Hence the HMO. HMOs are basically privatized, competing, versions of the National Health Service. Each insurance company competes to provide the worst service with the highest hidden (or not so hidden) charges.

I hope that helps.

Re:What's an HMO?

[Anonymous Coward]

HMO = Have Money Or [Die]

Re:What's an HMO?

[avdp]

Oh interesting, I've been on 3 different HMOs so far (Aetna, HealthAmerica and UPMC - the last two are for the Western Pennsylvania region) and all three have worked the way I described. And all 3 worked with my current doctor so I've never had to change doctors (or my kids pediatricians) throughout these HMO changes.

So what?

[0123456]

From what I understand the British government have blown about $25,000,000,000 on a computer system for the National Health Service which doesn't work worth a damn and which the doctors didn't want in the first place.

Private incompetence pales in comparison to government incompetence.

Re:So what?

[CmdrGravy]

I think there's a whole load of incredibly expensive and functionally useless computer systems in the NHS, this latest one is in good company.

Re:So what?

[Anonymous Coward]

Private incompetence pales in comparison to government incompetence.

Actually, they're about on a par.

Kaiser has 8.5 million patients, so $4 billion = $470 per patient.

NHS has 60 million patients (population of UK), so $25 billion = $420 per patient.

Re:So what?

[Mumpsman]

What are you talking about? http://www.metro.co.uk/news/article.html?in_articl e_id=46710&in_page_id=34MTAS is an excellent web server.

Re:So what?

[sarble]

Private incompetence pales in comparison to government incompetence. The NHS contracted several private IT companies to do the work for them for that very reason... erm...

Re:So what?

[orielbean]

Remember that the doctors don't ever want any sort of centralization. They don't want any regulation either. Both of those things increase cost of doing business. Always.

The govt needed some sort of controls to be had in their system, at the expense of the doctors and patients. Remember, healthcare is a triangle, so any change to one side's cost will have an effect on the other two sides. Doctor/Hospital - Patient - Payer (insurance company along with the employer and/or government).

Of course, being a big bureaucracy they found a way to cock it up bigtime.

Re:So what?

[sfjoe]

Private incompetence pales in comparison to government incompetence.

Actually, the oposite is true. For example, the US Medicare system spends about 98% of its funding on patient care. Even the largest and most efficient private insurers only can manage about 90%. The radical right-wing wants to preserve the profits of private insurers and so they spend a great deal of their propaganda efforts highlighting mistakes in the public sector so it appears that government is less efficient than it actually is.

Re:So what?

[swillden]

For example, the US Medicare system spends about 98% of its funding on patient care.

Administrative overhead is just about the worst possible measure of efficiency. What you want to measure is amount of needed services delivered for the cost. Medicare's low administrative overhead means that it doesn't have the administrative capacity to avoid paying for unnecessary procedures or to catch providers upcoding visits or procedures or outright inventing them. Even ignoring the question of how much of the services Medicare pays for are actually necessary, Medicare has a huge rate of outright fraud -- around 10% by most estimates, though some industry analysts think it's much higher.

Medicare needs to spend more of its budget on administration, so that it can cut down on the fraud and reduce the amount of unnecessary care that it's paying for. It's entirely likely that if Medicare were to spend 8% of its budget on administration, rather than 2%, its budget could be cut by 15-20% without affecting the quality of care delivered.

Re:So what?

[sfjoe]

Administrative overhead is just about the worst possible measure of efficiency.

Are you sure that bloated management salaries aren't a worse measure?

Long-term Kaiser patient disputes wild claims

[msblack]

This 45-year old lifetime Kaiser patient believes the media frenzy has blown this out of proportion. Kaiser Permanente undertook the monumental task of converting its patient records system into an electronic information system. Converting the hundreds of paper records for each of millions of patients is truly a monumental task and some problems will occur.

Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.

I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

As a KP-IT employee it's good to know your visits were made better - not worse - by IT. But....not a single outpatient facility in California has switched to e-prescribing yet....So are you outside California or just our PR department?

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

I agree with this. I've been with Kaiser for the past couple of years, after having suffered through some truly bizarre and terrible PPO service (three different health plans). Another poster said that only idiots would use an HMO. Well, in my experience, Kaiser has been great so far. Of course, I haven't stressed the system yet, and I hopefully will stay healthy enough never to have to stress the system, but... so far, so good!

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

Your situation may be more unique than you think. I used to do telephone surveys for Kaiser. Most of those surveyed were highly dissatisfied with Kaiser with the chief complaint being a massive bureaucratic system where a patient could easily get lost in the system. One guy I spoke to had an ear infection that was so bad he was about to go deaf. He kept getting referred from one doctor to another and no one would help him because he wasn't given a referral, or he was referred from the wrong dept., and so forth. This wasn't all that unusual from those I spoke to.

Re:Long-term Kaiser patient disputes wild claims

[Mumpsman]

As someone who was up until 1:30 this morning frantically loading RAs into Chronicles in preparation for my Sunday go-live, I really needed to read your comment today. Dealing with Epics goat cluster of poor design decisions can easily lead an IS shop into a downward spiral of negativity and depression, loosing sight of the real purpose - patient care.

Does the "pain" I feel really matter when there are actual sick people up in that building relying on providers who need up-to-date and accurate information to deliver care? Perspective is important. Either that or I am about to lose my mind.

Re:Long-term Kaiser patient disputes wild claims

[msblack]

You're plain wrong. They have switched to "e-prescribing" as I did not take a traditional 4x6 prescription not to the pharmacy. It was all by computer!

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

I'll take that as you being in California. Which means no e-prescribing. Which means if you didn't carry the slip it was faxed. Sorry to burst your bubble.

Re:Long-term Kaiser patient disputes wild claims

[juuri]

As a 45 year Kaiser member you should. What they have done and are still tuning is nothing special. Other HMO/MMOs have already done such and many did this years ago for a far cheaper price. Yes, what you saw was convenience yesterday and that is an awesome selling point. What you didn't see what the horrendous roll-out time for this project, the multiple attempts at re-inventing the wheel and the wasted excess in monies spent which is passed on.

It really pains me every-time I see one of these large organizations who outsources these huge projects onto contractors like this. In many cases it would be *just* as fast and more importantly *cheaper* to create a new internal division or ramp up their own IT staff to complete these projects. Outsourcing projects of epic scale is lazy, wasteful and usually involves something shady. I've seen it time and time again in the IT portion of multiple industries and you would think by now, more people would have caught on to what a sham it is.

Re:Long-term Kaiser patient disputes wild claims

[sboutwell]

hahaha... Umm.. You must be someone that leads a VERY BLESSED life. I've been waiting now for a Surgical appointment for nearly 3 weeks. We can NEVER get through to anyone that can ACTUALLY schedule an appointment. And they have been scheduling them on a PAPER NOTE BOOK at our local Kaiser office. I also turned in over 90 pages of Medical records that I personally have collected from my former doctors before transfering to Kaiser. How many of them made it into my ELECTRONIC Record. 15 Pages, the REST POOF disappeared in to THIN air. The included relevant LABS run in the last 1 year regarding my condition and were needed by my Specialist for my surgery. I was told I needed to round them up again and give them to them. I was also told by the folks that Run the records department that a VIRUS wiped out all of their Medical Records.

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

dunno, don't care how it got there, they just enter it into the computer and I don't do anything... fax or whatever.

Re:Long-term Kaiser patient disputes wild claims

[ruben.gutierrez]

I'm curious to know what you mean by outsourcing. I'm not very familiar with Kaiser's project, but I doubt that their entire implementation staff was outsourced.

Re:Long-term Kaiser patient disputes wild claims

[stefanlasiewski]

In many cases it would be *just* as fast and more importantly *cheaper* to create a new internal division or ramp up their own IT staff to complete these projects.

I seem to remember that Kaiser had an internal division which was working on a electronic-medical records project back in 2000/2002. They shut down the project, laid off a bunch of staff, and started over with a whole new system.

I know that not everyone has had a good experience. I have used Kaiser in the past, and it was horrible. For us, this new Kaiser system has been like a breath of fresh air.

My family has only had 1-2 scheduling problems with Kaiser since we joined a year ago. It's been very easy to make appointments in person, via the web, etc. I've never had a very long wait, and the medical staff are always able to get our medical records. We can send messages to our doctors via kp.org, and the doctors usually within the same business day.

Before Kaiser we went with a well known PPO-- the experience was horrendous, the costs were ridiculous and I spent too much time on the phone dealing with their disorganization.

Re:Long-term Kaiser patient disputes wild claims

[AaronW]

It's interesting then that I went to a doctor's appointment on Monday and the doctor quickly typed in the medication and clicked on the pharmacy I wanted to go to and everything was done with no paper. The facility is out patient only in my city in northern California. It's been like this for the last two years. Everything is electronic, with a computer in every examination room. It also ties in to their web site so I can access the information and get test results through the web or schedule appointments through the web. And this is not Kaiser *shudder*.

My family had Kaiser for a while and all I can say is NEVER AGAIN.

Re:Long-term Kaiser patient disputes wild claims

[Anonymous Coward]

I work in KP-IT too. He did not say that everyone was using healthconnect, just that there was no paperwork. I had the same thing at my totally non-HC site when I went to the doctor last.

Re:Long-term Kaiser patient disputes wild claims

[sgt_doom]

Amen, brother! (Or sister!) Multiply your experience by 1,000 times worse - and you've just described my past experience with those clowns......

Re:Epic Systems? No wonder...

[Mumpsman]

Oh, Epic Systems? No wonder. Dude, you're f****ed.

*sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.

Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.

A previous article...

[Mad_Rain]

There is a Computerworld article from the previous slashdot story that seems pretty helpful in understanding the meltdown of their electronic medical records systems. They say that they are running the world's largest Citrix server system, and it does not scale well for their purposes.

As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).

Re:A previous article...

[gladish]

Wait, you mean Vista has medical billing software built-in now? Is that home or pro only?

Re:A previous article...

[NeutronCowboy]

Citrix? Hah! No wonder they have issues. Citrix servers are an absolut pain to properly maintain.

Re:A previous article...

[bockelboy]

How is the VA's system to work with? As far as I know, it's a huge success story compared to the British and KP failures.

Re:A previous article...

[fuego451]

I must say that my "favorite" has been VistA [vistasoftware.org] (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista).

As a recipient of VA health care, I have been very impressed with the efficiency and reliability of the VA records system. I can go in for x-rays, lab tests or whatever and twenty or so minutes later be in consultation with my doctor who has the results on his computer. Prescriptions and referrals are also handled by the system with no paper work needed to be carried by the patient. It is nice too that any VA office I visit will have access to my medical information. I even receive computer generated reminders of appointment dates.

I had Kaiser health care for 24 years and rarely had any records related problems but I always felt like an errand-boy carrying around paperwork. Their biggest problem then was patient care. I got smacked in the face by a surfboard one day and I knew that, at the very least, my nose was broken. The doctor said there was nothing he could do; my nose was straight (for the first time in many years from previous fractures) and I had controlled the bleeding. Four days later I get a call telling me to come to the emergency room immediately and that this was a medical emergency. Seems someone got around to looking at the x-ray of my face and I had a fracture completely around the base of my nose with a chance of brain injury and cerebral/spinal fluid leakage.

Re:A previous article...

[Dark Paladin]

The VistA system uses a very interesting database/programming language called MUMPS - long story short, it was doing 20 years ago what Oracle and EJB 3.0 are doing today. Variables declared in the correct way in MUMPS are automatically created in the database - the programmer doesn't have to worry about how that's done, or how to connect to the database. They just make up their variables, define their methods, and the system does the rest. There are newer implementations such as Cache that use the MUMPS language and expand on it (unless I'm mistaken, I believe there are some Object Oriented tools to go with it).

It's one of those things that never got big attention, though the technology was pretty advance and simple when it was created.

Re:A previous article...

[willutah]

But from what I understand, MUMPS has fallen behind the times and programmers are hard to come by, at least according to this programmer:

http://www.emrupdate.com/forums/post/13072.aspx

He starts out with "Not only is it written in mumps, on top of that, it uses the "VA Fileman", which is a DBMS written in mumps... it's hard enough to find a Mumps programmer, pile on top of that a mumps programmer that knows fileman, talk to your bank for a loan, believe it or not - a VA Fileman / Mumps programmer will cost more than a tank of gas in a Hummer."

The whole thread on VistA is here: http://www.emrupdate.com/forums/post/13072.aspx

This all could have been avoided...

[eln]

If only these guys had used Hyperion to manage their data, they wouldn't be having this issue. Hyperion can effortlessly manage up to 12 GIGABYTES of data, and all you have to do is partition it into 3 different pieces.

Hyperion: If it's good enough for Google, it's good enough for you.

Re:This all could have been avoided...

[Anonymous Coward]

If only these guys had used Hyperion to manage their data, they wouldn't be having this issue.

Yeah they wouldn't have had to deal with this issue as they would of been fired a long time ago for buying a system that can't stay up for a week without reboots.

Re:This all could have been avoided...

[Anonymous Coward]

12 GB is a drop in the bucket. Try terabyte.

Young people are cheap to insure

[msblack]

Granted, I wish I could opt out of paying for any medical and get the money in hand and pay my own, rare, expenses but I don't have that option. It would be cheaper for me to get the money and pay when I go to a visit than it is to piss away money every paycheck and not use it. Your message sounds to me as though you're fairly young. Have you ever checked insurance premium charts? The cost to cover a 20-something is about half the cost of someone in their late 40s and above. The whole idea of HMOs and group insurance was to spread the risk among huge patient pools so no one patient would ever face large medical bills for the unexpected. So while 20-somethings fuss over $400 monthly premiums deducted from their paycheck, keep in mind that the level amount deducted from all employees keeps costs the same for older employees. Or would you like to argue against level-premium risk pools?

Re:Young people are cheap to insure

[smooth wombat]

So what you're saying is, I should happy that my money is being used to keep down the medical expenses for my 250 pound, pack-a-day smoker who sucks down a two-liter bottle of Pepsi every day co-worker, right? Why should I, a healthy, never-had-anything-wrong-with-them, including never breaking a bone despite participating in sports for over a decade, have to finance someone elses bad habits?

While the concept of pooling resources sounds good, in the end it only ends up hurting more than it helps. You can look at any situation where pooling exists, except lowering drug prices, and the same applies.

Here's a great example. Both my parents are retired and in very good health. Each has one prescription drug they take yet pay over $6,000 a year for medical coverage ($3,000 each). $6,000! This was the result of the Bush administrations push to have everyone have some kind of medical coverage by forcing people into various programs but conveniently having a 'doughnout hole' gap in coverage. A gap where, in addition to paying for outlandish coverage premiums, people have to shell out tons of money out of their own pockets because these mandatory programs won't cover the expenses until you reach certain levels. See this breakdown of how much people will have to pay in addition to their yearly premiums to cover this gap.

And for the record, I'm 40 and in perfect health. Since it's my money and body, I should be the one to decide if I want health insurance, and the high premiums that go along with them, or if I want to pay out of my own pocket.

Re:Young people are cheap to insure

[LurkerXXX]

Then why don't you do just that? Every employer I've ever worked for has had the option of not taking/paying for the companies health insurance, in case you get a better deal through your spouses employer, etc. Why not just opt out and quite bitching then? Of course, if you do come down with something nasty, you are going to be well and truly screwed, but that's your choice.

Re:Young people are cheap to insure

[drgonzo59]

Of course, if you do come down with something nasty, you are going to be well and truly screwed

That is a problem with healthcare in this country. If you let companies make a profit off of anyone's illness, everyone will be screwed (except the CEO's of those healthcare companies). I lived in the Soviet Union and I can confirm that as bad as the Soviets were they had very good basic healthcare. It didn't matter what you had who you were, you could get treatment. You just had to walk into a hospital tell them what 'hurts', show them your medical history record (a paper notebook ) and you would get treatment. You didn't have to sell your house and get in dept up to your ears just because you had to have surgery.

Here you will die if you cannot pay. This is in a country were supposedly human life is more valued that in those other "barbaric" countries. I think I would trust Uncle Sam with my healhcare and pay higher taxes than pay the same taxes and more to get the CEO of healthcare conglomerates rich. At least with the government there could be some transparency and more accountablity.

Besides, why are the doctors so freaking rich in this country and have this status of "gods". It seems like most of them only go into the profession to make money rather than care for the patients. In other countries doctors are just like everyone, just like the teachers, programmers, engineers and accountants. It only takes memorization to become a doctor, they don't have to develop or invent a new process or theory like the PhD's. Why the fuck are they treated above everyone else?

Re:Young people are cheap to insure

[LunaticTippy]

The problem with this is that we have universal coverage already. Anyone can walk into the ER and get their problem dealt with. Most of them can't afford to pay, so those with coverage have to pay a lot more. The higher premiums get, the more people opt out, and this drives premiums up yet further.

In the end, we'll have only sick/old people with insurance, costing $10k/month, and everyone else with no coverage and an unpayable $1m hospital bill if they fall down the stairs.

That seems pretty dysfunctional to me. I don't think our healthcare system should be based on bankruptcy and weird cost-shifting.

It's sort of like auto insurance. It works a lot better if everyone has it. If you let people who are safe drivers opt out it drives up premiums, forcing others to opt out, driving up premiums, etc. You'll end up with insurance costing thousands per month and most accidents between 2 uninsured people.

Incidentally, you sound like you'd like a high-deductible HSA insurance plan. I'm on one, and it is perfect for those of us who don't go to the doctor all the time. I sock away money tax free in investment accounts and it grows year after year, there for me for any medical expense. I like paying for things like glasses and dentist visits with tax-free money. Hopefully I'll have tens of thousands saved up in the account by the time I need chronic care, and I've got 100% coverage over my deductible in case I get cancer.

Re:Young people are cheap to insure

[wazza]

> It only takes memorization to become a doctor, they don't have to develop or invent a new process
> or theory like the PhD's. Why the fuck are they treated above everyone else?

Ouch... fair enough, you might be just airing some sour grapes here, but I have to disagree.

I work with doctors (my job is ultrasounds in the neurology dept. of a large public hospital in Australia), and I can tell you this: it might only take memorization to pass your exams (although I personally have my doubts about even this assertion), but there is an _enormous_ amount of other skill required to be a good doctor.

Just to name one, a doctor worth their salt has to have excellent social & communication skills. It's often very hard to get accurate information from patients regarding their symptoms & signs. This is fair enough - they're sick, and most likely worried, and some don't want to tell you how bad things really are. But decent doctoring is so much more than just "It only takes memorization" that I think I'll just give you the benefit of the doubt, and stick that part of your post into the "whinge box".

(As for your final point, doctors are treated above everyone else primarily because people respect them for having the sheer balls to make decisions that *will* drastically affect someone's life, and be accountable for those decisions afterwards.)

Re:Young people are cheap to insure

[drgonzo59]

doctor worth their salt has to have excellent social & communication skills

Yeah, so do used car salesmen.

doctors are treated above everyone else primarily because people respect them for having the sheer balls to make decisions that *will* drastically affect someone's life, and be accountable for those decisions afterwards.

Oh, that's why? Well how about the civil engineers that build bridges. A doctor screws up once, one patient dies, an engineer screws up and hundreds could die. Obvously that is not the reason. It is not as if in other countries doctors are stupider or kill people every day on average they are just as good as American once, except that they don't behave like Gods. Whenever you put a pricetag on a curing a human illness you will get a situation where you end up with most doctors just wanting to make money. I have heard countless times of parents wanting their offspring to be either doctors or lawyers, not because the parents have a strong sense of justice and want to "root the evil from this world" or want their child to dig through people's guts because somehow they are very altruistic, but because they want them to make more $$$$$. So I think more than half of the people in medical school are there because of the money. If it would be a profession where only people who really want to help would go into, everyone would be better off. As I said, I lived in the ex-Soviet Union and went to the doctors there, had surgery, and other minor procedures. I grew up healthier and better taken care of and I didn't have to get into debt up to my ears because of it.

They aint got nothin' on Portland....

[Anonymous Coward]

Although they spent much more, the city of Portland Oregon spent $20 million on a new system to bill water. Yes, city water.
Didn't work and they scrapped everything.
How freakin' dumb do you have to be to do that????

Percentages and Marketing speak

[WarlockD]

Dell offers contracts for 99.9% system availability. This means that if you pay the service contract, your within 30 miles of a depot, they guarantee the system won't go down because of hardware for 8.76 hours out of the year. In training we were calculating all that. Technically, it calculates to two "no post" service calls out of the year. So it looks really good when you market 99.9% reliability.

It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.

Hours in a year 8760
99% = 87.6 hours
99.2% = 70.08 hours
99.7% = 26.28 hours

If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.

Re:Percentages and Marketing speak

[NeutronCowboy]

You know what's sad? I started my IT career with Compaq at it's (nee DEC) non-stop computing department. 99.99999% uptime for these suckers. Redundant everything, hotswapping of HDs, all in 1999. Can't believe people are touting 99.9% hardware availability as a success. Not when 99.8% availability for critical apps is becoming a de-facto minimum....

Re:Percentages and Marketing speak

[aaarrrgggh]

In fairness, they had some huge growing pains with the project that simply were not properly planned out. They outgrew all of their data centers at once, bought a new one that they thought they could just move right into... and had to spend $100M to upgrade it. That forced them to stay in the facilities that were bursting at the seams longer, and work through some extremely high risk activities.

In theory, 99.7% uptime is a system outage of 6 hours once a quarter for maintenance. It might take them a couple years to get there, but they have some great people on board now to at least reduce the facility contribution significantly. Unfortunately, it isn't likely that they can keep 99.9% availability for input power to the IT systems.

Re:Percentages and Marketing speak

[fishbowl]

>In theory, 99.7% uptime is a system outage of 6 hours once a quarter for maintenance.

No! That outage is not considered in the uptime requirement, since it's *scheduled*.

Re:Percentages and Marketing speak

[jhfry]

Scheduled maintenance windows are usually not calculated in uptime measurements. Nor are times the system is unavailable during batch processing, or other predictable downtimes.

I cannot say that is a High Availability system, such as this, that they use the same standards.

And to say that they can't achieve 99.9% availability for power is entirely false. Rackspace, who hosts my companies website, has far better than 99.9% uptime on their infrastructure. That's what redundancy and backup generators are for.

Sure, servers go down, but in a High Availability environment, those are redundant too. I would suspect that most of their downtime is related to software issues... and most of it is the result of poor purchasing, planning, and implementation choices... entirely preventable.

Re:Percentages and Marketing speak

[renoX]

I think you're wrong..
Sure there are computer with 5 9 availability, my first job was programming on a Stratus, an expensive but very reliable computer, but
1) even with a Stratus those 5 9 are mostly theoretical, in one year, I've seen:
- power outage: the maintenance guy cut the wrong cable.
- several software failures.

2) a complex system is usually less reliable than the parts used to build it.

Re:Percentages and Marketing speak

[NeutronCowboy]

True - the seven nines were theoretical, and I hadn't seen any data on one of those machines running for a year. However, I don't count application failure as part of the server going down. Those 7 nines were strictly for the server itself being up... which definitely cuts down on the complexity. I agree that if you start adding actual applications into the mix, you'll be very lucky to even get to three nines.

I think though that if you start with hardware that is only 99.9% reliable, it will be impossible to get to 99.9% app availability, and even anything approaching 99.8% is going to require a lot of creative accounting of what constitutes an outage.

Re:Percentages and Marketing speak

[aaarrrgggh]

And to say that they can't achieve 99.9% availability for power is entirely false. Rackspace, who hosts my companies website, has far better than 99.9% uptime on their infrastructure. That's what redundancy and backup generators are for.

Kaiser isn't Rackspace. Kaiser isn't a bank, either. Kaiser's primary data center related to Health Connect is/was in a sad state-- systems loaded to 90% or more of capacity, eliminating whatever redundancy once existed. Major problems with utility electrical supply, generator systems, and UPS alike. They decided to expand their ~20-year old UPS system rather than replace it! This alone caused major issues to their reliability.

Redundant servers are great, but when they are in the same building, supplied from the same PDU, and often in the same rack, there isn't a whole lot of real redundancy provided. That is when the core infrastructure really determines the net system reliability.

As for excluding scheduled maintenance from availability calculations, that is hogwash. If a business unit does not have the services required, the system is down and that counts against availability. It might be a low severity incident, but it is an incident none the less.

The worst?

[Wiseazz]

'the worst [technology] project I have seen in my 25 years in the business.'

Yikes. If his 25 years of experience are anything like my 10 years, then that's saying a LOT.

My dentist...

[antdude]

My dentist office still uses DOS/text mode on a PC (no idea what's the setup -- maybe a 386?).

The State of Maine fails:

[autocracy]

Maine does a system cutover over 2 years ago, and its still broken. Doctors go out of business because they can't get paid, and everything's a mess. I think this one wins :-/

http://www.cio.com/article/20133/Maine_s_Medicaid_ Mistakes

Re:The State of Maine fails:

[rickb928]

Maine was having trouble paying claims before the new system was running. One of my clients in 2004 was struggling to get paid when they implemented a 'new' claims system, which the CIO article either ignored or wasn't part of the reporter's viewpoint. It was supposed to be a portal to the old Honeywell system, and it flamed out.

But even more interesting, I see nothing lately about the case management system in Maine for DHS now HHS. I've moved to Phoenix. Did they every actually implement that? Or was that just another flameout? Last I heard, they spent >$10M and basically gave up and started over. While the old system was failing outright.

Kaiser sure isn't the only medical outfit that has blown a system. My hospital client in about 1998 implemented an ER case management/records system, just as HIPAA was getting going. The goal was to eliminate handwritten and illegible notes. The result? Well, the system gave the ER docs a system where they could t-y-p-e the notes in. Sweet. Best part? the original goal wasn't actually to eliminate illegible notes, but to eliminate unreadable and nonsensical notes, making the records standardized by using standard, boilerplate notes to describe patients in standard, agreed-upon language. Woops, the system let docs input free-form text as an enhancement to the standard text. Yup, docs just blew through the boilerplate and typed in whatever they used to scribble. Same drivel. Oh, and the vendor underestimated the storage space by 80%. We kept expanding the array until we ran out of slots. I left the project, and the incoming crew inherited a mess. No word, but the rumor was they threw it all out. Of course.

Where I'm at now, we're enjoying a classic J2EE development nightmare. No good requirements, bad code, long development cycle, fundamental components missing/broken, desperate push by management to get something 'out the door', no matter how bad it is. As a breath of fresh air, the 'legacy' team has an opportunity to update the old PC-based app at breakneck speed to sort of compete with this new and dysfunctional system. And my money is on the old guys.
How fitting that my secret word to enter this post is 'disaster'....harrr...

-rick

Windows system doesn't scale. What a shock.

[ericferris]

From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.

"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."

So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.

The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.

The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.

Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.

Re:Windows system doesn't scale. What a shock.

[Mumpsman]

This is probably my biggest beef with Epic and other Mumps-based apps that use a bloated client. There's just no need for it. IDX http://www.gehealthcare.com/usen/hit/products/cent ricity_business/index.html has a very nice Web front-end that does everything in a browser (albeit with unnecessary IE dependencies). Epic has a thick VB6.0 client and CPRS (The VAs VistA GUI http://www.hardhats.org/) is massive.

I know it's a lot of code to re-write, but it's not like vendors need to throw out the database code. Just fix a mistake made 10 years ago, not the one from 1964. Cache allows for some nice things in a browser based app (XEN? SVG?). Having that kind of UI really helps to manage desktops in a distributed environment like a hospital.

Re:Windows system doesn't scale. What a shock.

[kahei]

"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it.

Considering the performance and flexibility problems observed even on small, normal Citrix installations, I'd have to say the above sounds like a bad move -- probably one among many in this project.

Heck, what could be sillier than basing a critical system on a Citrix deployment of untested scale and unconventional topology?

The way normal people would do it is use an X11 graphic application

Ah, it appears that a Slashdot-crazed unix fanatic could be sillier!

Re:Windows system doesn't scale. What a shock.

[NeutronCowboy]

Citrix? *snicker* Yeah, I'm not surprised that they have issues. I monitor those babies for a living, and they are the most temperamental systems I've ever seen. Not to mention incapable of properly scaling or working under load. You want scale? Get a new citrix server, and hope to god the load-balancing doesn't barf. I'm not sure Win32 is their culprit, but Citrix is definitely one of their problems.

Re:Windows system doesn't scale. What a shock.

[ericferris]

I said: The way normal people would do it is use an X11 graphic application

You replied: Ah, it appears that a Slashdot-crazed unix fanatic could be sillier!

Silly? Why? Please explain.

I didn't say X11 was THE way to do it. But I've seen large deployment of X11-based graphical applications running on Unix servers (AIX to be precise) that were accessed first from a variety of thin clients (X terminals among others) and later from Windows machines running X11 display servers. The whole thing was reasonably fast. They worked well enough that management was in no hurry to recode the apps to make them browser-based.

These days, when I am spec'ing new distributed GUI-based applications, I don't recommend writing against the X11 stack, but your mileage may vary.

Re:Windows system doesn't scale. What a shock.

[kahei]

Sure, I agree that there's a body of legacy x-windows apps (usually with windows clients) that's chugging away pretty effectively. In fact, many of them have reached a considerable level of stability, and they're more responsive than web apps while still being quite scalable. I don't think I'd go from that to calling X11 'the way normal people would do it', though, at least not these days.

I think the path of least controversy these days is a web app. Of course that just raises the question of what you'll have on the server side -- .NET, 'simple' Java, custom code, or J2EE...

Re:Windows system doesn't scale. What a shock.

[The Bungi]

Yeah, I'm sure this is Windows' fault. Because I've never seen a successfull application like this one (sans the Citrix shit) running on Windows. And I've certainly never seen an application being fucked up on UNIX. So therefore the fact that this thing is broken cannot possibly be the result of bad decisions, bad design, bad management, etc. Let's blame "Uncle Bill's broken OS", because it's fashionable and gets us lots of karma on Slashdork.

Re:Windows system doesn't scale. What a shock.

[ruben.gutierrez]

You definitely cannot compare IDX to Epic. I've worked with both environments, and IDX is several magnitudes smaller in terms of functionality and data management. Not to mention IDX is not very easily customized. And, the support of IDX hasn't been comparable to Epic's support, at all. I'm not saying which system/vendor is better or worse for their particular purposes. I'm just saying their apples and oranges.

Kaiser Permanente has significant problems

[Pedrito]

Having been, unfortunately, a former user of Kaiser-Permanente's services, I can say with confidence that their problems span way beyond software problems. 7 years ago, I was misdiagnosed with asthma by 3 Kaiser doctors. A 4th finally agreed that I didn't have asthma (and I don't). My actual problem was panic attacks caused by excessive adrenaline production, which no Kaiser doctor was able to determine. I had to go outside of Kaiser to find a doctor competent enough to determine the actual problem and treat me appropriately (with beta-blockers for the adrenaline instead of mind-numbing drugs like Paxil and Valium).

Add to this the reports of KP dumping homeless patients on the streets. There was the kidney transplant scandal. The patient information becoming available online scandal. And all of this in just the past 2 years. Kaiser is clearly flawed from top to bottom. They are the poster-child for real health care reform in this country.

Re:Kaiser Permanente has significant problems

[drinkypoo]

Those who can, do. Those who can't, teach. Those who can't teach work for Kaiser. They're the lowest-rent motherfuckers in the US health industry and everyone knows this.

Epic Systems

[DrDitto]

Epic Systems-- selling $4b systems based on clunky, non-relational databases you've never heard of.

Re:Epic Systems

[Anonymous Coward]

So... Is everything you've never heard of not worthy of use? Cache and various other MUMPS implementations are widely used in the health services IT and have been around for decades. (Epic, IDX, Misys, ACS, Siemens, and Meditech all sell systems based on some form of MUMPS.)

Oh puleeezzze

[Anonymous Coward]

You honestly think that this is a technology issue? Your just as dumb as the Kaiser/Epic guys if you do.

Epic dude: "uh, yah, it's a technology issue. It's Windows fault. Give us another $20bil and we'll port it to Linux and it will be soooooo much better"

Absolute BS. Bad architecture and implementation is bad architecture and implementation, I don't care what OS or environment you build the thing on. X11 running on Win32, yah, that's a wonderful idea, I can't wait to see that deployed for 100000 users.

Interesting website ..

[rs232]

Interesting website ..

They didn't get huge and rich by...

[Anonymous Coward]

she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.
 
...squandering all their money on the "latest, greatest" bleeding edge, blaze-the-pioneer-trail technology that keeps them on a perpetual upgrade and replace treadmill that only does good to keep the corporate appetite of their IT vendors satisfied.

An enterprise-level, mission-critical system for a huge company should only be implemented if it has an expected useful service life of no less than 15 years from go-live until replacement. And at 5 years prior to that replacement date, the wheels should be set into motion for the next new replacement system becuase it really does take that long to make a full transition.

"Big Business" is sick and tired of the IT industry forcing premature obsolescence down their throats. Expect things to come back round full-circle to the way things used to be when IBM was the king, systems were expensive as hell, tech staff was expensive as hell, but the systems and the people who ran them would stick around for longer than a flash-in-the-pan moment.

Not surprised here.

[Anonymous Coward]

After a year and a half in the medical billing industry supporting software that does just what KP HealthConnect does, I can't say I'm surprised. Kaiser Permanente is one of the worst providers of medical healthcare in the industry. You're far better off picking out a good Blue Cross/Blue Shield plan and seeing another doctor.

Kaiser sub-standard care.

[TheGeneration]

My father had a heart attack about a month ago. He is a member of Kaiser. Kaiser has it's own hospitals, doctors, and entire network that you must use. His heart attack resulted in him needing to undergo heart surgery. (The advantage to this sort of system is that my father's heart surgeon performed over 1000 bypasses every year and as a result is in the top tier of experts.)

For me the biggest issue came when it was time for discharge. The handed me a hand written sheet with about 12 different medications scribbled in "Doctorese" cursive writing. I couldn't read half of the sheet, and only the briefest of notes for each medication was listed. I was shocked that I didn't receive a computer printout with the medication instructions, especially for medications needed to be taken in the case of quadruple bypass heart surgery.

After many hours of sorting through the medications, trying to figure out if he had already been given a dose at the hospital or not, etc... I realized that when I had visited a non-Kaiser emergency room for a sprained ankle I got a multi-page printout with very thorough instructions for the medication and aftercare. On another visit when I had 107 temperature it was the same thing. I was very certain of the doctors orders for post-visit care.

With Kaiser though I was left confused with short hand written notes about a dozen medications. Confusion is not a state I wanted to be in when I'm charged with the care of one of the most important people in the world (to me) who has a life threatening condition!

Re:Kaiser sub-standard care.

[Anonymous Coward]

"Kaiser has it's own hospitals, doctors, and entire network that you must use."

That is not true. You can absolutely seek health care from people outside their network; it simply requires a referral (easy to get) and some extra paperwork. You won't get as low a copay, but what do you really expect? Your father should look into his policy.

Re:Kaiser sub-standard care.

[TheGeneration]

Considering he was having a heart attack "shopping around" wasn't exactly our priority at the moment.

Re:Kaiser sub-standard care.

[barzok]

Did you ask for the instructions to be rewritten so you could read them? Ask for more detailed notes? Ask which meds had been administered already, and when?

If so, did you get satisfactory results? If not, why?

I really hope you didn't walk away saying "ok, so this is what they give me, I guess I have to figure it out on my own." As a patient or care-giver, IMHO if you don't understand or can't read what you've been given, you need to get it straightened out before you leave the hospital/doctor. And the hospital/doctor shouldn't be saying "no" to your requests for more/better information.

Re:Kaiser sub-standard care.

[TheGeneration]

Yes I did ask for more details, and never recieved them. It took them 3 hours to get a wheelchair to his room so they could discharge him. Apparently there is one wheel chair for the entire cardiac unit (there were about 40 patients.) At that point I was worried that if I started insisting they do anything else that we would miss our "wheelchair" and it'd be reassigned to another room and we'd spend another 3 hours waiting for our turn in the round robin scheme for discharges they had going on. I also figured I could just call the pharmacy or the cardiologist once I got home and get more detail (which I did, but because their computer system was fugged up they weren't sure if he had gotten his medication or not.)

Re:Kaiser sub-standard care.

[TheGeneration]

I forget to mention that the nurses were all Filipino spoke broken English and were nearly impossible to convey ideas to/from. While they were excellent nurses, the language barrier was significant and complicated the post-care instructions considerably.

Re:Kaiser sub-standard care.

[rahvin112]

As someone who not only has been discharged from fairly major surgery, including an angiogram of the brain and eye surgery that involved entering the brain cavity, and dealt with the problems of getting other loved ones out of complicated medical situations, I think I can say affirmatively when getting someone discharged the top priority is keeping things comfortable and sane for the discharged patient. It can do more harm to the person you love to make them wait 4 hours while you argue with nurses who clearly don't know what is going on (usually because they just started their shift and they never worked with the patient, or because they simply can't read the doctors handwriting on the chart), or can't be understood.

In fact a wait of that long in a situation like that could result in life threatening complications as heart surgery often involves the use of blood thinners and home dosage is either done by self injection or pills which are not monitored as closely as the heparin solutions given through IV. It's critically important that when discharging a patient the information does NOT need to be requested and in fact it should all be preprinted waiting for the pickup. This is standard practice in EVERY other hospital in the US. Bad information leads to malpractice suits, and not having a policy on information provided at discharge, or not following that policy can lead to VERY very expensive negligence or malpractice lawsuits.

If I were a shareholder of KP I would be livid after reading all this information. The company is clearly mismanaged and if it's not corrected immediately the entire company is at risk, not just from overspending of a change in cashflow, but lawsuits as this information release gives ambulance chaser lawyers significant basis for suit. Loss of information tied to implementation of a digital system that is inferior to the prior system and doesn't meet industry standards for availability (Healthcare IIRC has a standard of 99.9999% availability at most organizations with down times of less than 30 minutes) or accuracy is a major lawsuit waiting to happen. All you need is someone that has critical complications that disables them permanently or a death which can be tied to information availability and you have a multi billion dollar negligence, incompetence and malpractice suit. If there are any lawyers of the type mentioned above reading Slashdot they are likely salivating and are considering commercials on TV to find clients.

Can you say "will never work in IT again" ?

[binaryspiral]

While maybe he thought he was doing a great thing by bringing a wild injustice to light... there isn't a chance in hell I'd hire him for anything remotely considered a computer or communication device - I bet many others won't either.

Repeat after me: sir, can you spare some change?

That'll be a useful skill for you.

Re:Can you say "will never work in IT again" ?

[Wolvie MkM]

Yea... How selfish of him doing the right thing trying to bring to light the potential endangerment of thousands of lives... Big trouble maker.

The world needs more people with spines like this kid, good for him.

Re:Can you say "will never work in IT again" ?

[Anonymous Coward]

This kid was a fresh graduate who had very little accurate information to base his rant on. And by doing it in the most damaging manner possible shows me that he's either a complete moron or a pompous ass who thinks he should be running the company.

And IMHO the world has enough of both of them.

Mod me down for speaking the truth - but I bet you wouldn't want him as an employee if you're a project manager or team lead knowing if you screw up, he's going to pound out a scathing email about all your mistakes and project issues - and bulk send it.

And as far as endangerment... their system was on the verge of disaster well before epic came in to fork lift them off their overloaded systems and on to their own software deployment.

Re:Epic Systems? No wonder...

[Cerberus7]

I work for a fairly large healthcare system that is in the midst of deploying Epic's products, the first go-live is coming soon. I've heard the project is a mess, and people are quitting left and right. I guess I'll find out more soon...

Anybody got some useful references on how much Epic sucks and how screwed we are? I'd like to know ahead of time so I can start looking for a new job.

Can you say "Will join consultancy"?

[Anonymous Coward]

He'll become a consultant like the rest of us awkward bastards that put our heads over the parapet from time to time. The people who make the presentations will turn his negative assessments of the screw-ups he discovers into massive revenue opportunities. If I say internally "so-and-so hasn't a file system checking clue because XYZ", the project manager may translate that as "There is an opportunity to improve your performance in such and such area, get the check book out."

He's just written the consulting industry a big CV saying "Hey, I understand how really big projects can really foul up."

Repeat article and get some perspective

[MrNovember]

First, the article is nothing new. It basically says the same thing as before, "some 22-year old got out of college and worked at Kaiser for 6 months only to find out that, hey, big IT is not perfect." I can't figure out why anyone cares what some dumbass kid is blabbering about. It would be completely different if it was the CIO saying this.

Second, it is true that the medical health system they're implementing is super-complicated but name one that isn't that does what everyone wants. The FBI spent $100m+ on a total failure of a virtual case file system which is far less complicated than an integrated medical health system. Kaiser themselves spent $1B trying to make their own system which was also a failure.

The current implementation is a big-ass system with 8.5 million patients and over 12,000 doctors and a hundred thousand other employees. Frankly, it's a miracle that any system works at all there let alone at 99.2% uptime. Think of all the moving parts! And, yes, I echo someone else's comment that I believe Citrix bears a portion of the blame for the uptime figure -- they kind of suck.

MrNovember

Re:Repeat article and get some perspective

[ruben.gutierrez]

I second that. Although, he didn't just get "out of college." He dropped out. Huge difference.

Typical

[Anonymous Coward]

I used to work for a major medical center. We were doing a re-architect of the system but when it came time to buy the middleware the product we recommended got shot down by the CIO and replaced with another middleware product that he had heard about at a cocktail party. It was very basic, lacked two-phase commit (we were reading and writing to multiple databases) and had a dot-boom marketing budget. I don't remember its name, but it was very popular for six months in the late 90's.

Anyway, when we explained the patient safety issues (misordered meds, wrong labs, etc) to the CIO he said, "the odds of that happening are low," (I had estimated 7 events per year, statistically), "and if there's a problem it'll be cheaper to settle the lawsuits than pay for the higher software maintenance".

The good middleware was maybe $100K cheaper than the really good stuff, which is small for the budget, but the CIO bonus was based directly on how much under-budget actual expenditures were, so that was maybe a $20K in his pocket decision.

I later heard form a colleague across the country a nearly identical story.

Anyway, I quit - I couldn't sleep at night supporting that kind of malfeasance. Most team members, however, didn't want to bother finding a new job.

Puny Americans and your tiny incompetence

[kahei]

Ha! Witness the failure of American-style capitalism to produce REAL waste! Why, here in the UK our glorious NHS have just recently thrown away 12bn of public money on a computer system that has ZERO percent uptime -- because nobody could decide what it was for and nobody wanted it! Compare that to the paltry few billions your private enterprise was able to throw away!

And that's not all! The graft, corruption, bribery and crime surrounding the NHS system was such that Accenture *refused to work on it* -- that's right, Accenture turned down a paycheck because they didn't want to be associated with the way the project was being conducted! Let's see some privately-owned HMO reach that kind of level -- THEN maybe I'll admit that your 'Capitalism' has some advantages!

And you know what else you get wrong? Kaiser's failure, like that of Enron and WorldCom, is big news! In the glorious United Kingdom, this kind of thing barely makes it to the fringes of public awareness! Why, the NHS project actually KILLED a few struggling UK startup companies just to distract attention from the activities of the bigger contractors (i.e. iSOFT). Let's see Kaiser do that. Can it do that? THOUGHT NOT! Because that kind of thing requires a GOVERNMENT that can tax and borrow!

(sound of 'Rule Britannia' gradually swells in the background)

And THAT, my American chums, is why the NHS is the envy of the world, and why I am proud to pay 10% of my income toward it.

Although obviously I have private health cover as well in case I get sick. Which all comes in kind of expensive.

Kaiser Swindling Congress

[Jimbo Moran]

This is interesting:

"Recently Kaiser has been trying to position itself as the "authority" on the automated medical record, and Kaiser executives have been trying to sell Congress on their amazing HealthConnect system.

The problem is HealthConnect is not a system: it's a fancy new label for a motley collection of old "legacy" applications combined with some new components from a vendor called Epic. The "Epic" modules were originally billed as the Automated Medical Record, but Kaiser was chiefly interested in the billing module, so they settled for a less ambitious project of combining a few Epic modules with the variety of workhorse systems that have been in place for years. It's an act of deception to present this situation to Congress as either a new, cutting edge system (even Epic is based on archaic technology) or a fully integrated system."

Read the rest

99.7% is a GOAL?

[multipartmixed]

Not to put too fine a point on it, I'd likely get fired for achieving that metric.

Re:Epic Systems? No wonder...

[ruben.gutierrez]

I work for a facility which has successfully implemented various Epic applications. I admit some of the work is more difficult than it needs to be. But, I think a lot of it depends on who is doing the work. In my experience, most of the analysts doing the work come from a clinical background. Some have technical backgrounds, and some think they have technical backgrounds. Clinical people and technical people are entirely different breeds! It's up to select individuals (mostly technical, again in my experience), to bridge the gap. If you don't have that bridge, then it really is a mess.

Can't expect patient to own their own care

[Anonymous Coward]

IAAD, and I can tell you this is a stupid idea. Plenty of patients are elderly and demented (Alzheimer's mostly) and are not even able to remember the time of day, much less 'own their own care'. Not to mention almost no patient could tell you that prescribing amoxicillin instead of vancomycin for a methicillin-resistant bug is a stupid idea.

Re:Epic Systems? No wonder...

[Cerberus7]

I think that's where the problem is. We have clinical people making technical decisions, which seems to be standard procedure for this company. Yeah, we're screwed.

If you had a temperature of 107F

[Flying pig]

your post-visit care presumably involved being placed in a wooden box and having your relatives say nice things about you.

Re:Epic Systems? No wonder...

[drgonzo59]

That figures. I applied to work for Epic and they couldn't even keep up with managing my phone number, contact info and interview appointment times. Then they had me take a test and one of the possible languages to write the test in was Cobol. Yeah, a paper and pencil programming test in Cobol. Sure they had C, C++ and Java but they also had Cobol for goodness' sake.

  Then I asked the recruiter what database systems they use and they mentioned CACHE and the MUMPS language. MUMPS is some archaic pre-C language and I'd rather eat razor blades than use it.

As if that wasn't enough, I did some more research about Epic and found all kinds of dirt on it. The founder and CEO, Judy Faulkner is crazy and doesn't appear in public for weeks at a time. Epic loves to hire recent grads from out of town (that keeps them isolated), give them a medium salary but work them 60 hrs/week. I said screw it and stopped replying to their email and picking up their phone calls. These people are crazy.

This whole mess with Kaiser P. just validates it. It was very stupid of Kaiser to pay hundreds of millions to Epic, it seems like they are reaping the "benefits."

I was interviewed to do internal training for this

[Anonymous Coward]

A few months ago, at the Pasadena, California one-stop-shopping office of the Unemployment Insurance Compensation entity [EDD], I had a first-round interview to be an internal trainer, training supervisor, and curriculum developer. The goal was to teach physicians, nurses, and technicians how to use the system. They wanted people with CS degrees and/or biomedical training.

I gave them a specially-composed Curriculum Vitae which emphasized my 40+ years of software experience, biomedical research publications, and 2000+ adult students I've taught in various classes. I appended my analyses of some refereed publications about the system.

Carefully not saying "you're overqualified" -- which people have learned in equivalent to the illegal statement "you're too old" -- they promised to contact me for the guaranteed second-round interview.

I bought a new suit for that eventuality.

Of course, they never contacted me for the second-round interview.

See what you get by always hiring the B-list people to handle a failed implementation of a badly-designed system, when the A-list people are carefully excluded, as a threat to the group-think?

Oh, and the EDD office wants me to stop printing "formulas" -- they said so in writing -- because they deem that misallocation of computer resource lab, for my applications for Tenure Track Math professorships.

Earth to bureaucrats, come in please...

-- Jonathan Vos Post

Re:Epic Systems? No wonder...

[drgonzo59]

Exactly. Just do a search for Epic Systems employment and you'll find plenty info Epic. I am sure your "sigh" means that you already do know. I wouldn't wish even my enemy to work for Epic Systems. That place is so messed up, it is scary. Their CEO and founder Judy Faulkner is crazy, she decorated the place with bright yellow and blue walls, crazy wierd "art" pieces and she never shows up to interact with anyone and just hides for days on end. That is still the best part. They hire naive fresh grads, promise them work in an "exciting upstart company" (even though it was started in the early 80's) and then gives them a fixed salary and makes them work 60 hour weeks. And they really love to hire H1B workers, they pay for their visas but them make them slave for the company for 5 years minimum. And MUMPS, they use MUMPS! Oh, my God...(I wanted to apply to work there, actually saw their job add on Slashdot and thought I'd give it a try) when I read the description of MUMPS it made Cobol look new and exciting.

I don't know how Epic ever managed to scam Kaiser to secure a contract with them, I am sure Kaiser learned their lesson after this...

Dilbert...

[evilviper]

they say their goal for system availability is 99.7% (they're currently at 99.2%)."

Reminds me of a PHB speech in a Dilbert book... Typed from fuzzy memory:

"This year our goal is 0 serious injuries! In hindsight, last year's goal of just 20 serious injuries was a mistake. We had to seriously injure 6 people to meet that goal."

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Epic Systems? No wonder...

[kallisti]

I have some difficultly believing any of this.

I worked for Epic ten years ago, when I joined they had about 40 employees, when I left they were pushing 300. Judy wasn't seen for long periods at a time because she was working insane hours, the woman never stopped working. The art was a little goofy, but at least she tried something more than cubicles. I NEVER worked a 60 hour work week when I was there, even in a big crunch. I was a key programmer on the Visual BASIC front end that apparently grew into this HealthConnect thing. I remember questions as to how scalable the system would be, I guess those questions have been answered.

Yes, MUMPS has some very weird properties, but it's really not that bad for what it does. You're comparing it to languages written 20 years later, and a lot of the really odd things (1 letter commands, 6 character filenames, for example) are long since history. Don't get me wrong, I'm glad to be rid of it, but it's not as bad as people make it out to be. It's made for a specific purpose and does it well.

We had few or no H1B people when I was there, I recall one guy from Bulgaria and that's it. I was treated extremely well, and for a job right out of college, I cannot complain. If all of this you say is true, then that's really sad.

This whole thing makes me really sad.

I worked there, but I'm all better now

[Covert Penguin]

I worked for KPIT back in the days when HealthConnect was known as the EHR (Electronic Health Record) project. EHR was deemed a failure after a few years, but in typical Kaiser IT fashion they merely changed the name of the project and continued down their futile path. EHR became CIS (Clinical Information System). It was a running joke among the employees about how we weren't allowed to call CIS what it was, "the new EHR". I wasn't surprised to hear that the failing project had been renamed a third time to keep it's momentum. Funny thing about Kaiser is that they'll probably scrap the HealthConnect name after this blows over. I've yet to see another organization as screwed up as Kaiser Permanente's IT department, and I was in the military for 4 years. Enjoy the show folks!

Re:Epic Systems? No wonder...

[Anonymous Coward]

Clinical and technical people need to learn more about the others area, but this should not substitute for real technical people calling the tech shots and real clinical people calling the clinical ones.

I am a technical person who has been able to create a strong working relationship with our clinical analysts and this has been hugely beneficial for all of us. I have learned more about their work than I ever thought I would and could probably pass for a CNA/QMA at times. They have learned a lot as well and are capable tech analysts as well as clinical.

However, I would not want them to make significant technical decisions concerning system architecture. Nor should I make any significant clinical decisions because neither of our knowledge bases are complete in the other's territory. There are way too many variables in both areas for one to pick up the other's knowledge completely in a casual way. Only truly in depth training can train either to do the other's job.

BC

Re:Epic Systems? No wonder...

[Cerberus7]

Clinical and technical people need to learn more about the others area, but this should not substitute for real technical people calling the tech shots and real clinical people calling the clinical ones.

I agree wholeheartedly. Where I work, though, the clinical people make all the decisions, and when things turn out poorly, the technical people get all the blame. There's a pretty substantial divide between the two, and neither side seems to want to set things right.

Disclaimer: I work for Epic

[Anonymous Coward]

Health care projects are always messes, and Epic installs always seem like huge messes until somebody flips the "on" switch because our software is customizable (and previously it was customizable to a fault, as in Epic employees would spend weeks asking painfully obvious questions about workflows to try to make everything "just right").

Most other health care software companies are much more out-of-the-box. Easy install, and no availability for training, and sucks to be you if your hospital isn't set up the way your vendor thinks it should be. There have been go-lives from these companies where nothing happens. Like, literally, nothing happens, no EMR, no CPOE, nothing.

Epic gets rated higher than everyone else pretty consistently in KLAS and Gartner - though others are catching up.

As for the Kaiser install, it's way over budget because they've ended up installing twice the software they originally intended to. The rough analogy I can make is that they meant to get a hamburger but they ended up getting onion rings and a milkshake too, and the rat bastard restaurant owner made them pay extra!

And as for attacks on the company itself, they're up front about what you'll be programming in, and while I was waiting to be interviewed I shared a couch with a Wile E. Coyote statue made of copper wire. Our eccentricity is well-documented and harmless. Plus our cafeteria is freakin' awesome.

Re:Epic Systems? No wonder...

[Anonymous Coward]

Hi Ron - don't worry, none of this is true. We are now over 2,500 people, and still growing at insane rates, even have an office in Europe...

--The one guy from Bulgaria