Fed-Up Hospitals Defy Windows Patching Rules

bingbong writes "According to Network World: 'Amid growing worries that Windows-based medical systems will endanger patients if Microsoft-issued security patches are not applied, hospitals are rebelling against restrictions from device manufacturers that have delayed or prevented such updates. Device makers such as GE Medical Systems, Philips Medical Systems and Agfa say it typically takes months to test Microsoft patches because they could break the medical systems to which they're applied. In some instances, vendors won't authorize patch updates at all.' This is the typical patch vs. crash problem. Unfortunately, the stakes here could be human lives."

Comments

Stop playing solitaire on my dialysis machine

[Anonymous Coward]

Why is hospital equipment running windows? Anyone that knows anything about embedded systems with high quality requirements know that you stay away from large OSes. Even Linux is avoided unless you need tcp/ip and if you don't then its better to have a small maybe even off the shelf OS. The Key is to limit the testing requirements and limit changes, which are goofy to test a life support system just to have the latest and greatest IE 6 or 7 that you shouldn't even, have hooked to a wide-open Internet anyway.

Re:Stop playing solitaire on my dialysis machine

[dekemoose]

They are running Windows for the same reason that they are connected to a network, some pinhead PHB somewhere is trying to save a buck. It's probably cheaper for them to develop on a Windows platform rather then on a proper embedded paltform. Just like its cheaper for them to put these devices on a shared network, rather than having them properly firewalled off onto their own secured environment. Follow the $$$.

Re:Stop playing solitaire on my dialysis machine

[mattOzan]

I find it hard to believe they are talking about life support machinery. No specific piece of equipment is ever mentioned, just the generic "medical devices." I'm thinking they are speaking more of hospital informatics systems, like Stentor and EpicCare. When a doctor can't read a patient's medical chart because the workstation is PWNED, or can't send an X-ray up to surgery because the router's been hijacked, that is definitely a problem; but it is somewhat less of a problem than your ventilator quiting because of a BSOD.

Sounds like a tech-challenged reporter reporting wide-eyed about crashing "medical devices" which she doesn't really understand.

Re:Stop playing solitaire on my dialysis machine

[Short Circuit]

Crashes would be a problem.

If it crashes, how do you know if the radiation dose was administered or not? Was it the whole dose? was it just part of the dose? Did the machine even turn off?

Those are awfully important questions for the doctors and radiation techs. Even moreso for the cancer patient that has to go through a battery of tests to determine the effect of a software glitch.

Re:Stop playing solitaire on my dialysis machine

[MindStalker]

But the point is still the same, you should run a machine with only enough code to do the job. Extra cruft is just risking "bugs" which could cost lives.

Re:Stop playing solitaire on my dialysis machine

[FunnyBunny]

Very honestly, most of these machines couldn't "kill omeone".

Hmm, a pain pump that doesn't correctly meter the morphine could easily kill someone.

I mean, if the radiation therapy machine crashes, nobody dies.

Wow, you mean if the control computer crashes leaving the shutter to the Cobalt source open nobody could die? How about gamma knife overexposing the brain stem, cooking the brain stem couldn't possibly kill someone. How about a faulty homing cycle where the radiation head homes to the patient table, even if a patient is there.

Do me a favor, don't work on human critical systems.

Re:Stop playing solitaire on my dialysis machine

[FFFish]

Don't be silly. The system should be based on an OS that is proven hard-core stable and real-time, like QNX, Microware OS-9/9k, etc.

There are a ton of good OSes out there for specialty applications and, surprise!, most of them don't involve Linux! Linux is not the be-all and end-all of OSes.

For human-life-critical applications, you should be using something that is demonstrably proven.

Re:Stop playing solitaire on my dialysis machine

[CyberGarp]

This happened to me in a hospital:

I was admitted for severe breathing difficulties and chest pain. This put me on the heart attack route. Turned out to be a rare form of asthma. While I set in a bed on oxygen, I looked up to watch my heart monitor flat-line. The crash cart crew runs in with all the resucitation equipment and my heart monitor starts beating again. They give me weird looks and examine me up an down to see that I'm doing great on the oxygen. This happens a second time. About 10 minutes later the hospital IS staff show up and examine it, and he says, "Aha, yours is set on the network to show the guy next door."

He leaves and I hear the crash cart go whizzing by my door.

Networked critical care systems are a bad idea--except to report a central monitoring station. Windows is an even worse idea. Why this kind of crap is tolerated is beyond me.

Shawn

Re:Stop playing solitaire on my dialysis machine

[Valar]

This is a very precise process. We are talking exposures MUCH less than a second. By the time a human operator can respond to the malfunction it is already too late. If the exposures were long enough that a human could administer them, then they would. No point in paying for a computer _and_ a tech, right?

Re:Stop playing solitaire on my dialysis machine

[danheskett]

Because the alternative is alot worse, that's why.

Imagine you are a small hospital, one with a 10 bed ICU. You have 10 patients. Can you afford to have someone near enough to each heart monitor to hear when it has an irregular heartbeart? Can you even detect a slightly altered heart-rate just by a casual listen/look every now and then? What about all the other funny intrumentation? Of course not. It would take one RN/CNA/Med. Tech per ICU patient per shift. In 3 shifts that's 30 full-time employees for a 10-bed ICU just to make sure nothing bad is happening on the monitor. That's a big staff. All the sudden you are spending $2M a year on just nurses/assistants for your 10-bed ICU. At best you can recoup $500k of that, maybe $1M if you have a really good ratio of paying/insured/uninsured/unpaying patients.

Imagine you are not a 10-bed ICU, but rather, a 750-bed ICU. Do the math.

Whats worses is that in your case it wasn't even likely a networking related problem, so much as it was likely that a the inputs from one machine were improperly patched through to a display and monitoring unit. (I've seen the same mistake before.. when you have a patch panel it's an easy mistake to wire jack 3 to jack 5 and vice versa).

It's not acceptable, but in reality, it's a cost of progress. The alternative to network'd equipment like this is worse care. Systems must be designed to be resliant, and some manufactuerer's are doing a bad job. But, by and large, medical technology is amoung the most robust in the entire computer industry. I've seen machines that run 24-hrs a day for years on end. Machines with duty cycles in the thousands of hours.
Networked care systems are coming, and many are here and work very well. Many many many more lives have been saved than damaged or lost by this type of technology. We need better systems, better platforms, etc - but throwing out the whole thing is absurd at this point in time.

Re:Stop playing solitaire on my dialysis machine

[Anonymous Coward]

They are not usually using Windows for embedded systems. They are using Windows on workstations.

I am a clinical doctor who programs in a couple of dozen languages and environments and follow the advancement in software solutions closely. I have been involved with clinical informatics only recently for the past couple of years though. Allow me to explain you some of the realities of the current health industry software. I admit I haven't sat down and structured the text well but I tried to put in as many issues as I could think of the moment.

The doctors want Windows or Macs. They want a familiar set up compared to what they use at home. It is very difficult to get doctors learn a completely different paradigm. There have been documented cases where nearly all the doctors in certain institutions rose up in arms because the developers thought they knew better and tried to force a solution onto them.

Most of the current set ups are almost always heterogeneous. We buy software from multiple vendors and bridge them together. This is because there are no completely integrated solutions as yet. GE and a few others are trying to close this gap but it is a VERY difficult one. Hospital information needs are not standard as your usual business information needs. The data processing here is often very simple but the volume and complexity of the data is overwhelming. It is not simple as Customers and Invoices. Clinical Medicine deals a lot with relatively abstract data with complex relationships. Most doctors know these relationships intuitively but there isn't enough published literature for a software developer to draw from. Clinical software is extremely expensive to build since the requirements are hard to establish. A lot of iterations are needed to fit the software to a given practice (This never gets completed usually and people settle for close enough).

Doctors themselves understand their needs best. A few doctors, while they don't hold CS degrees, practice design patterns or do EJB, do quite well to put together MS Access databases to solve their problems where professional software developers have not yet tread. Many times, they distribute these to their colleagues freely (Open Source if you will). Few even sell them. They may not be the best designed tools but they work. Mac's FileMaker and Linux's Total Rekall? don't exactly come close. Windows tools also have a larger number of books available to learn from.

Platform and tool costs are trivial, developer costs are not. A study in Human Computer Interactions is very essential here. Rich user interfaces are always preferred. Non-windows platforms don't have sufficiently advanced RAD tools. I really wanted Kylix to succeed. But I don't see any momentum behind it anymore. Veteran's Affairs Hospitals have built a remarkably physician friendly system. They are rightly proud of their constantly iterative development. They used Delphi but now that the system is stable (for user experience stand point) they are looking for other platforms. They looked at .NET. I heard they were trying Java now. Personally I am not sure it is the right choice for the client but we shall see.

The loss of work hours because the software does not fit the workflow at a given hospital is far far greater than losses due to worms and viruses.

The software should be as intuitive to use as possible. Should not require reading manuals. Hospitals always look whether the given software will slow the physician down in any way because physician time is very expensive and they rather have them seeing patients and generating revenue.

There is a case for cross-platform tools at the moment too. It is a case of mobility. Most doctors like to be able to review a patient's case online and advice on the phone when necessary. Many vendors provide web pages and applets for this but they often end up very unergonomic. But since the need is often information retrieval rather than data entry, they are accepted in the absence of the better alt

Why do they need patching?

[Anonymous Coward]

Why are they even accessible on the internet? Seems like these should be in a secure private network unlikely to be attacked.

Re:Why do they need patching?

[AKAImBatman]

Why are they even accessible on the internet? Seems like these should be in a secure private network unlikely to be attacked.

Who said they're on the internet? Consider the following scenario:

The Hospital PCs are connected to a primary server that backs up all data and managed the PCs.

The Primary Server has a leased line or occasional dial-up to transfer data to a state-wide backup and update site.

The backup and update site has firewalled internet access for a VPN to GE, and troubleshooting purposes.

GE communicates with customers via internet email. One clerk in a backroom opens an attachment with an RPC worm. Within a half-hour the entire chain is compromised.

Any question on why having a monosystem Windows network is a bad thing? Even ONE Unix server in there would help break the chain.

FDA?

[gtrubetskoy]

...when the FDA eight years ago began allowing off-the-shelf software in medical devices, it didn't foresee the kinds of security issues, such as computer worms, that plague networks.

OK.... We now have the Food and Drug Administration in charge of computer security?

Doesn't have to be a issue

[bs_testability]

Medical machines responsible for human life should never need to be patched. The software was tested at one point and should be controlled to stay at that test point until it is to be retested. For machines running windows this means they should be segregated from other parts of yoru network and should be airgap firewalled from the rest of the world. Intenet worms and email trojans shouldn't be relevant.

Re:Why in the hell...

[pclminion]

put these devices and systems behind something as simple as a $50 hardware NAT firewall, especially for a device that costs hundreds of thousands - or millions - of dollars?

How is a firewall going to stop an insider from exploiting the network? Does working in a hospital magically transform a person into a paragon of morality?

Can't say I'd blame Microsoft this time around.

[Rude Turnip]

I'm not a big fan of Microsoft, but I don't think the quality (or lack thereof) of their products is the issue here. I've read from their EULAs that their products are not suited towards critical applications (ie nuke facilities, life support). My point is that although a EULA is not a legally-binding contact, the fact that MS is stating in public Windows shouldn't be used in critical applications should tell you something. The bottom line is that if GE, Philips or Agfa build a medical system, they should be responsible for that product from the software up to the hardware. The fact that *they don't have control* over one of the components in their products (the underlying OS) is negligent, IMO.

I would get laughed out of court if I tried to blame a critical problem with a report I wrote on my secretary, and the same should happen with these companies if somebody's loved one dies from their irresponsibility.

Everyone's asking why aren't they firewalled...

[foxtrot]

Firewalls won't help. If it runs Windows, some idiot's going to bring in a CD full of pictures from his latest vacation and the CD's going to be infected with MyDoom or (heck, probably and...) Sobig or any number of other nasties. Or it's going to be something he wants to print on the nice laser printer at the office.... there's a hundred ways to get infected just by clueless users.

Pretty soon, the internal network's either too busy generating random traffic to do anything else-- and even if the Big Iron of the business, the dialysis machines and heart-lung devices and all those wonderful things that better damned well not break work fine, you've still got the terminal the nurse sits in front of that keeps track of when to issue you your shot that keeps you alive spending half its time rebooting because it's got Sasser.

This is not a problem a firewall can solve, and it's pretty darned big: You can't go throwing software around willy-nilly to solve this problem (even though the real problem is that the users _are_ throwing software around willy-nilly), so you can't just go "oooh! A next-day patch from Microsoft, let's hope their two hours worth of QA before it walked out the door was good enough!".

-JDF

Re:Stop with the security through obscurity crap

[LWATCDR]

"Why, exactly? Because nobody would know how to hack your tiny little proprietary OS? That's crap and you know it."

The reason it the smaller the OS the less you have to test it. The whole KISS thing. Keep it simple stupid.

On a standalone ebedded system you do not need support for TrueType fonts, every printer and USB device known to man, or even video playback. On an Embeded device you often only need a few functions but those functions have to work. If you have ever programmed under windows you will find all sorts of APIs just do not work or do not work the way they are documented. Windows programers just program around these issues. You should always use the smallest OS that you can get away with for the device you are using. Linux is a good option for very flexable embedded devices. I would tend to stay clear of X and use nano-x myself.
There are many off the shelf ebeded OSs the most popular I can think of is QNX. For life critcal systems I would go for QNX over windows any day.