Slashdot Mirror
Fed-Up Hospitals Defy Windows Patching Rules
bingbong writes "According to Network World: 'Amid growing worries that Windows-based medical systems will
endanger patients if Microsoft-issued
security patches are not applied, hospitals
are rebelling against restrictions from device manufacturers that have
delayed or prevented such updates. Device makers such as GE Medical Systems,
Philips Medical Systems and Agfa say it typically takes months to test Microsoft patches because they could break the medical systems to which they're applied. In some instances, vendors won't authorize patch updates at all.' This is the typical patch vs. crash problem. Unfortunately, the stakes here could be human lives."
Why is hospital equipment running windows? Anyone that knows anything about embedded systems with high quality requirements know that you stay away from large OSes. Even Linux is avoided unless you need tcp/ip and if you don't then its better to have a small maybe even off the shelf OS. The Key is to limit the testing requirements and limit changes, which are goofy to test a life support system just to have the latest and greatest IE 6 or 7 that you shouldn't even, have hooked to a wide-open Internet anyway.
Why are they even accessible on the internet? Seems like these should be in a secure private network unlikely to be attacked.
pshaw! what's a few human lives when network security is at stake?
OK.... We now have the Food and Drug Administration in charge of computer security?
...do they not just put these devices and systems behind something as simple as a $50 hardware NAT firewall, especially for a device that costs hundreds of thousands - or millions - of dollars? (Or better yet, why does the vendor not integrate such protection if they're relying on network-connected Windows systems for device control/interaction?)
The norm is that these devices may need to connect *out* to something else, but don't necessarily need any inbound connections, so a hardware firewall, or even a host-based software firewall, would work perfectly in most instances; those that do need externally initiated inbound communication can *still* set up the necessary rules to allow such communication to take place. And yes, it is just this simple. (I did RTFA, and noted that some vendors actually recommend this, but that, startlingly, "there have been several instances in which viruses originated from medical instruments straight from the vendors"!)
I work for a hospital,and I have to say that our network may be 'stable' but it really sucks. We run Windows2000 Pro with many problems, and frequent crashing. If one of our secondary databases crashes, as they seem to do often, we have to wait a day or two until we can get a reboot of the system because the main database runs on the same server. Productivity really goes down the tubes sometimes to allow for the 'stable' network.
Boxing Equipment Reviews
Medical machines responsible for human life should never need to be patched. The software was tested at one point and should be controlled to stay at that test point until it is to be retested. For machines running windows this means they should be segregated from other parts of yoru network and should be airgap firewalled from the rest of the world. Intenet worms and email trojans shouldn't be relevant.
I'm not a big fan of Microsoft, but I don't think the quality (or lack thereof) of their products is the issue here. I've read from their EULAs that their products are not suited towards critical applications (ie nuke facilities, life support). My point is that although a EULA is not a legally-binding contact, the fact that MS is stating in public Windows shouldn't be used in critical applications should tell you something. The bottom line is that if GE, Philips or Agfa build a medical system, they should be responsible for that product from the software up to the hardware. The fact that *they don't have control* over one of the components in their products (the underlying OS) is negligent, IMO.
I would get laughed out of court if I tried to blame a critical problem with a report I wrote on my secretary, and the same should happen with these companies if somebody's loved one dies from their irresponsibility.
Bill Clinton: Pimp we can believe in. - The Shirt!!!
My father works for GEMS as a Field Service Engineer; he repairs and installs X-Ray Machines, CAT Scanners, and Mamography machines. As far as I know, GEMS doesn't run Windows on any of it's boxes (other than Engineer Laptops). Most of their older systems are UltraSPARC/SunOS boxes. The newer ones are Intel Xeon/Red Hat rigs with their own custom window manager. Heh, he's even called me in a few times to help him with some Linux problems.
It makes sense to me, GEMS and the Hospitals aren't going to risk $500,000 to $2,000,000 machines because of Microsoft's poor track record. Not to mention, a bug in the software can bring down the system for hours, until someone can come in and fix the problem. My Dad has problems all the time with doctors breathing down his neck. Most the time they have a full schedule, and when a x-ray tube blows it can take up to 4 or 5 hours to replace. Not including shipping from Wisconsin or France.
Bugs are just features that have been fixed.
I.e. while one can build a simple manometer the reality is that blood pressure devices used today probably have all sorts of interdependancies that can cause a ripple effect, so one should be pretty darn careful before just applying patches licky-split ... in a work discussion earlier today, we talked about how one of the recent Microsoft security patches broke one of our applications.
Hulk SMASH Celiac Disease
I work in one of the top hospitals in the US (Top 100 Wired, top 25 in a lot of the US News and World Report rankings, etc) as the principal technology architect, and I can say that people are idiots for going nuts and patching immediately.
Our CIO, who's pretty well respected among his peers, asked us last week on deployment schedules for this. We pushed back and said, if we deploy now, we'll run into a host of issues. Over the weekend we did some cursory testing against most of our Patient care apps (a lot are web based) such as Cerner Millennium and GE's CentricityWeb. We're far ahead in the CPOE game for healthcare, so our devices are used for input of labs and orders.
Most of the biomed equipment we have doesn't run Windows. Personally, if you do your environment right, then you shouldn't have to worry about viruses and stability.
Healthcare doesn't function like the rest of the business world. It's a completely different animal.
I work with MRI scanners, so I know about these issues very well, and here's an example from my own experience:
An old colleague of mine got funding to start his own reasearch group, meaning he got his own MRI scanner. He asked me to consult on some software that would extract the data from the console of a Siemens scanner (at the time, the console was based on an OLD version SunOS, whose native compilers did not even conform to standard ANSI C) and send it directly to another computer running software that we use for data analysis. The dialect of C was a little strange, but within a week, I was able to get the software together, and my colleague was able to do the type of experiments he wanted to. And his scanner hummed along. This was back in 2001.
Fast-forward to the present. His console has since been "upgraded" to Windows XP system, and in the times I've spoken to him, he's had nothing but bad things to say about the stability of the "upgraded" system. And it's not that he had a choice, as support for his previous system was phased out. So now patients, doctors and reasearchers in his group are at the mercy of the moods of an XP system. And mind you - this system is not even on a publicly accessible network. It is on its own dedicated, private network, and its stability still can't be maintained, even by the support staff of the scanner manufacturer.
When it comes down to it, Windows still does not have the stability (never mind the security issues to cut it in really "mission-critical" situations). Maybe in cases where you need your e-commerce site up, running, and handling 1000s of transaction per second. But NOT when peoples' lives are involved.
Firewalls won't help. If it runs Windows, some idiot's going to bring in a CD full of pictures from his latest vacation and the CD's going to be infected with MyDoom or (heck, probably and...) Sobig or any number of other nasties. Or it's going to be something he wants to print on the nice laser printer at the office.... there's a hundred ways to get infected just by clueless users.
Pretty soon, the internal network's either too busy generating random traffic to do anything else-- and even if the Big Iron of the business, the dialysis machines and heart-lung devices and all those wonderful things that better damned well not break work fine, you've still got the terminal the nurse sits in front of that keeps track of when to issue you your shot that keeps you alive spending half its time rebooting because it's got Sasser.
This is not a problem a firewall can solve, and it's pretty darned big: You can't go throwing software around willy-nilly to solve this problem (even though the real problem is that the users _are_ throwing software around willy-nilly), so you can't just go "oooh! A next-day patch from Microsoft, let's hope their two hours worth of QA before it walked out the door was good enough!".
-JDF
All computer systems involved in patient care (and paper tracking as well) are forced to go through governmental processes for design, documentation and testing. These regulations add weeks, if not months, to system changes, regardless of change scope.
Case in point is the drug study setup. Setting up data entry screens and processes can take up to 6 months for a given trial, and that trial may only run 3 months for the study metrics. If any of these processes are documented incorrectly, and entire trial can be dropped and the drug denied.
This, in the hospital realm, is all about CYA. If a piece of equipment is not certified to this extent, the hospital can be held more liable for patient injuries if said equipment falters.
> Unfortunately, the stakes here could be human lives.
Soon to be made into a movie starring Uma Thurman.
It's called "Bill Kills".
assert(birth_date<time-86400)
The article mentions one thing that needs to be emphasized, which is where the FDA guy states that they're not going back to the dark ages where systems don't talk to anything else. For years, every device was on its own proprietary network (if it was on a network at all), and talked to itself and absolutely nothing else. This was bad.
In only the last couple of years (because medical IT is very behind the rest of the IT industry in a lot of ways) these devices have moved rapidly to using commodity protocols and network infrastructures, driven by hospitals' needs to do all of this more cheaply, and not have a lot of chaos.
Also, they want to provide some value add on top of the monitoring systems. For instance, it's nice to be standing by the patient's bed and see the monitoring data. It's even better to be able to export that data to another system so that it's more useful, or display it on a website so MDs can see it. All of this requires networking capability, and Microsoft (like it or not) is considered a leader in the field for server software, and has a large division providing solutions to healthcare.
Overall, the more advanced features you want a clinical system to provide, the more that system needs to integrate with other systems. Companies have given up reinventing the wheel on this every time, and are basing what they do on standard software and protocols. Microsoft is one of those. We try to avoid it whenever possible, however in most instances the decision for one product over another is based on clinical value, and not IT preference.
"Why, exactly? Because nobody would know how to hack your tiny little proprietary OS? That's crap and you know it."
The reason it the smaller the OS the less you have to test it. The whole KISS thing. Keep it simple stupid.
On a standalone ebedded system you do not need support for TrueType fonts, every printer and USB device known to man, or even video playback. On an Embeded device you often only need a few functions but those functions have to work. If you have ever programmed under windows you will find all sorts of APIs just do not work or do not work the way they are documented. Windows programers just program around these issues. You should always use the smallest OS that you can get away with for the device you are using. Linux is a good option for very flexable embedded devices. I would tend to stay clear of X and use nano-x myself.
There are many off the shelf ebeded OSs the most popular I can think of is QNX. For life critcal systems I would go for QNX over windows any day.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Why don't they design their software, so that it doesn't break when patches are applied?
You don't seriously believe that Microsoft gives anyone advance notice of what the patch is going to break, do you? Have you seen the ambiguous and undetailed language that goes with the WinXP SP2 patch? There's nothing actionable in there, certainly nothing testable. Until GE gets it and tests it, and authorizes it for the build, it's an astonishingly risky thing to install it.
21cfr11 mandates that only the tested configuration can be used, and if the hospital choses to violate that federal statute, they are not just at risk of screwing up their scanner, but they're technically in violation of federal statute.
I'm not defending Microsoft here, nor am I saying it's smart to have Windows in scanners, but it's there (less now than 5 years ago, but still there). The penalty for using it is that it's quite likely that some piece of malware _will_ find its way into the scanner. They're more vulnerable if they don't patch, they are going into an unsupported (and unsupportable) configuration if they do patch. The only answer is to not use Windows, but until all the 'doze-based scanners are history, they're stuck with it.