Slashdot Mirror
Fed-Up Hospitals Defy Windows Patching Rules
bingbong writes "According to Network World: 'Amid growing worries that Windows-based medical systems will
endanger patients if Microsoft-issued
security patches are not applied, hospitals
are rebelling against restrictions from device manufacturers that have
delayed or prevented such updates. Device makers such as GE Medical Systems,
Philips Medical Systems and Agfa say it typically takes months to test Microsoft patches because they could break the medical systems to which they're applied. In some instances, vendors won't authorize patch updates at all.' This is the typical patch vs. crash problem. Unfortunately, the stakes here could be human lives."
Why is hospital equipment running windows? Anyone that knows anything about embedded systems with high quality requirements know that you stay away from large OSes. Even Linux is avoided unless you need tcp/ip and if you don't then its better to have a small maybe even off the shelf OS. The Key is to limit the testing requirements and limit changes, which are goofy to test a life support system just to have the latest and greatest IE 6 or 7 that you shouldn't even, have hooked to a wide-open Internet anyway.
Why are they even accessible on the internet? Seems like these should be in a secure private network unlikely to be attacked.
pshaw! what's a few human lives when network security is at stake?
OK.... We now have the Food and Drug Administration in charge of computer security?
Why are these things on any sort of publicly accessable network? They should, at least, be on a private network that's physically separate from everything they don't absolutely need to talk to & firewalled all to hell.
my sig's at the bottom of the page.
...do they not just put these devices and systems behind something as simple as a $50 hardware NAT firewall, especially for a device that costs hundreds of thousands - or millions - of dollars? (Or better yet, why does the vendor not integrate such protection if they're relying on network-connected Windows systems for device control/interaction?)
The norm is that these devices may need to connect *out* to something else, but don't necessarily need any inbound connections, so a hardware firewall, or even a host-based software firewall, would work perfectly in most instances; those that do need externally initiated inbound communication can *still* set up the necessary rules to allow such communication to take place. And yes, it is just this simple. (I did RTFA, and noted that some vendors actually recommend this, but that, startlingly, "there have been several instances in which viruses originated from medical instruments straight from the vendors"!)
I work for a hospital,and I have to say that our network may be 'stable' but it really sucks. We run Windows2000 Pro with many problems, and frequent crashing. If one of our secondary databases crashes, as they seem to do often, we have to wait a day or two until we can get a reboot of the system because the main database runs on the same server. Productivity really goes down the tubes sometimes to allow for the 'stable' network.
Boxing Equipment Reviews
Medical machines responsible for human life should never need to be patched. The software was tested at one point and should be controlled to stay at that test point until it is to be retested. For machines running windows this means they should be segregated from other parts of yoru network and should be airgap firewalled from the rest of the world. Intenet worms and email trojans shouldn't be relevant.
I'm not a big fan of Microsoft, but I don't think the quality (or lack thereof) of their products is the issue here. I've read from their EULAs that their products are not suited towards critical applications (ie nuke facilities, life support). My point is that although a EULA is not a legally-binding contact, the fact that MS is stating in public Windows shouldn't be used in critical applications should tell you something. The bottom line is that if GE, Philips or Agfa build a medical system, they should be responsible for that product from the software up to the hardware. The fact that *they don't have control* over one of the components in their products (the underlying OS) is negligent, IMO.
I would get laughed out of court if I tried to blame a critical problem with a report I wrote on my secretary, and the same should happen with these companies if somebody's loved one dies from their irresponsibility.
Bill Clinton: Pimp we can believe in. - The Shirt!!!
Survery says... Beeep! Beeep! Beeep!
What "security" or other risk with a turnkey standalone system? I'd rather risk the remote chance of someone breaking into my room to run CAT-5 to my vitals monitor rather than a BSOD (possible REAL death in this case) because Service Pack x broke some obscure function and failed to alarm the nurse when my heart stopped.
Do the morons at the hospitals run Windows Update on the defibrillators?
The manufacturers have tested and retested and regression tested everything that goes into those medical devices (or they say, anyway), so why deviate from a known good combination without a compelling reason?
This comment does not necessarily represent the views and opinions of the author.
My father works for GEMS as a Field Service Engineer; he repairs and installs X-Ray Machines, CAT Scanners, and Mamography machines. As far as I know, GEMS doesn't run Windows on any of it's boxes (other than Engineer Laptops). Most of their older systems are UltraSPARC/SunOS boxes. The newer ones are Intel Xeon/Red Hat rigs with their own custom window manager. Heh, he's even called me in a few times to help him with some Linux problems.
It makes sense to me, GEMS and the Hospitals aren't going to risk $500,000 to $2,000,000 machines because of Microsoft's poor track record. Not to mention, a bug in the software can bring down the system for hours, until someone can come in and fix the problem. My Dad has problems all the time with doctors breathing down his neck. Most the time they have a full schedule, and when a x-ray tube blows it can take up to 4 or 5 hours to replace. Not including shipping from Wisconsin or France.
Bugs are just features that have been fixed.
The recent times I've been in hospitals I've checked to see what they're running. The two major hospitals near me don't appear to have the real "life and death" equipment running Windows. I'm talking about vital stat monitors and other surgical recovery equipment. I've seen certain medical records being accessed on Windows-based systems. Perhaps then there could be issues with lost information as to current prescription or observational data being lost or corrupted.
But even then wouldn't such systems be running separate from the public Internet? If so, on top of that wouldn't they be secure enough so that executives with their laptops can't just plug in and hose things up? With even entry-level expertise IT staff should be able to separate these boxes onto some sort of a VLAN that would secure them by default. What are the IT folks' take on this who are working front line in the medical arena?
I was going to complain about how Windows is not appropriate for embedded devices, but then I reread the article for examples. They don't make one mention to any kind of "device." The only thing they mention is some system by Kodak for transferring images. I think the word "device" is there to scare the public into thinking that their heart monitors and chemotherapy machines are going to be infected. I doubt these devices have hard drives or TCP/IP connections to infect. More likely, they are talking about hospital computer systems. My experience in the Medical Informatics biz is that this sector is technologically further behind than any other section of IT.
I.e. while one can build a simple manometer the reality is that blood pressure devices used today probably have all sorts of interdependancies that can cause a ripple effect, so one should be pretty darn careful before just applying patches licky-split ... in a work discussion earlier today, we talked about how one of the recent Microsoft security patches broke one of our applications.
Hulk SMASH Celiac Disease
I work in one of the top hospitals in the US (Top 100 Wired, top 25 in a lot of the US News and World Report rankings, etc) as the principal technology architect, and I can say that people are idiots for going nuts and patching immediately.
Our CIO, who's pretty well respected among his peers, asked us last week on deployment schedules for this. We pushed back and said, if we deploy now, we'll run into a host of issues. Over the weekend we did some cursory testing against most of our Patient care apps (a lot are web based) such as Cerner Millennium and GE's CentricityWeb. We're far ahead in the CPOE game for healthcare, so our devices are used for input of labs and orders.
Most of the biomed equipment we have doesn't run Windows. Personally, if you do your environment right, then you shouldn't have to worry about viruses and stability.
Healthcare doesn't function like the rest of the business world. It's a completely different animal.
I work with MRI scanners, so I know about these issues very well, and here's an example from my own experience:
An old colleague of mine got funding to start his own reasearch group, meaning he got his own MRI scanner. He asked me to consult on some software that would extract the data from the console of a Siemens scanner (at the time, the console was based on an OLD version SunOS, whose native compilers did not even conform to standard ANSI C) and send it directly to another computer running software that we use for data analysis. The dialect of C was a little strange, but within a week, I was able to get the software together, and my colleague was able to do the type of experiments he wanted to. And his scanner hummed along. This was back in 2001.
Fast-forward to the present. His console has since been "upgraded" to Windows XP system, and in the times I've spoken to him, he's had nothing but bad things to say about the stability of the "upgraded" system. And it's not that he had a choice, as support for his previous system was phased out. So now patients, doctors and reasearchers in his group are at the mercy of the moods of an XP system. And mind you - this system is not even on a publicly accessible network. It is on its own dedicated, private network, and its stability still can't be maintained, even by the support staff of the scanner manufacturer.
When it comes down to it, Windows still does not have the stability (never mind the security issues to cut it in really "mission-critical" situations). Maybe in cases where you need your e-commerce site up, running, and handling 1000s of transaction per second. But NOT when peoples' lives are involved.
Firewalls won't help. If it runs Windows, some idiot's going to bring in a CD full of pictures from his latest vacation and the CD's going to be infected with MyDoom or (heck, probably and...) Sobig or any number of other nasties. Or it's going to be something he wants to print on the nice laser printer at the office.... there's a hundred ways to get infected just by clueless users.
Pretty soon, the internal network's either too busy generating random traffic to do anything else-- and even if the Big Iron of the business, the dialysis machines and heart-lung devices and all those wonderful things that better damned well not break work fine, you've still got the terminal the nurse sits in front of that keeps track of when to issue you your shot that keeps you alive spending half its time rebooting because it's got Sasser.
This is not a problem a firewall can solve, and it's pretty darned big: You can't go throwing software around willy-nilly to solve this problem (even though the real problem is that the users _are_ throwing software around willy-nilly), so you can't just go "oooh! A next-day patch from Microsoft, let's hope their two hours worth of QA before it walked out the door was good enough!".
-JDF
All computer systems involved in patient care (and paper tracking as well) are forced to go through governmental processes for design, documentation and testing. These regulations add weeks, if not months, to system changes, regardless of change scope.
Case in point is the drug study setup. Setting up data entry screens and processes can take up to 6 months for a given trial, and that trial may only run 3 months for the study metrics. If any of these processes are documented incorrectly, and entire trial can be dropped and the drug denied.
This, in the hospital realm, is all about CYA. If a piece of equipment is not certified to this extent, the hospital can be held more liable for patient injuries if said equipment falters.
> Unfortunately, the stakes here could be human lives.
Soon to be made into a movie starring Uma Thurman.
It's called "Bill Kills".
assert(birth_date<time-86400)
The article mentions one thing that needs to be emphasized, which is where the FDA guy states that they're not going back to the dark ages where systems don't talk to anything else. For years, every device was on its own proprietary network (if it was on a network at all), and talked to itself and absolutely nothing else. This was bad.
In only the last couple of years (because medical IT is very behind the rest of the IT industry in a lot of ways) these devices have moved rapidly to using commodity protocols and network infrastructures, driven by hospitals' needs to do all of this more cheaply, and not have a lot of chaos.
Also, they want to provide some value add on top of the monitoring systems. For instance, it's nice to be standing by the patient's bed and see the monitoring data. It's even better to be able to export that data to another system so that it's more useful, or display it on a website so MDs can see it. All of this requires networking capability, and Microsoft (like it or not) is considered a leader in the field for server software, and has a large division providing solutions to healthcare.
Overall, the more advanced features you want a clinical system to provide, the more that system needs to integrate with other systems. Companies have given up reinventing the wheel on this every time, and are basing what they do on standard software and protocols. Microsoft is one of those. We try to avoid it whenever possible, however in most instances the decision for one product over another is based on clinical value, and not IT preference.
"Why, exactly? Because nobody would know how to hack your tiny little proprietary OS? That's crap and you know it."
The reason it the smaller the OS the less you have to test it. The whole KISS thing. Keep it simple stupid.
On a standalone ebedded system you do not need support for TrueType fonts, every printer and USB device known to man, or even video playback. On an Embeded device you often only need a few functions but those functions have to work. If you have ever programmed under windows you will find all sorts of APIs just do not work or do not work the way they are documented. Windows programers just program around these issues. You should always use the smallest OS that you can get away with for the device you are using. Linux is a good option for very flexable embedded devices. I would tend to stay clear of X and use nano-x myself.
There are many off the shelf ebeded OSs the most popular I can think of is QNX. For life critcal systems I would go for QNX over windows any day.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
This is just one of the many huge problems inside hospitals these days. Many people do not realize how often just a simple name and patient number gets assigned to the wrong person. Records get swapped with someone else or a gender or age gets changed. All these life threatening mistakes are human error. The problem is that the transcriptionists get paid per word. Not whether they word is correct and the document they transcribe is correct. It's also all about money and internal politics. They choose systems not based on whether its a good match for the hospital and the patients but based upon which board member is in bed with which company. They'll spend 10s of millions of dollars on a new system just because some higher up gets a kick back or has a golfing buddy. Then the system turns out to be total crap and they start the process all over. All the while they raise their cost of doing business and push it off to the patient.
Knowing what I know there is no way in hell I will ever go to a hospital unless I'm already dead. Cause they'll kill you just sitting in the waiting area.
All computer systems involved in patient care (and paper tracking as well) are forced to go through governmental processes for design, documentation and testing
So, if the hospital installs an uncertified piece of software on the machine, then they would be at risk if death or injury occurs, not the vendor.
If someone was injured by an unpatched machine, the hospital could pass liability back to the manufacturer - after all, they were in full compliance with the federally tested machine configuration. In which case, the manufacturer would be held liable for any injuries.
But it doesn't stop there. The manufacturer could easily and convincingly claim that Microsoft overstated the reliability of their operating systems, and the failure was due to Microsoft's code. Convincing a jury that a Windows crash caused the injury would be a trivial exercise for even the most inexperienced attorney; almost everyone has had some experience with a Blue Screen of Death.
Now comes the interesting part. Yes, the manufacturer may have agreed to the EULA, and may not be able to sue Microsoft. The patient, however, did not agree to the EULA, and having been damaged by Microsoft's code, could easily convince a jury, that in spite of the EULA, because Microsoft knew that their code was being used in medical devices failed to show due diligence to protect the user. Microsoft can't weasel their way out of this one, because the EULA doesn't apply to the patient. And, unlike the software liability cases, a medical malpractice case could easily charge the defendant with millions, or even billions of dollars in punitive damages.
The society for a thought-free internet welcomes you.
But there are a lot of applications that are not themselves critical, but could play a part. I work for a company that does materials management software for hospitals. This stuff is tweaked for efficiency, and hospitals rely on it. It runs on Windows only. Doesn't sound quite like the importance of a pacemaker, right? Well let's say the hospital gets hit by a virus. Yes, it happens, even with firewalls. Now their materials system is fubar, and they are used to it having the right supplies on hand at the right times. If it is low on something, it reorders it automatically. Now they are screwed, and they don't have something that they really need. Someone could die.
Hospitals have to operate on razor thin margins, and they can't stock millions upon millions of dollars of everything. They look to lower their on-hands inventory as much as possible.
There is all kinds of software in the hospitals that can go horribly wrong, not just the obvious stuff.
My beliefs do not require that you agree with them.
Why don't they design their software, so that it doesn't break when patches are applied?
You don't seriously believe that Microsoft gives anyone advance notice of what the patch is going to break, do you? Have you seen the ambiguous and undetailed language that goes with the WinXP SP2 patch? There's nothing actionable in there, certainly nothing testable. Until GE gets it and tests it, and authorizes it for the build, it's an astonishingly risky thing to install it.
21cfr11 mandates that only the tested configuration can be used, and if the hospital choses to violate that federal statute, they are not just at risk of screwing up their scanner, but they're technically in violation of federal statute.
I'm not defending Microsoft here, nor am I saying it's smart to have Windows in scanners, but it's there (less now than 5 years ago, but still there). The penalty for using it is that it's quite likely that some piece of malware _will_ find its way into the scanner. They're more vulnerable if they don't patch, they are going into an unsupported (and unsupportable) configuration if they do patch. The only answer is to not use Windows, but until all the 'doze-based scanners are history, they're stuck with it.
Seriously, is the REAL problem the OS? I think the REAL problem is insecure networks. Lets think for a second about all of the Windows/IE vulnerabilities in the past several months... how many of them matter if you're not connected to a network? Windows 2000/XP in my experience has been quite good, and when properly maintained (ie: no junk installed), provides a very stable platform. No one should be "surfing the web" from the deliberation machine, nor can I really see why it would need a serious network interface.... Let alone access anything on the internet! I think what hospitals REALLY need are security experts to take a good long hard look at their network and decide what SHOULD, and what SHOULDN'T be on the LAN... and if some level of network connectivity is needed (ie: the ability to monitor equipment from across the hospital), this should be on a totally separate VLAN with NO access to the internet.... Internal routing only, no exceptions. Computers connected to this LAN wouldn't have removable media bays, so the threat of worms, etc should be mitigated by general inaccessibility.
I know everyone on Slashdot would LOVE to blame the OS, but really... the fault is not with the OS as much as it is the networking admins, and even more likely, the administration for not providing the NAs with the support they need to make a properly secure network.