Slashdot Mirror
Fed-Up Hospitals Defy Windows Patching Rules
bingbong writes "According to Network World: 'Amid growing worries that Windows-based medical systems will
endanger patients if Microsoft-issued
security patches are not applied, hospitals
are rebelling against restrictions from device manufacturers that have
delayed or prevented such updates. Device makers such as GE Medical Systems,
Philips Medical Systems and Agfa say it typically takes months to test Microsoft patches because they could break the medical systems to which they're applied. In some instances, vendors won't authorize patch updates at all.' This is the typical patch vs. crash problem. Unfortunately, the stakes here could be human lives."
Why is hospital equipment running windows? Anyone that knows anything about embedded systems with high quality requirements know that you stay away from large OSes. Even Linux is avoided unless you need tcp/ip and if you don't then its better to have a small maybe even off the shelf OS. The Key is to limit the testing requirements and limit changes, which are goofy to test a life support system just to have the latest and greatest IE 6 or 7 that you shouldn't even, have hooked to a wide-open Internet anyway.
Why are they even accessible on the internet? Seems like these should be in a secure private network unlikely to be attacked.
OK.... We now have the Food and Drug Administration in charge of computer security?
Why are these things on any sort of publicly accessable network? They should, at least, be on a private network that's physically separate from everything they don't absolutely need to talk to & firewalled all to hell.
my sig's at the bottom of the page.
Take cheap shortcut, expect these kind of problems.
All these computers should be running on UNIX servers connected to dumb terminals.
Medical machines responsible for human life should never need to be patched. The software was tested at one point and should be controlled to stay at that test point until it is to be retested. For machines running windows this means they should be segregated from other parts of yoru network and should be airgap firewalled from the rest of the world. Intenet worms and email trojans shouldn't be relevant.
Wouldn't it also be alot more likely that a patch would make it through the testing phase without crashing anything important if the patch maker had access to the source code of the OS?
So...add another argument!
I say Open Source for our health.
You teach a child to read and he or her will be able to pass a literacy test. - George W. Bush
How is a firewall going to stop an insider from exploiting the network? Does working in a hospital magically transform a person into a paragon of morality?
I'm not a big fan of Microsoft, but I don't think the quality (or lack thereof) of their products is the issue here. I've read from their EULAs that their products are not suited towards critical applications (ie nuke facilities, life support). My point is that although a EULA is not a legally-binding contact, the fact that MS is stating in public Windows shouldn't be used in critical applications should tell you something. The bottom line is that if GE, Philips or Agfa build a medical system, they should be responsible for that product from the software up to the hardware. The fact that *they don't have control* over one of the components in their products (the underlying OS) is negligent, IMO.
I would get laughed out of court if I tried to blame a critical problem with a report I wrote on my secretary, and the same should happen with these companies if somebody's loved one dies from their irresponsibility.
Bill Clinton: Pimp we can believe in. - The Shirt!!!
Survery says... Beeep! Beeep! Beeep!
What "security" or other risk with a turnkey standalone system? I'd rather risk the remote chance of someone breaking into my room to run CAT-5 to my vitals monitor rather than a BSOD (possible REAL death in this case) because Service Pack x broke some obscure function and failed to alarm the nurse when my heart stopped.
Do the morons at the hospitals run Windows Update on the defibrillators?
The manufacturers have tested and retested and regression tested everything that goes into those medical devices (or they say, anyway), so why deviate from a known good combination without a compelling reason?
This comment does not necessarily represent the views and opinions of the author.
I'm sorry, but no matter what OS these devices are on, WTF are they doing on a generally available network where they can be crashed and where security updates are necessary? They should be completely isolated!
This is not so much a Windows problem as opposed to a lazy network admin's problem.
Isolate those damn machines!!! Don't have network ports just opened everywhere! Come on, this is why network admins get paid the big bucks!
MS patches before have caused considerable slowdown and possible icompatabilities before(that isn't to say they are the only ones with bad patches). If your computer slows down or has a problem, it's a minor inconvience, imagine what would happend if a life support machine went down. There is no way that MS can test for every conceivable setup, they just try to get the most general problem down and rely on others to test them on their systems. :P
The problem is using an operating system that was meant for the home/server for a much different purpose, in this case running life support machines. The things were built 8 years ago, but even then there were OSs made for embedded systems. Now there is real-time embedded linux. While I'm not going to say it's perfect, it has what is needed and nothing more
The more features you add to a system, the more places you have to exploit it. Minimalism in design is always key
Why does anyone assume that doctors, nurses, etc. are any better at securing their laptops than the rest of the public?
Pshaw, what a pant load. Here's a more rational look at this.
1: Chances are, your life won't be at stake. Any doctor or nurse worth their salt should be able to keep you alive without a computer. It's not like it's sitting in the room beside you, monitoring you. At least, not one running Microsoft
2: Any System Administrator worth his/her salt never, ever, ever puts a patch on a critical system without first testing, testing, testing on another system.
3: Also, any System Administrator with half a brain puts some type of firewall in place between the world and critical systems.
If the above three conditions are not true then the failure has occured in more important places then Microsoft or the Software Provider.
And BTW, Linux is not the solution here. Sure the vendor might be able to put together a fix faster with open source but there would still be some lag time; assuming the software vendor chose to make a fix at all and not take the same attitude they are taking with Microsoft.
It take more faith to believe in evolution than it takes to believe in God
Firewalls won't help. If it runs Windows, some idiot's going to bring in a CD full of pictures from his latest vacation and the CD's going to be infected with MyDoom or (heck, probably and...) Sobig or any number of other nasties. Or it's going to be something he wants to print on the nice laser printer at the office.... there's a hundred ways to get infected just by clueless users.
Pretty soon, the internal network's either too busy generating random traffic to do anything else-- and even if the Big Iron of the business, the dialysis machines and heart-lung devices and all those wonderful things that better damned well not break work fine, you've still got the terminal the nurse sits in front of that keeps track of when to issue you your shot that keeps you alive spending half its time rebooting because it's got Sasser.
This is not a problem a firewall can solve, and it's pretty darned big: You can't go throwing software around willy-nilly to solve this problem (even though the real problem is that the users _are_ throwing software around willy-nilly), so you can't just go "oooh! A next-day patch from Microsoft, let's hope their two hours worth of QA before it walked out the door was good enough!".
-JDF
The fact that people are installing patches on these machines against recommendations to do so scares the living shit out of me. I know that these people have good intentions but the road to hell is paved with good intentions. They don't know all of the variables. Some patch might introduce a new feature (something that does happen from time to time with MS patches) that causes the software to malfunction. This could cost lives. I really think a $50 firewall box would be a much better idea.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
I don't know how GE and Philips do their stuff, but in the systems that I work on, the computer that controls the actual X-Ray's and gantry movements don't use window's, its a custom, very stripped down version of Unix. We do use windows in several other of our devices, such as the imaging system. But if any of those systems should go down, the worse that will happen is a loss of image quality. The doctor will still have X-Ray, and Gantry movement, and the ability to remove the anything he has in the patient, or even continue the proceedure. It won't look pretty, but it will still work.
I can't imagine Philips and GE doing any differntly. None of the medical manufactures want to take a chance of putting something critical on a windows machine, and killing a patient due to a windows system crash.
"Why, exactly? Because nobody would know how to hack your tiny little proprietary OS? That's crap and you know it."
The reason it the smaller the OS the less you have to test it. The whole KISS thing. Keep it simple stupid.
On a standalone ebedded system you do not need support for TrueType fonts, every printer and USB device known to man, or even video playback. On an Embeded device you often only need a few functions but those functions have to work. If you have ever programmed under windows you will find all sorts of APIs just do not work or do not work the way they are documented. Windows programers just program around these issues. You should always use the smallest OS that you can get away with for the device you are using. Linux is a good option for very flexable embedded devices. I would tend to stay clear of X and use nano-x myself.
There are many off the shelf ebeded OSs the most popular I can think of is QNX. For life critcal systems I would go for QNX over windows any day.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
on life-safety equipment, why in hell is ANY outside operating system in use??? you CAN control bugs in your own code if it is YOUR OWN code. get back to machine language FSMs for the specific purpose on a piece of hardware like a monitor. it is irresponsible in the extreme to rely on somebody else's box 'o' bugs as part of your life-safety system. period. anything in that realm that needs wide access should have an outboard trusted "my code only, dammit" interface that the wild wild web plugs into.
basically, it's just pseudocode that anybody is writing any more, anyway. flip it through a different compiler, a cheaper machine language compiler, and debug with a logic analyzer if you have to. this is what the better high school kids were doing in the late 60s and early 70s, anyway, kids like wozniak and gates and kildall. wasn't any rougher for me to debug in the late 70s and early 80s than anything else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
They are.
The ultrasound machine that they use on you isn't running windows.
The computer hooked up to it, which handles the image analysis, display, and archiving, however, probably is.
Vintage computer games and RPG books available. Email me if you're interested.
First I didn't read the article. I have worked in a hospital for over 10 years. From personal experience I can say a hospital can provide some of the most interesting computer setups that you will find. And all of them are considered critical. I don't know if they are referring to servers running Windows or to actual medical devices running it, but I can say that they do exist.
Down time in a hospital is extremely hard to come by, many systems are used by many departments and no one wants to be down for an hour for patches. Microsoft really isn't the problem here, though it would be nice to blame them. Most hospitals run the gamut of OS platforms, from AIX, Linux, Windows 95/98/2000/XP (yes we still have 95 in use, and some medical devices actually run 98, scary huh.), Apple OS 9/X, SCO Unix, that's all I can think of at the moment, but I'm sure there are more that I don't know about. All of the release patches. We have servers on site that we pay for that we are not allowed to do anything with, we don't even know the passwords. Sometimes that's fine with us because we were never given instructions on how to fix their problems, so better to just bug their support than us. Other systems we have some control of, but the way they were certified with the FDA we can't do anything with the system. In fact, just a few months ago I helped setup a system for our Labor and Delivery department to help with fetal monitors. This system seemed like it will do everything they need, however it is almost completely separated from our network, with the exception of an ADT feed. We are not even allowed to turn on automatically adjust for daylight savings, because that wasn't how the system was certified. Will this system ever get patches, not by us, and I doubt by the vendor. They had separate network drops installed from our network and that's the way it's going to be. Not only that but part of their backup process actually involves a floppy diskette.
Couple the FDA issues, with nobody wanting to spend money (for network equipment) and nobody here to do the work and you have a prime problem for a disaster. Viruses are a huge issue in a hospital a virus can take down many systems with no problem, you might say it needs to be more secure well tell that to the companies that require open shares for their product to work. Viruses are also a problem in hospitals when you consider the computer experience of many nurses and doctors. Some don't understand that an email can show up from someone they know and not really be sent by that person, so they trust the source and then we have an infection. Our POP3 server checks for new dat files every hour and still by the time we get the latest dat files the viruses have already been received by people. There is no way to win that short of time delaying email by like a day and that wouldn't go over well.
I've gone on too long, now most of these problems won't directly affect your patient care, aside from maybe slowing it down a bit. It can cause problems if you frequent that facility and they have previous studies and results to look at but suddenly they don't have access to them. Or that could even be the case in the current visit. One good thing with all these systems though is that they are redundant at times, so your allergies for example may be in 3 or 4 different systems, so if one is down they should still be able to find it in one of the other systems.
Also, don't forget that hospitals haven't quite made it to that paperless Nirvana.
AC signing out.
This is just one of the many huge problems inside hospitals these days. Many people do not realize how often just a simple name and patient number gets assigned to the wrong person. Records get swapped with someone else or a gender or age gets changed. All these life threatening mistakes are human error. The problem is that the transcriptionists get paid per word. Not whether they word is correct and the document they transcribe is correct. It's also all about money and internal politics. They choose systems not based on whether its a good match for the hospital and the patients but based upon which board member is in bed with which company. They'll spend 10s of millions of dollars on a new system just because some higher up gets a kick back or has a golfing buddy. Then the system turns out to be total crap and they start the process all over. All the while they raise their cost of doing business and push it off to the patient.
Knowing what I know there is no way in hell I will ever go to a hospital unless I'm already dead. Cause they'll kill you just sitting in the waiting area.
The problem is that staff need connectivity to application servers, and the same staff need access to a ton of other servers, including outside governmental services on the Internet. You can't segregate the "critical" servers from the user's PCs very easily, so the "critical" servers are usually one hop away from the Internet, via the users' PCs. In any case, the managers making decisions where I've been can't make the case for putting the users through the increased difficulty of doing things securely.
:^)
Another thing is that we're under huge pressure to give physicians and radiologists access to data via the web. This could help save lives, if a patient's physician can look at their ultrasound, etc from his hotel while he's on vacation, etc, but the price you pay (which never counts for much with our managemnet) is decreased security. I am in this situation with some SW vendors who refuse to support a system if we let Windows Update automatically patch their system. They're afraid that they'll waste some support time on a problem related to a M$ patch breaking the OS or something their code depends on. I'm tired of seeing services killed and machines hung by what appear to be patchable exploits, so I'm doing it anyway. By doing this, you're giving the vendor a "get out of supporting their own app for free" card.
A final perspective is the class war between technical folks and the suits, who in my health care career have been non-technical folks who don't really like or understand technology, just data and applications, and in my current case, who seem to have a psychological/emotional problems with technical people in general.
When a clinical staff member here asks for some new functionality, or complains about having to change their password, management always comes down on their side, security be damned, because the implication is that if we require clinical folks to do _any_ extra work, or don't give them some new one-click, time-saving feature, we are impairing their ability to care for patients. It's the same way with supporting applications or hardware after hours, if a printer's jammed, it's perceived as being equivalent to a patient bleeding to death. Oh my god, it's "affecting patient care"! That's one of the reasons management doesn't want to tell a clinical user "no" Any time we say "no" we're perceived as being a problem. Those types of users can't see far enough don into the technical aspects fo things to understand the threats, just that they have to remember another password, or click another button.
Enough of this ranting. I'm getting disgusted with the whole thing all over again!
If you can't tell yet, I've had enough of being a technical proletariat. I'm sick and tired of dealing with Microsoft OS's and applications, and since there's not much else IT work in our area, I'm starting a new career in teaching with taking a 40% pay cut to teach at a local university.
By this weekedn, this will no longer be my problem
But there are a lot of applications that are not themselves critical, but could play a part. I work for a company that does materials management software for hospitals. This stuff is tweaked for efficiency, and hospitals rely on it. It runs on Windows only. Doesn't sound quite like the importance of a pacemaker, right? Well let's say the hospital gets hit by a virus. Yes, it happens, even with firewalls. Now their materials system is fubar, and they are used to it having the right supplies on hand at the right times. If it is low on something, it reorders it automatically. Now they are screwed, and they don't have something that they really need. Someone could die.
Hospitals have to operate on razor thin margins, and they can't stock millions upon millions of dollars of everything. They look to lower their on-hands inventory as much as possible.
There is all kinds of software in the hospitals that can go horribly wrong, not just the obvious stuff.
My beliefs do not require that you agree with them.
Configuration Management means:
- controlling the Configuration of equipment, in order to ensure consistent behavior.
Unfortunately, Configuration Management often does not take into account the fact that when you put a system on a network, it becomes part of a larger system, and unless you manage the entire network of systems, then you cannot really control your conditions, nor can you ensure consistent behavior.
This needs to be taken into account as a basic "sky is blue" assumption of Configuration Management.
Sadly, it is not.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
This has been a real problem for a very long time in many industrial applications. And it is not limited to the OS but the box as well.
The temptation is way to great for the bean counters and greedy sales typs to switch the robust hardware and OS for the commodity type and save a bundle up front.
Consider a $500 PC and an $2500 industrial PC. If you let the bean counter do the math he will tell you about the 3ghz P4, GeForce 4 100 gig hdd v. the P3 20 gig with an average video card.
Then you explain that the OS's have the same disparity in cost and he starts to get confused
I have said many times before that we have Windows not because it was best but because it was cheapest. Same with the clone PC. MS got to be the default OS because it was generaly 50% of what the other OS's were.
Now when it comes to saving lives the cost should not matter, however, it is still a business. And there are still bean counters and greedy sales people who get to make some very powerful decisions.
Preface: this is NOT a Microsoft/windows bash..
Why in the world are they using a desktop operating system of any kind on medical equipment?
I wouldn't care how stable it was, that doesn't belong in that market.. Embedded systems that are dedicated to the need are what should be used...
---- Booth was a patriot ----
Over the last 10 years, everyone's become accustomed to Windows. Everyone has Windows. Once everyone got Windows, they wouldn't use anything that didn't work on Windows. So, vendors began migrating everything to Windows. (I used to work for a software company and now work at a hospital). So now, all the vendor's software runs on Windows, and probably runs just fine... provided the Windows version remains the same as the one it was tested on, no patches are applied, and no other apps are installed onto the same machine. But, users are used to running everything they want on Windows. That, after all, is the point of Windows. Plus, Windows is way cheaper than other options. Not to mention training. So, we're stuck with Windows apps, and there's really no cheaper alternative out there. This would be fine and dandy, if the only problems with Windows were worms and viruses. But no, like regular windows, Windows breaks really, really easitly.
Even the few vendors I've seen who have balls enough to release a Linux version of their software are tied to specific distributions, specific kernels, etc.
"Would it kill you to put down the toilet seat?" -- Maya Angelou
The real problem is not all about patching. Many of these medical devices that rely on Windows are running on default installs. It is nearly impossible to keep a machine with a default install of Windows from getting a worm or virus when attached to a large enterprise network. Worms travel too quickly. Vendors and IT shops are blindly applying patches without testing them.
If the folks building these machines would take the time to turn off unneccessary services, and do some basic hardening (there are several excellent hardening guidelines for Windows avaialble from SANS, NIST, and other places) many of the worms would not be as big a problem. Couple this with some firewalling, IDS, and logical network segregation (as mentioned in the article) and the patches become less relevant.
I work at a hospital and am working with teams developing FDA-compliant medical device software (much to my chagrin they are using Windows). The server build they have developed has been deployed in "the wild" for a couple of years without MS patches and without infection. Why? because they are only listening on one port and have taken the time to disable a bunch of unneccessary stuff.
We need to change the way we look at security flaws and build the machines right in the first place. We can't rely on patches as the sole means of securing systems from every worm that comes along -- especially not when the systems are providing medical care!
Seriously, is the REAL problem the OS? I think the REAL problem is insecure networks. Lets think for a second about all of the Windows/IE vulnerabilities in the past several months... how many of them matter if you're not connected to a network? Windows 2000/XP in my experience has been quite good, and when properly maintained (ie: no junk installed), provides a very stable platform. No one should be "surfing the web" from the deliberation machine, nor can I really see why it would need a serious network interface.... Let alone access anything on the internet! I think what hospitals REALLY need are security experts to take a good long hard look at their network and decide what SHOULD, and what SHOULDN'T be on the LAN... and if some level of network connectivity is needed (ie: the ability to monitor equipment from across the hospital), this should be on a totally separate VLAN with NO access to the internet.... Internal routing only, no exceptions. Computers connected to this LAN wouldn't have removable media bays, so the threat of worms, etc should be mitigated by general inaccessibility.
I know everyone on Slashdot would LOVE to blame the OS, but really... the fault is not with the OS as much as it is the networking admins, and even more likely, the administration for not providing the NAs with the support they need to make a properly secure network.