Slashdot Mirror
Medical Equipment Crashes During Heart Procedure Because Of Antivirus Scan (softpedia.com)
An anonymous reader quotes a report from Softpedia: The device in question is Merge Hemo, a complex medical equipment used to supervise heart catheterization procedures, during which doctors insert a catheter inside blood veins and arteries in order to diagnose various types of heart diseases. According to one such report filed by Merge Healthcare in February, Merge Hemo suffered a mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer. Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors' computer. The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly.
Our antivirus is completely up to da
Upgrading to Windows 10......
SJW's don't eliminate discrimination. They just expropriate it for themselves.
Picking an OS that clear says not use it for real time possible life endangering task is a huge mistake!! QNX, RT_Linux, and more!!! Hello!!!
Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect. The product security recommendations, (b)(4), explicitly state, "the intent of these guidelines is to configure the anti-virus software so that it does not affect clinical performance and uptime while still being effective. To accomplish this, the anti-virus software needs to be configured to scan only the potentially vulnerable files on the system, while skipping the medical images and patient data files. Our experience has shown that improper configuration of anti-virus software can have adverse affects including downtime and clinically unusable performance. ".
Use some dedicated hardware with a custom software system with only components designed for the purpose of the machine and nothing else. Harden and sanity check the hell out of the I/O and connect THAT to your idiot box.
Twinstiq, game news
This is interesting; the configuration on a device like this should be highly controlled. I have no experience with medical devices, but I know that process control equipment generally has vendor approved configuration (and often they only certify one AV vendor so even if our corporate contract is with vendor A, we have to use vendor B for the process control stuff because that is what is certified by the control system vendor. They also have very specific settings you have to use. Failure to follow the settings could result in lack of process control at a critical time. It seems medical stuff must be under similar (if not even more restrictive) configuration control. Having AV do a "scan" every hour is very stupid since any competent AV is doing on-access scanning anyway. I would expect the vendor for the software has specified folders / files / etc. that must be exempted from the scan as well (vendors for process stuff such as Yokogawa, etc. specify that). Seems to be a configuration failure on the part of the facility.
For what? This was an antivirus scan and the report itself doesn't mention an OS. Furthermore, this crash brought down the whole system. If developers are writing their software to utilize drivers, they ought to make sure those drivers aren't so buggy that the mere stopping of data will tank the entire system...especially a system that should be as close to "bulletproof" as bulletproof can be in the technological sense of the word.
It just writes itself.
Antivirus systems aren't useless(I wouldn't trust their 'disinfection'; but they at least catch people reusing obsolete exploits and sometimes provide warnings that something is amiss); but this is one of those situations where hearing that antivirus software is running is a giant red flag: it usually means that a full-fat desktop/server OS with a network connection and who-knows-what-else running on it is doing the job of a dedicated computer. Quite probably being allowed to retain state over time except for the ever so occasional re-imaging. That just isn't going to go well. Even if your application needs full Windows whatever for some reason, there are plenty of ways to keep it on a much tighter leash than just shoving a desktop at the problem and hoping Norton can save you. If a system is contained by the network so that it can only talk to the external hosts it absolutely needs; and is booting from a clean, static, image every time(with all changes discarded after any data generated during the session are moved elsewhere) you are a great deal safer.
Why would anyone use Windows for a real-time critical application? There are small real-time OS's designed just for such applications.
[Insert pithy quote here]
For what? This was an antivirus scan and the report itself doesn't mention an OS. Furthermore, this crash brought down the whole system. If developers are writing their software to utilize drivers, they ought to make sure those drivers aren't so buggy that the mere stopping of data will tank the entire system...especially a system that should be as close to "bulletproof" as bulletproof can be in the technological sense of the word.
Windows can never fail - only we can fail Windows.
And Bulletproof and Windows never belong in the same sentence.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
"Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors' computer. "
I seriously doubt the computer was owned by the doctor. More than likely, it was procured, set up and managed by a team of IT specialist at the hospital/clinic who know little to nothing about the software that might be running on it. Likewise, if the company supplying the software isn't providing a dedicated, hardened box to run the software on, they share the blame as well. Or, I have seen dedicated boxes with all kinds of crap loaded on them by operators who had no clue what the consequences might be. The bottom line here is that maybe computers should be kept out of the operating room. Or maybe doctors shouldn't be allowed to use them.
There is no need to mention an OS - the only system that such problems with viruses is Windows, and the only OS that embeds a virus scan in the kernel IS windows. No other OS locks data like that.
"as close to "bulletproof" as bulletproof can be"
Certainly leaves out using Windows.
. ,even though it wasn't?), and that being able to use their knowledge of Windows is a benefit that will make their system better.
It is easy to fall for the siren-song hype from the marketeers that the general purpose operating system is up to the task (remember Microsoft's marketing push that Windows CE was a real-time operating system
Whether it is a weather application being used on live television, or a computer being used in an operating room, Microsoft has shown that Windows is not a proper steward of serious systems programming.
Why can't we use bulletproof and Windows in the same sentence? According to the report it was the AV scanner that caused the application to crash. The PC was then required to be rebooted for the application to start working correctly. Arguably the client software is at fault for not being able to recover from a situation where "communications" get lost. In this case, it didn't sound like the Windows system had any issue. Furthermore, I have experienced many Windows servers who are happy to sit in a corner and chug away for years without issues. Does Windows have its flaws? Sure, but so does any other operating system - and in general I don't find Windows to be so unstable these days. It's usually 3rd party software, written to use higher level privileges than it really needs, to take down Windows. But any poorly written, high privilege software can take down any OS.
I used to work for a company that built ophthalmic ultrasound machines. It was Windows based (unfortunately). IT departments, being who they are, wanted to put things like antivirus on it. Then the doctors would complain that the MEDICAL INSTRUMENT wasn't performing as advertised. They send it in to us for 'repair'. We remove the shitty antivirus (and all the other crap that IT guys would install on it), then it works perfectly again. We return it.. and IT guys would screw it up again. Rinse, repeat, ad infinitum.
MEMO TO IT GUYS: Stop treating medical instruments like they're desktop computers! Find another solution, or AT LEAST be smart about how you're installing your junk on it, IT IS A MEDICAL INSTRUMENT, DAMNIT!
Are YOU using the TOOL, or is the TOOL using YOU? Think about it!
IIRC the EULA of every Windows version so far said that the OS must not be used in life-or-death critical operations.
Not that it isn't used in, say, nuclear plants (which are explicitly cited in the EULA, btw), but if you use something that is clearly not good enough for the job, and even tells you that it's too crappy for important tasks, well, you can't really complain, can you?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
This is what I don't get. Why the hell is AV software running on a realtime apparatus?
1: If AV software is needed for legal eagle reasons, code a scanner for embedded use that runs -only- when the machine is offline and not doing anything. When the switch to online it is flipped, any scans and such get stopped immediately.
2: A medical machine should be air-gapped anyway, with firmware updates done via files on a signed SD card. There should never be a vector for introducing malware onto a machine without physical access.
3: Have the designers even done testing where the AV software (or even worse, GWX) fires up during a procedure? This is basic Q&A here, and for the astronomical cost of medical equipment, should be assumed that this was done.
From TFA, I'd lay the blame of this at the feet of the device maker. They need to use a real OS, or at least ensure that there is no state their environment can get into that can cause this.
Why can't we use bulletproof and Windows in the same sentence? According to the report it was the AV scanner that caused the application to crash. The PC was then required to be rebooted for the application to start working correctly. Arguably the client software is at fault for not being able to recover from a situation where "communications" get lost.
It is not reasonable to single out the OS, the AV software, or the application. The three were combined, along with some specialized hardware as a system with an arguably life-or-death role in the OR. This was a bad choice, for all the reasons previously stated. If you're going to place the system in a role as critical as heart surgery, far more serious attention should have been paid to it's availability and reliability. Yes, the AV scan disrupted things, the OS had no way to know that the application software is as important as it was, the application software failed spectacularly when the expected resources weren't available, etc. The big fail was the decision to place a patient's life in the hands of that rickety and untested system.
If this machine running an AV is so intimately tied to a medical procedure having to do with the human heart, I hope to God its not also on the internet and is a standalone machine. As a standalone machine, I don't see any reason for any AV beyond perhaps a scanner if/when a USB drive is inserted. And since the machine in question has an AV that so spectacularly crashed, I'm gonna go out on a limb and speculate it was running Windows.. (shudder)
THANK YOU, Edward Snowden!! Americans owe you a debt of gratitude (whether they know it or not..)
This is what I don't get. Why the hell is AV software running on a realtime apparatus?
1: If AV software is needed for legal eagle reasons, code a scanner for embedded use that runs -only- when the machine is offline and not doing anything. When the switch to online it is flipped, any scans and such get stopped immediately.
2: A medical machine should be air-gapped anyway, with firmware updates done via files on a signed SD card. There should never be a vector for introducing malware onto a machine without physical access.
3: Have the designers even done testing where the AV software (or even worse, GWX) fires up during a procedure? This is basic Q&A here, and for the astronomical cost of medical equipment, should be assumed that this was done.
From TFA, I'd lay the blame of this at the feet of the device maker. They need to use a real OS, or at least ensure that there is no state their environment can get into that can cause this.
The AV software wasn't running on the medical device, it was running on the Doctor's computer. The Doctor's computer has a software app that gathers data from the medical device and, it seems, that there is some requirement for the medical device to be able to read this data as well. Or perhaps the App has some command and control functions. Either way, the AV software ran, freezing up the app on the doctors computer and causing the medical device to crash.
In my opinion, the hospital should have an air-gapped dedicated system for this instead of relying on the doctor's laptop.
From the connected article, the antivirus software on the doctors' PC was configured to run a scan hourly, and when it was scanning the application's folders, it froze access to the files in those folders. The application was designed to require real-time access to its data, and failed spectacularly when it was blocked, crashing the computer. Fortunately, the situation was not time-critical, and the doctors were able to take the time to reboot the computer and restart the application without endangering the patient. However, a future interaction of this type may not end so benignly.
Ignoring the fact that the application was badly designed so that it didn't fail gracefully, Merge Healthcare's documentation for their product explains that the application requires real-time access to its data and recommends white-listing its folders to prevent an antivirus scanner from locking off access to the data. So the blame can be laid both at the feet of Merge Healthcare for building software that didn't fail gracefully and at the feet of the hospital for improperly configuring the virus scanner to prevent it from interfering with a real-time application.
Not that it isn't used in, say, nuclear plants
It's not. There's not an industrial control system in existence nuclear or otherwise which runs it's control routines on a windows platform. They run on proprietary code embedded in control processors which happen to take input from a piece of software over the network which may be based on Windows. Should that piece of software (or the underlying OS) go down, nothing at all happens and the controllers happily keep controlling.
Windows is nothing more than a TV remote control in this case. The TV doesn't magically change channel or turn off when the batteries in the remote die.
Having a Windows based medical system is stupidity in itself. Even having an antivirus scan in embedded software is ridiculous, they should be stand alone devices, not dependent upon some apathetic home consumer company, not connected to the internet, etc. And yet so many developers are so amazingly uneducated and inexperienced that they think Windows is the perfect solution to everything, managers love Windows because they can hire so many cheap ass developers for it and mistakenly think they can save time this way. Maybe Windows is not the flaw, but the Windows mindset certainly is.