Medical Equipment Crashes During Heart Procedure Because Of Antivirus Scan (softpedia.com)

An anonymous reader quotes a report from Softpedia: The device in question is Merge Hemo, a complex medical equipment used to supervise heart catheterization procedures, during which doctors insert a catheter inside blood veins and arteries in order to diagnose various types of heart diseases. According to one such report filed by Merge Healthcare in February, Merge Hemo suffered a mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer. Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors' computer. The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly.

Comments

Windows 10 update will kill human beings

[Anonymous Coward]

There is blood on your hands nadella.

Re:Windows 10 update will kill human beings

[mlw4428]

For what? This was an antivirus scan and the report itself doesn't mention an OS. Furthermore, this crash brought down the whole system. If developers are writing their software to utilize drivers, they ought to make sure those drivers aren't so buggy that the mere stopping of data will tank the entire system...especially a system that should be as close to "bulletproof" as bulletproof can be in the technological sense of the word.

Re:Windows 10 update will kill human beings

[Ol Olsoc]

For what? This was an antivirus scan and the report itself doesn't mention an OS. Furthermore, this crash brought down the whole system. If developers are writing their software to utilize drivers, they ought to make sure those drivers aren't so buggy that the mere stopping of data will tank the entire system...especially a system that should be as close to "bulletproof" as bulletproof can be in the technological sense of the word.

Windows can never fail - only we can fail Windows.

And Bulletproof and Windows never belong in the same sentence.

Re: Windows 10 update will kill human beings

[Anonymous Coward]

What about the sentence you typed?

Re:Windows 10 update will kill human beings

[Anonymous Coward]

There is no need to mention an OS - the only system that such problems with viruses is Windows, and the only OS that embeds a virus scan in the kernel IS windows. No other OS locks data like that.

"as close to "bulletproof" as bulletproof can be"

Certainly leaves out using Windows.

Re:Windows 10 update will kill human beings

[Anonymous Coward]

It's a good thing for you that your skill at giving blowjobs is much better than your skill at trolling.

Re:Windows 10 update will kill human beings

[mlw4428]

Why can't we use bulletproof and Windows in the same sentence? According to the report it was the AV scanner that caused the application to crash. The PC was then required to be rebooted for the application to start working correctly. Arguably the client software is at fault for not being able to recover from a situation where "communications" get lost. In this case, it didn't sound like the Windows system had any issue. Furthermore, I have experienced many Windows servers who are happy to sit in a corner and chug away for years without issues. Does Windows have its flaws? Sure, but so does any other operating system - and in general I don't find Windows to be so unstable these days. It's usually 3rd party software, written to use higher level privileges than it really needs, to take down Windows. But any poorly written, high privilege software can take down any OS.

Re:Windows 10 update will kill human beings

[Anonymous Coward]

Doesn't mention an OS but said OS needs hourly virus scans. It's Windows.

Re:Windows 10 update will kill human beings

[Rob Y.]

Antivirus software itself can be the weak link in Windows. I had Avast AV seemingly freeze Windows 7 when I tried to launch my own app on it - even after just building the app with Visual Studio on that box. By 'freeze', I mean not only refuse to run the app, but do it without popping up any notification, and without failing in a way that Windows Explorer can recover from. I would end up with multiple processes in Task Manager that could not be killed from there, and the entire Windows launcher frozen.

The solution was simple enough - whitelist the directory tree containing my own stuff. It'd have been a lot simpler had Avast notified me what it was doing... But the fact that Avast would - or could - do this silently, and seem to crash the system is beyond bizarre. Whether that's partially Windows' fault - or whether Avast is just that shitty, it sure doesn't end up making Windows look good. Not to mention the fact that AV software is 'necessary' in the first place on a dedicated medical instrument.

Re:Windows 10 update will kill human beings

[Opportunist]

IIRC the EULA of every Windows version so far said that the OS must not be used in life-or-death critical operations.

Not that it isn't used in, say, nuclear plants (which are explicitly cited in the EULA, btw), but if you use something that is clearly not good enough for the job, and even tells you that it's too crappy for important tasks, well, you can't really complain, can you?

Re:Windows 10 update will kill human beings

[Anonymous Coward]

This is what I don't get. Why the hell is AV software running on a realtime apparatus?

1: If AV software is needed for legal eagle reasons, code a scanner for embedded use that runs -only- when the machine is offline and not doing anything. When the switch to online it is flipped, any scans and such get stopped immediately.

2: A medical machine should be air-gapped anyway, with firmware updates done via files on a signed SD card. There should never be a vector for introducing malware onto a machine without physical access.

3: Have the designers even done testing where the AV software (or even worse, GWX) fires up during a procedure? This is basic Q&A here, and for the astronomical cost of medical equipment, should be assumed that this was done.

From TFA, I'd lay the blame of this at the feet of the device maker. They need to use a real OS, or at least ensure that there is no state their environment can get into that can cause this.

Re:Windows 10 update will kill human beings

[Registered Coward v2]

For what? This was an antivirus scan and the report itself doesn't mention an OS. Furthermore, this crash brought down the whole system. If developers are writing their software to utilize drivers, they ought to make sure those drivers aren't so buggy that the mere stopping of data will tank the entire system...especially a system that should be as close to "bulletproof" as bulletproof can be in the technological sense of the word.

Alternatively, only certify the software to run on hardware they provide and configure; since there is no way that can anticipate what else will be running on a non-certified machine. Even with a certified machine, unless you make it impossible to load anything else someone will find a way to load a program that crashes yours. I had that happen on a server we installed to run a specific program; despite clear warnings not to install any software on it someone did and crashed our program. their excuse, "We saw it had some HD space so we decided to install it on this machine;" proving nothing is fool proof because even fools can be ingenious.

Re: Windows 10 update will kill human beings

[Ol Olsoc]

What about the sentence you typed?

Pork Rinds? That or a Chewbacca coffee mug!

Re:Windows 10 update will kill human beings

[Ol Olsoc]

Why can't we use bulletproof and Windows in the same sentence? According to the report it was the AV scanner that caused the application to crash. The PC was then required to be rebooted for the application to start working correctly.

It's time to throw all of the software vendors under the bus now.

Seriously, without trying to be a smartass, whether the operating system is at fault, or the people who write software for it are at fault, it makes for a whole system that simply is going to fail.

Because people have been sold on the concept that everything has to be on the internet. So the system better have AV software on it. So we end up with little "Oopsies" that may occur at any time.

And because it's a whole lot easier to figure out the inevitable failures after they happen than to test every possible situation before they do.

Re:Windows 10 update will kill human beings

[Joe_Dragon]

What about systemd?

Re: Windows 10 update will kill human beings

[Joe_Dragon]

Chewbacca defense will be used to get out of having to pay out in any lawsuit from this.

Re:Windows 10 update will kill human beings

[Jawnn]

Why can't we use bulletproof and Windows in the same sentence? According to the report it was the AV scanner that caused the application to crash. The PC was then required to be rebooted for the application to start working correctly. Arguably the client software is at fault for not being able to recover from a situation where "communications" get lost.

It is not reasonable to single out the OS, the AV software, or the application. The three were combined, along with some specialized hardware as a system with an arguably life-or-death role in the OR. This was a bad choice, for all the reasons previously stated. If you're going to place the system in a role as critical as heart surgery, far more serious attention should have been paid to it's availability and reliability. Yes, the AV scan disrupted things, the OS had no way to know that the application software is as important as it was, the application software failed spectacularly when the expected resources weren't available, etc. The big fail was the decision to place a patient's life in the hands of that rickety and untested system.

Re:Windows 10 update will kill human beings

[Joe_Dragon]

Windows and read only file systems do not work that well also some apps need to be able to wire files when running and can crash / error out if they can not.

That are things like deep freeze that kind of have read only but that does not work in all settings and needs more config / work to get app / windows updates right.

Re:Windows 10 update will kill human beings

[Ol Olsoc]

What about systemd?

Or Pikachu.

Re: Windows 10 update will kill human beings

[Ol Olsoc]

Chewbacca defense will be used to get out of having to pay out in any lawsuit from this.

You are pretty good at this stuff!

Re:Windows 10 update will kill human beings

[LVSlushdat]

If this machine running an AV is so intimately tied to a medical procedure having to do with the human heart, I hope to God its not also on the internet and is a standalone machine. As a standalone machine, I don't see any reason for any AV beyond perhaps a scanner if/when a USB drive is inserted. And since the machine in question has an AV that so spectacularly crashed, I'm gonna go out on a limb and speculate it was running Windows.. (shudder)

Re:Windows 10 update will kill human beings

[David_Hart]

This is what I don't get. Why the hell is AV software running on a realtime apparatus?

1: If AV software is needed for legal eagle reasons, code a scanner for embedded use that runs -only- when the machine is offline and not doing anything. When the switch to online it is flipped, any scans and such get stopped immediately.

2: A medical machine should be air-gapped anyway, with firmware updates done via files on a signed SD card. There should never be a vector for introducing malware onto a machine without physical access.

3: Have the designers even done testing where the AV software (or even worse, GWX) fires up during a procedure? This is basic Q&A here, and for the astronomical cost of medical equipment, should be assumed that this was done.

From TFA, I'd lay the blame of this at the feet of the device maker. They need to use a real OS, or at least ensure that there is no state their environment can get into that can cause this.

The AV software wasn't running on the medical device, it was running on the Doctor's computer. The Doctor's computer has a software app that gathers data from the medical device and, it seems, that there is some requirement for the medical device to be able to read this data as well. Or perhaps the App has some command and control functions. Either way, the AV software ran, freezing up the app on the doctors computer and causing the medical device to crash.

In my opinion, the hospital should have an air-gapped dedicated system for this instead of relying on the doctor's laptop.

Re:Windows 10 update will kill human beings

[srmalloy]

From the connected article, the antivirus software on the doctors' PC was configured to run a scan hourly, and when it was scanning the application's folders, it froze access to the files in those folders. The application was designed to require real-time access to its data, and failed spectacularly when it was blocked, crashing the computer. Fortunately, the situation was not time-critical, and the doctors were able to take the time to reboot the computer and restart the application without endangering the patient. However, a future interaction of this type may not end so benignly.

Ignoring the fact that the application was badly designed so that it didn't fail gracefully, Merge Healthcare's documentation for their product explains that the application requires real-time access to its data and recommends white-listing its folders to prevent an antivirus scanner from locking off access to the data. So the blame can be laid both at the feet of Merge Healthcare for building software that didn't fail gracefully and at the feet of the hospital for improperly configuring the virus scanner to prevent it from interfering with a real-time application.

Re:Windows 10 update will kill human beings

[Salgak1]

Actually, FDA regs on most medical devices generally require re-certification after a software update.

And re-certifications are expensive. So many companies simply don't update or patch until a new version of their plastic fantastic gear ships.

You would **think** that medical types would understand the value of prophylaxis against infection . .

Re:Windows 10 update will kill human beings

[kheldan]

As I have said in my own comment: This is a MEDICAL INSTRUMENT, not a general-purpose computer, and as such there should not be any non-approved-by-the-manufacturer software installed on it, EVER. I worked for a company that built a medical instrument and we'd have this sort of problem all the time, and it would take some pretty hard pounding with rather large mallets to get it through the thick skulls of some IT departments that they can't just install whatever the hell they want on our medical instrument and expect it to function correctly. Eventually we had to have our software engineers vet and approve a specific antivirus for use, and threaten them with voided warranties if they installed anything else.

That wasn't the end of it, though, either. They'd install all sorts of remote access crap on it, screencap crap, and other nonsense that had nothing to do with it being a medical instrument, then they'd complain about how it's not working correctly as a medical instrument. We'd get it back, uninstall all their extraneous nonsense, then magically it starts working correctly again. Sadly, though: Rinse, repeat several times, before they get the clue that they can't just treat it like a desktop computer!

Then there's the doctors who can't be bothered to learn how the gods-be-damned thing works before going to use it. I swear, if I'd've gotten even one more support call from an operating room with a patient on the table, and the sound of a heart monitor beeping in the background, I was going to lose it. I kid you not on this!

Re:Windows 10 update will kill human beings

[thegarbz]

And Bulletproof and Windows never belong in the same sentence.

What OS does bullet proof belong in a sentence with? Name an OS that can prevent a client application from crashing due to being poorly coded?
Windows doesn't crash itself. At least I haven't heard of a case of windows spontaneously stopping for shits and giggles since the 49.7 day bug of Windows 95/98. Windows is only typically brought down by poorly coded drivers, poorly coded software, or failing hardware, all of which should be under tight control on medical equipment anyway.

Re:Windows 10 update will kill human beings

[thegarbz]

Not that it isn't used in, say, nuclear plants

It's not. There's not an industrial control system in existence nuclear or otherwise which runs it's control routines on a windows platform. They run on proprietary code embedded in control processors which happen to take input from a piece of software over the network which may be based on Windows. Should that piece of software (or the underlying OS) go down, nothing at all happens and the controllers happily keep controlling.

Windows is nothing more than a TV remote control in this case. The TV doesn't magically change channel or turn off when the batteries in the remote die.

Re:Windows 10 update will kill human beings

[Opportunist]

No, but it sure does change the channel when the cat steps on it, doesn't it? The TV obeys the commands of the remote even if the remote for some odd reason should short out and send a false command.

Try a better analogy.

Re:Windows 10 update will kill human beings

[Ol Olsoc]

And Bulletproof and Windows never belong in the same sentence.

What OS does bullet proof belong in a sentence with? Name an OS that can prevent a client application from crashing due to being poorly coded? Windows doesn't crash itself. At least I haven't heard of a case of windows spontaneously stopping for shits and giggles since the 49.7 day bug of Windows 95/98. Windows is only typically brought down by poorly coded drivers, poorly coded software, or failing hardware, all of which should be under tight control on medical equipment anyway.

Oh please, I get Windows systems to fix after every update. It doesn't take a BSOD to render an application unusable.

If you aren't the absolute proof of the ultimate Windows shill, there is none. Jesus dude, Let us just hope that you never go for a medical procedure, the Windows based computer fucks up, you suffer and die.

With your last breath, and on your tombstone, you'll want everyone to know "This was NOT Window's fault!!

For you see, my dear chachalaca, you go apeshit nuts, fearing that I disparage your precious Operating system, when I'm saying that not one, not any, not in a million years, should a life critical ever be connected in any way, any form, any remote chance of connecting to the internet.

Windows or otherwise. And as Windows based systems are compromised in hospitals all over the country, rest easy that it is never their fault. Perhaps its time that Hospitals shift to Operating systems that accept some accountability.

Re:Windows 10 update will kill human beings

[Darinbob]

Having a Windows based medical system is stupidity in itself. Even having an antivirus scan in embedded software is ridiculous, they should be stand alone devices, not dependent upon some apathetic home consumer company, not connected to the internet, etc. And yet so many developers are so amazingly uneducated and inexperienced that they think Windows is the perfect solution to everything, managers love Windows because they can hire so many cheap ass developers for it and mistakenly think they can save time this way. Maybe Windows is not the flaw, but the Windows mindset certainly is.

Re:Windows 10 update will kill human beings

[Darinbob]

IT departments are often run by people with the least amount of real world experience. They see the word "computer" and then insist that it must have antivirus, be monitored for application not approved by IT, have corporate approved configuration, etc. I swear I got a notice that my Macbook Pro was due to be upgraded to Windows 7. Do these people stay up worrying at night that their automobile's fuel injection system doesn't have malware protection, or that grandpa's pacemaker doesn't have the latest version of Office?

Re:Windows 10 update will kill human beings

[mlw4428]

You mean bad developers? There are plenty of Linux systems that break too - it's bad developers. Windows itself can't be blamed, you should look more at the education system, managers who think that coders who go to "coding boot camps" or were trained in India are proficient developers. You get what you pay for and, honestly, most developers are just that - cheap. To suggest it's a "Windows" mindset completely evaporates the responsibility that American managers have for how their is developed. It also shunts responsibility away from companies who believe that software developers from 3rd world nations are, on the whole, just as competent as American developers but magically work for peanuts.

Re:Windows 10 update will kill human beings

[ChumpusRex2003]

The defective application in this example is a electronic records system. It allows the doctors/nurses/technicians to enter medical data during a procedure, collects X-ray image metadata from the imaging equipment and combines with the doctors notes for transmission to a medical records system, etc.

Re:Windows 10 update will kill human beings

[bluescrn]

"Armored cars may have bulletproof windows"

Re:Windows 10 update will kill human beings

[Hognoxious]

Count me as slightly reassured. I figured it wouldn't be the cardiac monitor, but I wondered if they were doing an operation while learning it from youtube.

Re:Windows 10 update will kill human beings

[Ol Olsoc]

"Armored cars may have bulletproof windows"

Cymbal crash!

Re:Windows 10 update will kill human beings

[thegarbz]

The OS doesn't randomly send valid data to a specific place any more than my TV remote suddenly starts changing the channel for no good reason.
And what the heck as a cat (operator error) got to do with an OS?

The analogy stands just fine.

Re:Windows 10 update will kill human beings

[thegarbz]

Oh please, I get Windows systems to fix after every update.

And I just ran do-release-upgrade on a 15.05 LTS version of Ubuntu linux rendering many applications unusable and requiring quite a bit of work to get the machine back to a usable state. What's your point again?

With your last breath, and on your tombstone, you'll want everyone to know "This was NOT Window's fault!!

I hope it does. Maybe we can finally implicate the vendor who produced a shoddy piece of programming that resulted in their application freezing when a 3rd party application locked a file it was using for scanning. I'm sure that could never happen on any other OS *rolls eyes*

For you see, my dear chachalaca, you go apeshit nuts, fearing that I disparage your precious Operating system, when I'm saying that not one, not any, not in a million years, should a life critical ever be connected in any way, any form, any remote chance of connecting to the internet.

Do you still have a point, or are you just spitting out random irrelevant words like "internet" that have nothing to do with the situation at hand?

And as Windows based systems are compromised in hospitals all over the country, rest easy that it is never their fault.

Oh if a windows system is compromised due to the OS it's most definitely a windows problem, and windows at fault. Interestingly that wasn't even remotely the case here.

Perhaps its time that Hospitals shift to Operating systems that accept some accountability.

Yeah we should start with all those Linux systems which we constantly see a stream of CVE's about. Oh wait you don't think a medical device will magically get a 0-day update do you? How naive.

*This post bought and paid for by no one. But if someone is reading, please how do I get a job where I can get paid for arguing with idiots on the internet? I really want to know. Microsoft HIRE ME PLEASE!

Re:Windows 10 update will kill human beings

[Ol Olsoc]

Oh please, I get Windows systems to fix after every update.

And I just ran do-release-upgrade on a 15.05 LTS version of Ubuntu linux rendering many applications unusable and requiring quite a bit of work to get the machine back to a usable state. What's your point again?

My point is that no one has ever had any problem with upgrading Windows. Right? A 100 percent upgrade success rate. And allow me to take your approach. If you upgraded Linux and had problems...

It...

IS...

ALL

Your ...

Fault,

Seriousy fellow, I have no idea how you have so many problems upgrading or using Linux, anyone who knows whether their ass is punched or bored, can make it work, so don't blame your problems on Linux. Blame it on Linux software writers, or your own personal inability.

After all, what is good for the Windows Goose, is likewise applicable to the Linux Gander.

And for what it is worth, I conosider what I wrote as equally ridiculous as what you wrote.P the difference? I don't believe what I wrote in sarcasm. However you are as convinced as a person could be.

Re:Windows 10 update will kill human beings

[doccus]

Doesn't matter if it was windows (which of COURSE it was) or not. Just like the case where some presentation was interrupted with a win 10 update screen, the problem is sloppy IT management. Hate to say it, bit the fault really isn't windows, or M$s.. The update should have been blocked prior to the presentation, and the virus scan should be set to *not* run when the system is performing an active operation. Period.
This should be pretty obvious to anyone in IT. Makes for lousy press though.. "Operation bllocked because of virus scan" sounds so much morre lurid..

Re:Windows 10 update will kill human beings

[lsatenstein]

It was probably XP. A lot of equipment, from Casino slot machines to medical equipment use XP as the base operating system.
  That is still true today.

Re:Windows 10 update will kill human beings

[Ol Olsoc]

It was probably XP. A lot of equipment, from Casino slot machines to medical equipment use XP as the base operating system. That is still true today.

Never got a BSOD in XP?

No problem

[NotDrWho]

Our antivirus is completely up to da

Upgrading to Windows 10......

Re:No problem

[Anonymous Coward]

Many(most) Hospitals and medical centers are still stuck on Windows XP, there's no upgrading to Windows 10.

Re:No problem

[Anonymous Coward]

Good tip, bro. Off to go install XP on all the critical machines.

Re:No problem

[xSauronx]

seriously. hospital IT has to lag way behind, often because vendor software doesnt support newer OS versions. I know a medical center that has thousands of desktops and only started rolling out windows 7 last year.

I was an intern there 5 years ago, and i was on the team that was deploying XP SP2 [yes, 2, not a typo] at the time.

many of their software vendors are the frickin worst.

there were some scanner pcs, like for x-rays or MRIs or something, i don tknow what, that ran Windows NT or Win2k--i would not be surprised if some of them were still there.

Re:No problem

[some old guy]

Having worked in biochem, it's not the hardware vendors causing the lag, it's the FDA-mandated cGMP validation and certification process that takes for.fucking.ever and has to be repeated for every tiny little change. Yes, it helps ensure quality and consistency, but it is painfully slow and discourages change, however desirable.

Re:No problem

[mrchaotica]

The only sane way to develop such a thing would be for the vendor to be responsible for the entire software stack from the firmware on up. This sort of stuff should never be built on Windows in the first place!

Re:No problem

[Anonymous Coward]

Some vendors are using Windows 3.11 and specifically tell you not to upgrade to a newer version. And... some of the workstation consoles are Windows CE networked to a mainframe running XP. That way users can't install crap or fuck with stuff.

Re:No problem

[St.Creed]

I really can't tell if you're joking or not. I guess you are, since you are very far from reality.

If you did mean this to be taken serious, I'd like to see you give a cost estimate for building a "hello world"-app as per your proposal.

Re:No problem

[bangular]

That would be impracticable. You could however require a "certified OS" that is more along the lines of QNX which supports QT. My guess is, the reason this isn't done is because managers consider Windows "off the shelf" and think it's going to be cheaper.

Re:No problem

[Anonymous Coward]

Sounds like the FAA. Software is not certified, complete systems are (in this case all the hardware and all the software, including AV). That would ensure that something like this would not happen when it's not supposed to if all procedures were followed correctly.

Re:No problem

[Opportunist]

You ARE aware that CE and XP are about as airtight as a sieve by now, yes? That users can't fuck with stuff is a HUGE claim when you make it for an OS that has not been patched for years but that has received new exploits on an almost daily base.

Re:No problem

[mrchaotica]

I'm not saying you should be redeveloping an entire OS every time. I'm saying that you should maintain a fork of the OS along with your application. In life-safety-critical situations, you need to have complete control of the environment in which the software is running.

Re:No problem

[JoeMerchant]

Your fork will contain all the vulnerabilities (known and unknown) that are present at the time of the fork.

The problem here is negligence in not patching the vulnerabilities when possible. If you're going to patch when possible, you might as well run with a standard release - delayed to give time to validate said release for acceptable regressions. This is sort-of what's done today, except that many validations are indefinitely delayed, until a substantial problem is reported.

Re:No problem

[Joe_Dragon]

Any os/2 still there?

Re:No problem

[Billly Gates]

You ARE aware that CE and XP are about as airtight as a sieve by now, yes? That users can't fuck with stuff is a HUGE claim when you make it for an OS that has not been patched for years but that has received new exploits on an almost daily base.

Mr. IT guy. This is your boss and the head of radiology. WHat the fuck did you do to my ability to go on the live internet with IE 6 and send my radiology results from the equipment? Get them back up ASAP! ... and I expect with 0 updates to be all secure 100% of the time. The vendor told me it was possible with that sexy training video just like the ones from teh SCADA equipment that run live on the internet with no firewall! Get it done

Re:No problem

[jbmartin6]

Not correct. I know medical vendors like to use this as an excuse. But the FDA continues to state that software does not need to be recertified except in cases of major changes to the functionality. The manufacturer is still responsible for quality testing however.

Ordinarily, FDA will not need to review software patches before a device manufacturer puts them in place. FDA views most software patches as design changes that manufacturers can make without prior discussion with FDA. ......For example, manufacturers need to seek FDA's approval or clearance before installing a software patch if it would change who it’s for, what it does, or how it works (a change in the indication for use), and/or it would make the device less safe and effective.

Re:No problem

[Anonymous Coward]

My touch sreen project has no OS. I am thinking of adding an RTOS.

The Problem is people like you that can not imagine a world of computers that do not have an OS.

Hello world, maybe 2 days.

Re:No problem

[thegarbz]

This sort of stuff should never be built on Windows in the first place!

Why not? When was the last time Windows has thrown a bluescreen without external help, poorly coded drivers, or failing hardware? Windows itself is incredibly resilient even for all the shit developers throw at it. Hell it can even cope with faulty video card hardware or crashing drivers mid games these days, not only with the OS surviving but without actually even disrupting the game.

It's been a long time since Windows was able to crash on it's own accord, like 1998 where the system went down every 49.7 days.

Re:No problem

[thegarbz]

I wonder if you're the developer of custom hardware and software, what do you think would provide a more bug free experience? A system which is common, off the shelf and expertise is very readily available? Or a custom OS with few developers to choose from a pool?

Given that what happened here had nothing to do with the OS, I'm inclined to believe that developing on a custom special purpose platform would not have improved the situation.

Re:No problem

[Anonymous Coward]

I know of one Cancer treatment radiology center in SoCal that is still using NT 4.0 on the controller machine for their RadScanner.

Re:No problem

[Opportunist]

Mr Boss.

I quit.

I refuse to work for idiots.

And I refuse to sit on ejector seats that I do not have the control over.

Signed,

Your former CISO.

Re:No problem

[Anonymous Coward]

I second this because I developed software in the blood collection/testing area for about a decade.

Business people only see software testing as a Cost that they want to avoid

As a result we had about a dozen ancient machines chugging away with a single software package running on each one

We made a concerted, and well documented, effort to write our own software for each system and then hosted them on Win VMs

In the short term it was a buttload of testing and documentation, but in the long term it was far more stable and...

Oh yeah, they riffed me, and all of the developers got pissed and left so... they pretty much hung themselves as far as support goes

Sometimes I quietly chuckle myself to sleep over that one

Re:No problem

[Billly Gates]

Hence why I refuse to work for hospitals. I contracted for a few

Re:No problem

[Darinbob]

However if you don't attach it to the network you're ok. And you shouldn't ever be using something that's a "standard release" in embedded systems, either a custom made OS, or an OS designed for embedded systems, not something stupid like Windows. Are there vulnerabilities? Sure, but no attacker should be able to have access if you put security around the system (no network, no plugging in random drives found in the parking lot, lock the room when it's in storage, etc). Using Windows is a shortcut that comes back to hurt the vendor and customer over time.

Re:No problem

[Darinbob]

They also think the developers will be cheaper, that no training will be necessary, etc.

Re:No problem

[Darinbob]

Because Windows is designed at every stage to be a general purpose computer. It is marketed primarily to corporate and home users. But these are specialized machines. They may not have a UI that conforms to any Windows standard, there is absolutely no need for a "desktop", they're always turn-key systems. Windows brings no advantages to the system unless it's as a display platform, in which case you separate the Windows machine from the medical machine and have them be independent (ie, take X-rays from one machine, send the images to a database, then the Windows machine can browse and display the images without screwing up the medical machine). If there is nothing whatsoever in the system that needs Windows then it's stupid to use Windows as the base. If you need tasks then use an RTOS.

Re:No problem

[HiThere]

To consider: Instead of adding RTOS, consider adding a separate processor without an OS. You only need an OS if more than one program is going to be using the processor at the same time. Micro-scale computers are now cheap (though I don't know how reliable). So if your project currently has no OS, and you just want to run two programs at once, you might use two separate processors. Or if ONE of them needs hard real-time, you might put a normal OS on the other one, and have it share the non-real-time programs.

Not sure how that would work out, but real-time OSs used to be a nightmare to work with, and any reasonable way of avoiding it was reasonable to investigate.

That said, of course it depends a lot on what your touch screen project is and what your real-time requirements are.

Re:No problem

[Darinbob]

Most radiology machines that run on custom operating systems, commercial RTOSs, free RTOSs, or no operating system at all, are still able to send DICOM conformat image files to the computer on the doctor's desk or to the image database. If the IT guy wants his database to crash then that's ok, as long as the machine that touches the patient is not based upon Windows.

Re:No problem

[HiThere]

Yes, but the fork can have a really trimmed down and tightened up communications connection. Say only allow RS232-c connections to communicate. It's not a high speed connection, but you aren't passing any hefty baud rate over it. (Well, not for most devices.) You don't need to allow internet connections at all. Or possibly you could use an RS232-c for input and a video connector for output.

If you handle the communications properly, the only vulnerabilities that will matter are those targeted specifically towards your device.

Re:No problem

[SvnLyrBrto]

A mainframe that runs on Windows XP? I doubt it... a lot.

Re: No problem

[Anonymous Coward]

Hmm. I have been relatively crash-free running W10, until a bad driver for the usb-serial port blue-screened @ 115200bd.

Re: No problem

[Anonymous Coward]

And ? Why do you want to share this ?

Re:No problem

[Opportunist]

I am still. But I would NEVER EVER take up that seat. No matter the pay.

I've met a few CSOs of hospitals. Invariably they are the worst of their ilk. By definition. Any CSO who knows his shit won't tough something like this with a 10 foot pole. What they get is the kind of person that can't land a decent job and has to settle for that hot seat, enjoy the ride while it lasts and take one for the team when (not if) the brown mass hits the rotating cooling device.

Re:No problem

[thegarbz]

But these are specialized machines. They may not have a UI that conforms to any Windows standard, there is absolutely no need for a "desktop", they're always turn-key systems. Windows brings no advantages to the system unless it's as a display platform, in which case you separate the Windows machine from the medical machine and have them be independent

Indeed, and this was the case here. The equipment itself didn't run windows, it did it's stuff and sent data to monitoring / logging software on windows via a serial link. The display and monitoring software was what froze. But more interestingly it froze entirely because someone didn't RTFM when they setup the machine.

Idiots and crappy programmers are OS agnostic.

Re:No problem

[JoeMerchant]

Something stupid like windows can be the perfect ticket to a rapid development cycle, getting a product to market ahead of the competition and more importantly: helping people rather than sitting around re-inventing the software wheel. Personally, I prefer something stupid like Ubuntu or Raspbian, but Windows is a valid choice, if your product doesn't suffer from its shortcomings.

If your product never connects to the network or external storage, then enforce that with hardware design. If you do connect to the network, I'd strongly advise staying away from Windows, especially 10. With some care in configuration, you should be able to "harden" windows to USB stick-borne illnesses.

Sure, every system can be developed from the ground-up, and if all were done that way, viruses would have a harder (but not impossible) time propagating. The problem comes in when the systems start to talk to each other, whether through network, or file system, or flashing light morse code - interfaces bring vulnerability, the more you have and the more universally accessible they are, the more vulnerable they become; can you remote update your software? That's the ultimate game-over security hole.

The products I develop improve and save lives, extending from a 9 month to a 30 month development cycle means fewer people get helped by the innovations, some will die while they are in development. If slapping a standard OS in the box gets it to market a year faster, that's much more benefit than the risks it typically brings.

Re:No problem

[Anonymous Coward]

Totally agree, mission critical or life critical software should never be built on Windows. We never used windows in Aerospace to run engines, flight controls, etc...

It's too big, you'll never know what's truly going on. There are alternatives, look to what the industrial guys are using to run things like nuclear power plants, large refineries, all of which if there was a crash would could have fatal consequences. For mission critical apps, you need to be able to see all the code, even the OS.

Windows was not designed for real time control applications.

Re: No problem

[HiThere]

OK. USB is probably OK too, though I'm less familiar with the protocols. You want to make sure that you control the driver, but if it's simple enough, and doesn't require commands to be embedded in the data (UGH: JavaScript, etc. HTML1 is probably ok, but why?) then it should be ok.

Now, of course, this won't protect you against being communicated to by devices that abuse the protocol, so you need to ensure that your interpretations of it are secure against anything except over-voltage (and you can try to protect against that, but don't feel secure).

But for controlling such a device RS232-C is really overkill, and for getting data out a video driver is all that you need. And often more. Medical device connections aren't usually space limited, so USB has minimal advantages in that way. And the connection protocols are simple enough that I once wrote one from scratch. (I wanted to control a printer from a terminal that had a documented output port. I forget why, but I later adapted that into an X-Modem connection, but I don't remember what the device was that it connected to. But the protocol was simple. The problem was cable wiring didn't match any known design, and I ended up needing to make about 50 custom cables. If they'd actually followed the RS232-C specs rather than using their own special pin-out mapping it would have been a lot easier.)

RT OS for Reatime tasks

[Anonymous Coward]

Picking an OS that clear says not use it for real time possible life endangering task is a huge mistake!! QNX, RT_Linux, and more!!! Hello!!!

Re:RT OS for Reatime tasks

[Anonymous Coward]

Nobody is listening to us. FDA is not reading the EULAs of the components of the solution under certification. Any reasonable person know that these systems might kill people. Nobody just don't care before it's their precious mommy that is going because of it.

Re:RT OS for Reatime tasks

[Anonymous Coward]

Around 2002 we had a vendor come in marketing a real time OS from Microsoft. Having not heard of such a product line from MS, I was quite skeptical. Once it became apparent every change whether in system required reboots, & w/ daily scheduled reboots. Skepticism became, "GTFO my shop." So yes there were vendors peddling "buzz word" real time OS solutions. Sadly this case seems to be the hospital doctors dictating IT policy based on the " Ooooooh, Aaaaaaaaaah," principle. Which is has been a rampant problem at quite a few hospitals.

Re:RT OS for Reatime tasks

[DarkOx]

I have often wondered about this. Does Microsoft sell Windows license with a EULAs that don't contain prohibitions for uses cases like these?

The Microsoft software was designed for systems that do not require fail-safe performance. You may not use the Microsoft software in any device or system in which a malfunction of the software would result in foreseeable risk of injury or death to any person.

In most other engineering professions if you picked a component specifically labeled and sold as not fit for use case you'd be taking on all kinds of liability. Can you imagine if an architect decided to build a parking deck and spec'd concrete be mixed from a cement product labeled "not for structural use?"

I can hear the lawyers salivating at the very idea. Yet Windows is used in off label ways seemingly all the time.

Re:RT OS for Reatime tasks

[xanthines-R-yummy]

I completely agree with you, but this case isn't like a Da Vinci surgery robot crashing (or going haywire!) in the middle of the surgery. It's more like the camera/imaging equipment crash. Yeah, the cardiologist was probably pissed/confused and the OR techs and nurses were freaking out a bit, but I doubt the patient was in any actual direct danger from the crash. Any danger would indirect, such as prolonging the procedure and exposing the patient to more anesthetic, or rendering the procedure futile and they'd have to try it all over again the next day.

Disclaimer: It's been more than a few years since I've seen a cardiac cath.

Re:RT OS for Reatime tasks

[Anonymous Coward]

I've been using QNX for the past 2 years. I'm hoping I never have to use it again and that it will die off. If you look at the forums you'll see it's pretty much dying of thirst in the desert as the tumbleweed rolls by anyway. The concept of the micro-kernel and its safety is undermined by the fact they had to use a synchronous message passing system to regain performance. It only takes one software mistake to forget to respond to a message and the whole system locks up. Reboot required and hopefully the patient didn't die on the operating table.

If you want an OS that's as bad as the virus scanner then go ahead, QNX all the way!!

And if you want a longer list of why QNX sucks the big one then I can go into all sorts of other aspects where its completely brain damaged in other ways.
P.S. Their documentation is utter tripe.

Re:RT OS for Reatime tasks

[Anonymous Coward]

Likely not the doctors, but the accounting office.

Doctors don't get the ability to mandate the equipment they use. They can insist - but that doen't mean the PTBs are listening.

Re:RT OS for Reatime tasks

[DarkOx]

I don't know, I mean IANAHS (I am not a heart surgeon) but it seems to me that we are using all kinds of imaging equipment to do things like laparoscopic surgeries that we could not have done before. This isn't like the lane departure warning sensor in your car failing, where you can just drive like you always used to do. Its seems very possible that the loss of imaging equipment in the OR mid surgery could throw the entire plan off in away that very well could endanger the patients life.

Even monitoring equipment: Is a modern anesthesiologist prepared to wait for nurse to have to count out a pulse in the middle of a procedure that was supposed to be done with technical assistance?

Re:RT OS for Reatime tasks

[peragrin]

Except the camera is how they see. You should look up the procedures for heart catjorization and how they put in stints. It is scary if you go blind at the wrong time.

Re:RT OS for Reatime tasks

[alantus]

Nobody just don't care before it's their precious mommy that is going because of it.

Ain't nobody got time for that.

Re:RT OS for Reatime tasks

[nmb3000]

Picking an OS that clear says not use it for real time possible life endangering task is a huge mistake!! QNX, RT_Linux, and more!!! Hello!!!

Absolutely, and I hope the manufacturer gets sued into oblivion followed by criminal litigation for the C-level. There should be zero tolerance for this kind of insane sociopath behavior that trades people's lives for dollars.

Everyone wants to use commodity hardware and a commodity operating system because it saves (a lot of) money and is "easier" to design and develop. The only problem is your Visual Basic 6 heart monitor with a UI written in Flash running on Windows 8 with McAfee and Microsoft fucking everything up ever 60 minutes should never have qualified as a "medical device".

Whether it's cars, heart scanners, multi-million dollar electron microscopes, or whatever else, we seem to be witnessing the death of solid systems designed by real engineers in favor crap thrown together by recent graduates of Kode Kamp where the primary design goal is supporting "apps".

Re:RT OS for Reatime tasks

[FunkSoulBrother]

Doctors don't get the ability to mandate the equipment they use. They can insist - but that doen't mean the PTBs are listening.

What's a PTB?

Re:RT OS for Reatime tasks

[Anonymous Coward]

That's called "cost benefit analysis". I.e. how much more money can I make if it do things using "Brand X" for something rather than the superior "Brand Y" and does that offset the cost of any _theoritical_ lawsuit which _might_ occur in the _future_ as opposed the near-term profit, which will result in _my_ getting a bigger bonus. E.g. It cost's _your_ mom her life, but it gets _me_ a new superyacht, so your cost is to my benefit - a "no brainer" for me! Anyway, by the time the chickens come home to roost on this, I'll be long gone to another company.

Re:RT OS for Reatime tasks

[Grishnakh]

I sure hope there's some utterly gigantic lawsuits for this bullshit before too many people get killed. There is absolutely no excuse for using Windows in any life-critical application. Even Microsoft says so in their EULA.

Re:RT OS for Reatime tasks

[worf_mo]

What's a PTB?

Pointy-tailed Boss

Re:RT OS for Reatime tasks

[Grishnakh]

Yeah, the cardiologist was probably pissed/confused and the OR techs and nurses were freaking out a bit, but I doubt the patient was in any actual direct danger from the crash. Any danger would indirect, such as prolonging the procedure and exposing the patient to more anesthetic, or rendering the procedure futile and they'd have to try it all over again the next day.

Those are all bad enough. Surgery is serious business, and forcing a surgery to be botched so that the patient has to go through with it again is a serious risk to life and limb. I sure hope the patient in this case sues the hell out of everyone involved here for using a Windows-based solution.

Re:RT OS for Reatime tasks

[Grishnakh]

Very interesting, usually QNX gets glowing praise, but it does seem like a lot of that praise comes from people who've never worked directly with it and are just passing on the general reputation, much like people saying how great BMWs are even though they've never driven one or worked on one. (How do you know that "German engineering" is so great if you've never actually worked on one of them? FWIW I've never worked on one either, I'm just pointing out this fallacy.)

What RTOSes would you suggest for this application? Green Hills maybe? They seem to make good use of it in military applications. Nucleus?

Re:RT OS for Reatime tasks

[St.Creed]

I sure hope the patient in this case sues the hell out of everyone involved here for using a Windows-based solution.

In that case the patient won't get very far with his case. The point the patient *could* try to win the case on is whether the GxP-process included testing on the operating system used for the procedure, and the other software on it. What were the installation instructions for the software and where they obeyed to the letter? If it says "Install on windows XP and use whatever you like because it will run" it's likely the vendors are liable. If it says otherwise, the hospital may be liable.

Of course, liable for damages means you have to have had any damage in the first place, which didn't actually happen. Being pissed off is no reason to claim damage where I live, but in the USA things may be otherwise.

Re:RT OS for Reatime tasks

[Opportunist]

But they effin' should!

Isn't that their job? To find out whether a system is "good enough" for use in life-or-death situations? How could they possibly approve a system that says ITSELF (i.e. it's not something you find out during testing, but the SYSTEM ITSELF TELLS YOU UPFRONT) it's not up to the job?

Re:RT OS for Reatime tasks

[bangular]

I would argue an RTOS is not necessary and may be cumbersome to use. A very minimal version of RHEL or similar would be ideal. No virus scan needed. OS updates could actually be heavily reviewed by the vendor since there would be so few on a minimal system.

Re:RT OS for Reatime tasks

[l0n3s0m3phr34k]

Hospitals don't get to "pick", they don't get to install their own OS. All this equipment spends a long time getting FDA certified (which there are several different versions of), the "end users" (the hospital) isn't allowed to do anything on these devices. Complain to the manufacturer, in this case it's IBM. A good friend of mine works for Oregon Health & Science University, he's been fighting with GE due to their "non-supported OS's" that GE is still shipping on "brand new" devices...like Windows 2000. GE blames the FDA. Round and round it goes.

Re:RT OS for Reatime tasks

[JoeMerchant]

Representing the legal profession's interests much?

From a straight risk-benefit perspective, the risks are very small as compared to the benefits. Sure, due diligence is expected, carelessness should not be tolerated, and there are regulatory systems in place attempting to ensure that best practices are followed. It's an imperfect system, but there's plenty of system in place already working to minimize stupid stuff like OP. Including the option to sue for damages, if damages actually happen.

Re:RT OS for Reatime tasks

[mattventura]

But technically, it wasn't MS's software that caused the failure, it was the AV software.

Re:RT OS for Reatime tasks

[Anonymous Coward]

In the aviation world, we have DO-178B

A Catastrophic – Failure may cause a crash. Error or loss of critical function required to safely fly and land aircraft.

B Hazardous – Failure has a large negative impact on safety or performance, or reduces the ability of the crew to operate the aircraft due to physical distress or a higher workload, or causes serious or fatal injuries among the passengers. (Safety-significant)

C Major – Failure is significant, but has a lesser impact than a Hazardous failure (for example, leads to passenger discomfort rather than injuries) or significantly increases crew workload (safety related)

D Minor – Failure is noticeable, but has a lesser impact than a Major failure (for example, causing passenger inconvenience or a routine flight plan change)

E No Effect – Failure has no impact on safety, aircraft operation, or crew workload.

Most displays are at Level A

Re:RT OS for Reatime tasks

[Anonymous Coward]

If they are in compliance, I guess they are safe from liability. It's nice to write the regulations that one can hide behind. I am sure government will do something to help its citizens...hahahahaha

Re:RT OS for Reatime tasks

[Bob the Super Hamste]

Never worked with QNX but have worked on BMWs as well as other cars. I find that BMW are put together pretty well and are easier to work on in general than other cars. For example on my previous car I had to replace the radiator sometime after 200,000 miles, I forget exactly when but I figured I would also do the water pump at the same time since at that point would be really easy to do and was on borrowed time. I had to take off the fan shroud, fan, disconnect the upper and lower radiator hoses, and then undo 2 clips. Compared to one of my buddies who I helped a couple of weeks ago change the radiator on his van the BMW was a dream, his van also required removal of the front grill, and bumper cover as well as removing a larger number of fasteners. Probably the worst vehicles I have worked on were VWs which is strange considering that for the longest time they were pushing the whole "German Engineered" thing while BMW and Mercedes Benz weren't.

Re:RT OS for Reatime tasks

[Anonymous Coward]

ROFL!!
Any sane person using Linux on embedded hardware would be using Yocto, not RHEL. Yocto is what the majority of the hardware venders have settled on supporting.

Re:RT OS for Reatime tasks

[jedidiah]

> Representing the legal profession's interests much?

Ambulance chasers are the only reason you have ANY legal representation. Despite their many flaws, they defend the little guy from the 1%. Otherwise, they would be free to turn you into Soylent.

Re:RT OS for Reatime tasks

[houghi]

In some places the EULA is utterly meaningless. Just because you clicked OK (or even signed a contract) does not mean it is legal.

Re:RT OS for Reatime tasks

[MyDogHasFleas]

Powers That Be

Re:RT OS for Reatime tasks

[Darinbob]

Microsoft wants to be in every market. Whether or not they have any experience or expertise in those markets. So they do want to get into the embedded and real time markets. And so they sell Windows version of those things and they're almost always a disaster, they add lots of time and expense as well while claiming to be simpler (usually by implying that any moron capable of writing a Windows application is suitable to write your real time applications as well).

Re:RT OS for Reatime tasks

[HiThere]

Real-time OSs have a long reputation for being a pain in the ass. I don't know whether this has changed in the last decade, but what has changed is the price of adding an additional CPU. You only need an OS if more than one program is going to be running on the same processor. If you can include enough processors for all real time jobs + 1 to run everything else on, you don't need an RTOS. You *do* need a good inter-processor communications mechanism (which must handle availability, as the real-time processors can't).

I'm just talking off the top of my head, as I've never designed the kind of system I'm talking about, but with the decline in price of low-end processors it seems like a quite reasonable approach. And RTOSs have the strong reputation of something to be avoided if at all possible.

Re:RT OS for Reatime tasks

[Anonymous Coward]

Someone could now be liable for any complications arising from the surgery where they were not previously liable, and if the patient can prove that they were really freaked out about it afterwards they might be able to sue for negligent infliction of emotional distress or something. It is hard for me to imagine someone winning a case like that without any actual damages to go with it, but then, IANAL.

Re:RT OS for Reatime tasks

[Anonymous Coward]

This issue was due to the data being unavailable as it was locked. The OS had F all to do with it. This was a software issue, and had nothing to do with the OS. It was the Doctors computer that had the issue.

Re:RT OS for Reatime tasks

[Anonymous Coward]

Country song about failing medical imaging devices incoming! The audience will be in tears.

Re:RT OS for Reatime tasks

[JoeMerchant]

> Representing the legal profession's interests much?

Ambulance chasers are the only reason you have ANY legal representation. Despite their many flaws, they defend the little guy from the 1%. Otherwise, they would be free to turn you into Soylent.

See, I don't view the ambulance chasers as defending the little guy from the 1% so much as using the little guys to extract big payouts from whatever deep pockets happen to be legally vulnerable so they might elevate themselves into the 1% as quickly as possible.

I don't feel protected by politicians on most issues, but if anything as egregious as Soylent Green were taking hold as "a thing," I have faith in our political system to produce sufficient candidates to squash something like that - far sooner than I think any lawyers might pull an Erin Brockovich on them. Most of the lawyers would be lined up to defend Soylent in the courts, looking for that fat payout.

Re:RT OS for Reatime tasks

[Anonymous Coward]

From the cartoon Dilbert, the Pointy Haired Boss is the exec who controls the department but has no knowledge of how or why it works (or doesn't).

I did not see the reference to PTB.

Re:RT OS for Reatime tasks

[Anonymous Coward]

Ha ha ha ha RHEL? You gotta be kidding, Ok its far better than windows but for a safety critical application - no thanks, still too many unknowns.
Yocto or Buildroot your own distribution - yeah sure, that might just work.

The whole reason RTOSs were brought up really is because you can control pretty much the whole system and know exactly what you've got on there - no weird services running in the background etc. You have a chance of testing the whole thing and being highly confident that it won't break.

You can make your RHEL like that but you'll be working backwards. Just build your own distribution with all the bits you need and none of the bits you don't. You've still got a lot of system to test though and most of it is provided stuff that you might not know how to test well.

RTOSs and linux are what I do for a job, ones a hammer and ones a screwdriver, both useful tools, you just gotta know when to use the right one. On some jobs its not an easy choice. All I know for sure is: don't use QNX.

Re:RT OS for Reatime tasks

[Anonymous Coward]

So QNX is a micro-kernel. This means the OS itself is quite small and can be certified fairly easily (at least compared to other OSs).
That's the plus points done. The consequence of this is that they leave everything else up to the user and have had to make some stunning compromises.
When you pass a message from one thread to another the thread of execution follows, the current thread that sent the message is blocked until a reply is received.
The effect of this is that you can end up blocked on a high priority thread waiting for a low priority thread to get around to reading its message queue. At this point people will say that there's a work around for this, yeah sure, a nasty sticky plaster for a fundamental design flaw. After the sticky plasters get so high you can't see the code for plasters you start getting a bit tired of it. You end up with the message handler spawning its own threads dynamically to handle messages so you don't block and then you have to mutex your variables that are contained in one thread because you have multiple message handler threads. I've never seen so many mutexes and semaphores outside of QNX code. Added to the fact that they don't treat everything as a file handle so you can't 'select' on messages and other i/o and you've got a hodge podge OS that is academically interesting but not something you want to use to really get something done. Oh the documentation is way too brief adn significantly incorrect at times too. It just hasn't got any of the benefits of many eyes going for it, it feels like there may be a one-eyed monster in a basement somewhere that looks at it once in a while. Help on forums is close to non-existent, a colleague posted a simple question on the QNX forums about 2 years ago and it still has no answer and is still on the first page of questions asked. You won't be getting any external support when you use it unless you pay the big bucks - which of course you'll already have had to do to even get a licence to work with it.
I've a longer list of problems but those are the core ones.

Re:RT OS for Reatime tasks

[Anonymous Coward]

So much AI chat? You do not need a high IQ to schedule such tasks, truly. But all schedulers fall into update at three in the morning and a few choices and that s it.

Re:RT OS for Reatime tasks

[Grishnakh]

Yeah, but these sound like problems common to many RTOSes (esp. the many-eyes problem). They just aren't widely used, and there's tons of them. They do work well for mission-critical environments though, especially avionics and military (which has a huge crossover). I worked in one 6-month job in that industry and it was interesting; they have a very rigid process for producing designs and code (heavily reliant on Rational software, namely DOORS). The code isn't very efficient, but that's not the goal, reliability is. But they weren't using QNX either, they were using an extremely small RTOS developed in-house with cooperative multitasking. I also worked at a company using Nucleus RTOS and did some work with that.

I think one big factor in small systems like the above is that the company making the system has total control of all the applications running on the device, so the whole thing is tested as a system. Users can't add anything, and 3rd-party software definitely isn't allowed. That prevents a lot of problems you see in general-purpose OSes, where the OS cannot trust the applications. In an RTOS, you can because it's all put together by the same team.

OS

[Anonymous Coward]

Should such a system really have a general purpose OS. There's an advantage to keeping things simple and having dedicated hardware.

Re:OS

[Anonymous Coward]

Should such a system really have a general purpose OS. There's an advantage to keeping things simple and having dedicated hardware.

So you keep things simple by installing tens of gigabytes of OS on it. Right. Might as well install Office while you're at it. "It looks you are performing open-heart surgery. Would you like help?"

Re:OS

[ChunderDownunder]

Any fancy software that displays information on a client PC and feeds clinical data into a patient records system should be epiphenomenal to the operation of the equipment.

But it seems like the hardware may have forgone having an embedded display and on-device control of the essential features, delegating those to a pc terminal.

Re:OS

[wardrich86]

MS Hospital Office Assistant. It's a pair of scissors named Snippy.

We've got some new features to get excited about

[JoeCommodore]

Don't turn off your PC

Manufacturer Narrative from FDA report.

[Anonymous Coward]

Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect. The product security recommendations, (b)(4), explicitly state, "the intent of these guidelines is to configure the anti-virus software so that it does not affect clinical performance and uptime while still being effective. To accomplish this, the anti-virus software needs to be configured to scan only the potentially vulnerable files on the system, while skipping the medical images and patient data files. Our experience has shown that improper configuration of anti-virus software can have adverse affects including downtime and clinically unusable performance. ".

Re:Manufacturer Narrative from FDA report.

[DarkOx]

To accomplish this, the anti-virus software needs to be configured to scan only the potentially vulnerable files on the system, while skipping the medical images and patient data files.

Right because there is no possible way the application could have a parsing bug handling patient files, that could lead to buffer overflow and an RCE or anything like that.

Its also true that similar bugs have never been found in commonly used image handling libraries...

Oh wait the second one is definitely not true and the first has at least a non-zero probability.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

Which completes excuses the unhandled exception in the product that they clearly knew about, or they wouldn't have so explicitly worded the instructions. I see the manufacturer failed to learn the lessons from the Therac-25.

Any system that requires humans to follow instructions that they read once a long time ago, but must follow exactly on a rarely performed task is an accident waiting to happen.

Re:Manufacturer Narrative from FDA report.

[Ol Olsoc]

Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect.

Fact is, this is exactly what you are going to get when using a Windows based system. You assume at the beginning that all problems are your fault, and that you must anticipate everything.

Based upon experience, they were also at fault because they were using Windows. Never at fault is the Windows operating system.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

Any system that requires humans to follow instructions that they read once a long time ago, but must follow exactly on a rarely performed task is an accident waiting to happen.

This is a system that requires humans to follow instructions when they're executing a rarely-performed task. Nobody should be installing AV software on a clinical workstation without understanding what AV software does and how that might impact real-time functions of the workstation. Reading the documentation for installed software would be a part of the due diligence involved in that task.

Also, your assumption that they clearly knew about an "unhandled exception" is ridiculously flawed. The point of that paragraph is simply - if you install AV on a Windows system that has a realtime workload, all bets are off as to performance of the realtime workload. That's what the docs say, and that's the fact of the matter, regardless of any residual bugs in the software relating to exceptional conditions.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

someone's got to link the relevant xkcd

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

You're trolling uncontrollably. Regardless of the operating system, it's just basic common sense not have certain automated tasks scheduled to run when critical equipment is being used. This smells of hardcore IT department failure. I've met guys like you in the field. When shit like this happens you just throw your hands up and say, "Ohhhhh! Well, it's WINDOZE!!!! Not my fault!", instead of admitting that you A) Don't know what you're doing, or B) Too lazy to do it.

Re:Manufacturer Narrative from FDA report.

[l0n3s0m3phr34k]

And running a scan EVERY HOUR seems overkill, not to throw puns around. These devices shouldn't even have real internet access, nor should they be accessed for anything that they need 24 virus scans run on them every day.With the speeds I've seen most scanners run at, it seems to me that this machine would be spending at least 25% of it's time running one of it's 8,760 yearly scans.

Re:Manufacturer Narrative from FDA report.

[Ol Olsoc]

You're trolling uncontrollably.

Of course you are 100 percent correct. Unfortunately, reality does not agree with you. A lot of places are using life critical applications with an operating system not ever supposed to be used that way. Since Windows can never fail, and all failures are anyone eles's fault but Windows, I have no idea why Microsoft didn't make them sign a document absolving them of any responsibility when it did fail.

Because like it or not, Windows systems which have the not Microsoft's fault - ever - failures all of the time. So is it that Micorsoft is so awesome, so perfect, or are the fanbois throwing all of h e Windows software writers iunder the bus?

It should never be used fo rlife critical applications - ever.

Troll I may be, but sometimes the truth is a Poe Troll.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

I can't believe I'm defending MS, but ...

Windows OS is not designed for hard realtime constraints or other life-critical systems and says as much. So, no. Windows is not at fault because it is being given responsibilities it was not intended to have.

Re:Manufacturer Narrative from FDA report.

[Ol Olsoc]

I can't believe I'm defending MS, but ...

Windows OS is not designed for hard realtime constraints or other life-critical systems and says as much. So, no. Windows is not at fault because it is being given responsibilities it was not intended to have.

From that standpoint, yes. That becomes a decision making process. Who decided to make Hospital networks and life critical applications so Windows centric?

And what does Microsoft know about them using it that way?

That might be the most liability exposure decision ever made, as we see that Hospitals, which have made their networked systems the forerunners of the Internet of things - are being held for ransom quite often these days.

Re:Manufacturer Narrative from FDA report.

[Jeremi]

And running a scan EVERY HOUR seems overkill, not to throw puns around.

Even if they scanned only once a month, the problem would still be there and could still bite someone. What Windows really needs is a checkbox that says "do not scan during open heart surgery" Or better yet, a handbook for software developers that says "do not write life-safety-critical software to run under Windows". :^P

Re:Manufacturer Narrative from FDA report.

[Ichijo]

there is no indication that the reported event was related to product malfunction or defect.

Should software be allowed to crash and turn the screen black when it can't open a file? Because that's what happened:

Unable to access real-time data, the app crashed spectacularly.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

Depends on if the labeling of the product says that will happen.

Re:Manufacturer Narrative from FDA report.

[Waccoon]

Yay, I was waiting for this. Blame the user. It's not like anything else besides anti-virus could lock up a PC unexpectedly, such as TrustedInstaller.exe.

I'd like to know why this company designed such critical software to be run on a multi-function PC or tablet. We've had decades to figure out this doesn't work.

Re:Manufacturer Narrative from FDA report.

[Anonymous Coward]

I appreciate that this happened, because the customary IT BS procedure of "install antivirus on it" has finally been documented as detrimental.

Scan every hour? Pure incompetence. What if it barely finished its scan and is constantly thrashing the HDD. Another known failure method instead of the virus boogeyman.

Redundant backup?

[Anonymous Coward]

For such a critical system, wouldn't it be wise to have redundant backups available? It seems negligent to me to rely on a single potential point of failure, with no backup.

Re:Redundant backup?

[gweihir]

Too expensive. Medical equipment already comes with an often insane price-tag.

Don't use windows for this

[HalAtWork]

Use some dedicated hardware with a custom software system with only components designed for the purpose of the machine and nothing else. Harden and sanity check the hell out of the I/O and connect THAT to your idiot box.

Re:Don't use windows for this

[Anonymous Coward]

An even better idea would be to have such an important piece of equipment NOT connected to the network.

Re:Don't use windows for this

[Khyber]

An even better idea would be to quit relying upon technology, because it won't always be there for you, as quite handily proven in this case.

If you, as a doctor, need a computer to perform a task that did not originally require one, you're failing the basic concepts of medicine.

Re:Don't use windows for this

[clodney]

Use some dedicated hardware with a custom software system with only components designed for the purpose of the machine and nothing else. Harden and sanity check the hell out of the I/O and connect THAT to your idiot box.

On the one hand, that makes perfect sense, and removes a whole bunch of failure modes.

On the other hand, that makes it a more expensive device to build and maintain, increasing the cost of health care all around, and ultimately squeezing out other components of health care.

Medical devices deal with risk minimization, and that often involves deciding which risks are acceptable. If the device can be sold for $5,000 based on a Windows PC, and $15,000 based on a proprietary hardware stack, and because of that a hospital buys 1 expensive machine instead of 3 cheap ones, patient outcomes as a whole will probably be worse.

And dedicated hardware has its own issues - it is rarely updated when vulnerabilities are discovered, and often lingers one for years after the manufacturer is gone or has moved on. At least putting an off the shelf PC in the system makes it possible to update some of the components.

Re:Don't use windows for this

[Anonymous Coward]

So, to be a cardiologist these days you should be born with a fiber optic port wired directly into your brain?

Your ideas intrigue me and I would like to subscribe to your newsletter.

Re:Don't use windows for this

[Grishnakh]

Without technology, doctors wouldn't be able to do a lot of the stuff they can do now.

This is like saying we don't need ultra-reliable avionics systems because we shouldn't rely on technology, so we should just go back to horses and wagons.

Relying on technology is fine as long as you do it right. Using Windows doesn't qualify there. It'd be criminally negligent to use Windows on an airplane's avionics systems, and it's no different for medical systems.

Re:Don't use windows for this

[Grishnakh]

You don't need a proprietary hardware stack, you just need a decent RTOS. There's plenty of them out there. But you're not going to be able to use Visual Basic with those.

Re:Don't use windows for this

[Khyber]

You don't need a computer to run a camera. To boot, a single-purpose medical device shouldn't need some massively complex operating system PLUS anti-virus.

Re:Don't use windows for this

[Khyber]

It's more criminally negligent to utilize general-purpose stuff for specialized medical operations.

This is why we have application-specific hardware.

Re:Don't use windows for this

[Dutch Gun]

Or, having learned a valuable lesson, we could instead fix the problems with the technology and make it more reliable.

That's generally how engineering works, after all, and it seems to do pretty well by us for the most part. It's why we know how to build buildings, bridges, airplanes, etc that generally don't randomly fail. When we do see a failure, we analyze what went wrong and try to learn from the sometimes painful lesson, and the next generation of products then takes what we learned into account.

Re:Don't use windows for this

[Anonymous Coward]

a single-purpose medical device shouldn't need some massively complex operating system

What a load of shit, "single purpose" is meaningless here. "Single purpose" encompasses a multitude of technologies.

Re:Don't use windows for this

[bored]

Right, which drives the prices up, and when presented with a less expensive pretty looking product the less expensive one will be chosen over the one with "proper engineering". The doctors aren't going to be technicality proficient enough to analyze the differences, which in theory the FDA will be doing for them. But the FDA, like most other government regulators is understaffed/etc because close to 2/3rds of the legislature believes that the government shouldn't be regulating stuff like this. So its better to spend 1.3T on couple hundred planes than provide reasonable secondary education, or competent regulation that keeps fertilizer plants (and other heavy industries) away from residential areas, monitors drinking water quality, or rebuilds bridges before they collapse.

Re:Don't use windows for this

[ultranova]

It's more criminally negligent to utilize general-purpose stuff for specialized medical operations.

This is why we have application-specific hardware.

We don't currently have the capacity to manufacture application-specific hardware in sufficient quantities to use it for medical devices. You'll have to wait until assembly lines can be reconfigured without slowing down, or 3D printers catch up to them on speed and product quality.

Our technology just isn't quite there yet.

Re:Don't use windows for this

[thegarbz]

Why not? Given that this was caused by the user not reading the fucking manual, how would any other OS have prevented this?

Re:Don't use windows for this

[Khyber]

Uhh, to record audio logs of surgical procedures, do you need more than a microphone and tape recorder? Do you need anti-virus?

Nope.

To record usage of medical instruments during said procedure, do you need more than paper and pencil and human to operate said equipment?

Nope.

Re:Don't use windows for this

[Khyber]

"We don't currently have the capacity to manufacture application-specific hardware in sufficient quantities to use it for medical devices."

I think you're failing to take the context of the story along wiht my comment.

To record logs, one only needs a microphone, and tape recorder.

We've had enough of those to supply the entire planet twenty times over, now.

You don't need a computer with anti-virus for such a simple damned thing.

So was this out of spec?

[GIL_Dude]

This is interesting; the configuration on a device like this should be highly controlled. I have no experience with medical devices, but I know that process control equipment generally has vendor approved configuration (and often they only certify one AV vendor so even if our corporate contract is with vendor A, we have to use vendor B for the process control stuff because that is what is certified by the control system vendor. They also have very specific settings you have to use. Failure to follow the settings could result in lack of process control at a critical time. It seems medical stuff must be under similar (if not even more restrictive) configuration control. Having AV do a "scan" every hour is very stupid since any competent AV is doing on-access scanning anyway. I would expect the vendor for the software has specified folders / files / etc. that must be exempted from the scan as well (vendors for process stuff such as Yokogawa, etc. specify that). Seems to be a configuration failure on the part of the facility.

Re:So was this out of spec?

[Anonymous Coward]

I'm wondering why that machine has A/V software to begin with, because SURELY this critical computer wouldn't have network connectivity....

Re:So was this out of spec?

[DarkOx]

To be fair on access scanning can be really harmful to performance and in many cases cause crashes and other I/O problems with applications that do heavy I/O with large files.

AV is basically only a valid approach to security in situations where high availability is not a hard requirement.

Re:So was this out of spec?

[dbIII]

Clearly there was no spec and it was a home computer used for a critical purpose - no design just duct tape and dust.

Re:So was this out of spec?

[Anonymous Coward]

It's in a hospital. Duh...

Re:So was this out of spec?

[jfdavis668]

The device was hooked via a serial cable to a PC to record the logs during the procedure. The PC antivirus ran, and locked the log file to scan it, causing the medical device to crash. Yes, that is a very bad way to design a machine.

Re:So was this out of spec?

[Anonymous Coward]

oblig:
https://xkcd.com/463/

Clearly, someone is doing their job horribly wrong.

Re:So was this out of spec?

[Desler]

Because malware can get on to computers via other routes than the network? Never heard of USB drives before?

Re:So was this out of spec?

[Overzeetop]

That's what they make epoxy for.

Re:So was this out of spec?

[fluffernutter]

You think hospitals have teams of admins do go around and keep the equipment configured?

Re:So was this out of spec?

[Anonymous Coward]

Because even critical computers at times have USBs plugged into them or CDs dropped into their HD to get updates.

Ever hear of Stuxnet? Completely isolated from the outside world, still ended up virused.

Re:So was this out of spec?

[Anonymous Coward]

"should be highly controlled".

It was. Someone in the hospital CIS(O)/ IT team built a Compliance document, someone flet scanning for problems every hours was the best idea, and that became the compliance regimen that the hospital has to follow, enforce through GPO push, MDM, and the like.

Each hospital builds their own compliance document, each one is different, and many are built by people who don't really understand what is going on with the technology- whether that's the physicians that are on the compliance review team that don't understand the tech, or the IT guys that don't understand how the tech is used in the practice of medicine.

Re:So was this out of spec?

[Anonymous Coward]

On access scanning is does impact performance nearly the way it used to. I have a large number of users with on access scanning turned on, and if I get a slowdown complaint it is almost always localized to one app or webpage and unrelated to antivirus. That and I blame the T1 to remote offices a lot.

Re:So was this out of spec?

[known_coward_69]

this is a doctor's laptop we're talking about. they go OCD crazy if it's not exactly how they want it to be

Re:So was this out of spec?

[Billly Gates]

If you change it and someone dies that is on YOU!

It is best not to touch it because the lawyers can sue the vendor. It is a mess and whoever touches it last is the one with the axe to the head as much as your intentions are

Re:We've got some new features to get excited abou

[Mr D from 63]

This seems like another example of the cure being worse than the disease.

Scanning for viruses during heart surgery

[ZipK]

It just writes itself.

Sometimes 'antivirus' is a warning sign...

[fuzzyfuzzyfungus]

Antivirus systems aren't useless(I wouldn't trust their 'disinfection'; but they at least catch people reusing obsolete exploits and sometimes provide warnings that something is amiss); but this is one of those situations where hearing that antivirus software is running is a giant red flag: it usually means that a full-fat desktop/server OS with a network connection and who-knows-what-else running on it is doing the job of a dedicated computer. Quite probably being allowed to retain state over time except for the ever so occasional re-imaging. That just isn't going to go well. Even if your application needs full Windows whatever for some reason, there are plenty of ways to keep it on a much tighter leash than just shoving a desktop at the problem and hoping Norton can save you. If a system is contained by the network so that it can only talk to the external hosts it absolutely needs; and is booting from a clean, static, image every time(with all changes discarded after any data generated during the session are moved elsewhere) you are a great deal safer.

Seriously?

[rlp]

Why would anyone use Windows for a real-time critical application? There are small real-time OS's designed just for such applications.

Re:Seriously?

[omnichad]

If anything, blame the FDA. They would have had to approve the software that requires this configuration.

Re:Seriously?

[jfdavis668]

The machine didn't use Windows. It was hooked to a PC to record the logs during the procedure so the doctor could review them later. The AV software locked the log to perform the scan, and the medical device crashed. They had to reboot the PC to keep working.

Re:Seriously?

[OzPeter]

The AV software locked the log to perform the scan, and the medical device crashed. They had to reboot the PC to keep working.

While not expected, this sort of failure (logging device unavailable) IS predictable, and IMHO should;t have caused the crash in the first place. I'm a big fan of a "broken windows" style of software assessment. If I can see something broken (no matter how trivial), it makes me start to wonder what other things could be amiss that I can't see.

Or to give a car analogy. If I have a car with mis-matched seams between panels, I'm going to be worrying about what other parts of the car have been treated with the same (low) level of care.

Re:Seriously?

[Anonymous Coward]

The machine didn't use Windows. It was hooked to a PC to record the logs during the procedure so the doctor could review them later. The AV software locked the log to perform the scan, and the medical device crashed. They had to reboot the PC to keep working.

Which means that the device's software had insufficient error checking. There are plenty of other reasons why it might be unable to write a log file. What if the doctor's PC reported disk full and refused to write the log? Would that cause a crash? Or if the hard drive failed in the PC? That would likely require more than just a reboot to resolve it.

Come on people. If you're writing software that lives will depend on, you *have* to make it absolutely bulletproof. You need to be able to handle these kinds of error condition gracefully and without crashing.

Yes, there is an issue that the virus checker locked the file. And frankly running a virus scan every hour? I bet he also complained regularly of his PC running slowly. But the real clanger here is the fact that life-critical medical software crashed hard simply because it couldn't write what it wanted to an external file system.

Re:Seriously?

[Overzeetop]

So it WAS a flaw in the medical device. An error thrown to indicate logging was non-functional is the proper result for a remote file being locked, not a system crash. Who QA'd this device, Microsoft?

Re:Seriously?

[gweihir]

What kind of messed-up device. This has been solved for ages. First, if the logs are critical, make a local copy. And second, if you send them off, use UDP so that network failures or failure of the remote logger does not block anything on the local machine. You know, like rsyslogd. But I guess this is just another example of cheaper-than-possible "programmers" at work, the kind that does not understand system administration or networking.

Re:Seriously?

[Anonymous Coward]

They're lucky the reboot didn't include updates

Re:Seriously?

[Anonymous Coward]

Also, I don't remember the last time I saw a AV scan lock read/write access to a file.

Re:Seriously?

[Anonymous Coward]

From friends in the medical industry, my bet is that this was solved many, many years ago....and that software is still going through testing. Often times there's dozens upon dozens of bug fixes that sit in the queue forever, until something is big enough for the manufacturer to put something in the pipeline and pay for the time and cost of the review. They probably just didn't test files getting randomly locked while they had them open for writing, and then implemented the fix that day when they figured it out. Getting the fix through FDA and onto the devices is quite the chore...

Re:We've got some new features to get excited abou

[menkhaura]

I see what you did here, tehehehe

LASIK computer

[Anonymous Coward]

When I had LASIK back in, oh, 2011, the computer that controlled the laser was running Windows 95 (I'm not kidding, at all).

Crap like this

[Anonymous Coward]

It's why I was so happy to ditch Windows when Apple went Intel. Now 10 years of no virus scans interfering with my work. Hallelujah!!!

Re:Crap like this

[Anonymous Coward]

I've been using Windows since 1994 and I never have used anti-virus software and still don't. If you don't download stupid shit and run it, and don't use shitty browsers, you won't have a problem. Every couple of years someone will freak out and insist I'm compromised without knowing it, and I let them run an offline scan ... nothing every time.

Re: Crap like this

[Anonymous Coward]

You realize that there's a built in scanner that you can't choose to disable easily on your computer, right?

You also realize that *nix/bsd also has viruses, right?

Re:Crap like this

[sqlrob]

Name a browser that hasn't had a vulnerability that can be used to install malware (Hint: even Lynx as had them)

Re:Crap like this

[Anonymous Coward]

Antivirus scans (manual and automatic) were just part of it for me. So was the need to reinstall Windows on just about an annual basis. So were driver compatibility issues. And endless updates. Windows was and is a mess. You go ahead and kid yourself that your struggling with Windows is a valiant, normal part of living--it's not. You're completely deluded.
Now go away, please. Any mention of Windows merely reminds me of all the on-hours, off-hours and vacation time I've blown performing mindless chores. I don't want to hear your latest horror stories.

Doctor's Computer? Really?

[unixcorn]

"Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors' computer. "
I seriously doubt the computer was owned by the doctor. More than likely, it was procured, set up and managed by a team of IT specialist at the hospital/clinic who know little to nothing about the software that might be running on it. Likewise, if the company supplying the software isn't providing a dedicated, hardened box to run the software on, they share the blame as well. Or, I have seen dedicated boxes with all kinds of crap loaded on them by operators who had no clue what the consequences might be. The bottom line here is that maybe computers should be kept out of the operating room. Or maybe doctors shouldn't be allowed to use them.

Re:Doctor's Computer? Really?

[Anonymous Coward]

I don't think they were implying the computer was actually owned or setup by the doctor, just simply pointing out it was the comptuer that was being used by the doctor for the procedure.

Re:Doctor's Computer? Really?

[thegarbz]

Hardly. A lot of medical devices come with nothing more than a piece of software that runs on the "doctor's computer". Given this was nothing more than a system that logged data during a procedure and the fault was more due to a poorly coded medical device it seems unlikely that it was dedicated hardware setup by a dedicated team.

Re: We've got some new features to get excited abo

[Anonymous Coward]

Kind of like a rectum exam to look for an enlarged prostate... Most men will die with prostate cancer than from it... No quack is pulling out a kernel of corn from my arse just to look for something that will most probably go to the grave with me. Unless she's a hot nurse, in that case she can finger bash my rim

Was the patient a politician?

[MiniMike]

If the patient was a typical politician, maybe this was actually a divide by zero error?

Real Time Operating System

[Anonymous Coward]

Who did not specify the use of a real time OS for critical equipment?

What the Fuck

[Anonymous Coward]

Why is antivirus needed on an air-gapped machine that doesn't accept USB devices or autorun? On these types of systems we use a system like Deep Freeze. Every reboot erases all changes made in the previous session. It essentially restores a system image every boot.

Re:What the Fuck

[Anonymous Coward]

Alot of these new medical devices are networked on the hospital's LAN because they populate the patient records with the data they collect.

Re:So was this out of spec? Yes. Aggregate system.

[Anonymous Coward]

The connection of the second device to the first made an aggregate system. I doubt the aggregate system was tested to the full required medical equipment specification.

rofl

[Anonymous Coward]

windoze? of course it's windoze! fuck you all who uses that shit in mission critical things! fuck m$.

Thanks a lot ...

[Anonymous Coward]

Obamacare

So, the computer is connected to the internet?

[angel'o'sphere]

Really?

Why anyone would put anti virus software on a computer that is isolated from the net, has likely all USB ports disabled etc. is beyond me.

Make the damn boot drive read only, put the data on a different drive/partition ... then you can even keep USB and DVD reader/writer accessible.

Just don't put a windows PC into any network unless you really knwo what you do.

General purpose OS not suitable for critical use

[QuietLagoon]

At some point, the developers of computers that are used in critical situations (medical operations, battleships, etc) will soon realize that it is to the detriment of their end users to use a general purpose operating system for systems.

.
It is easy to fall for the siren-song hype from the marketeers that the general purpose operating system is up to the task (remember Microsoft's marketing push that Windows CE was a real-time operating system ,even though it wasn't?), and that being able to use their knowledge of Windows is a benefit that will make their system better.

Whether it is a weather application being used on live television, or a computer being used in an operating room, Microsoft has shown that Windows is not a proper steward of serious systems programming.

And on the bright side...

[tekrat]

It didn't try to update to Windows 10 in the middle of the procedure!!!!

I swear there has to be an international body that can declare Windows as a virus that must be eliminated from the planet before humanity can move forward.

Logfiles?

[Khyber]

Whatever happened to a simple audio log? We've got recorders that encode directly to MP3. Just make the recording and copy it into patient files after the end of procedure.

This 'do everything with a computer' mentality is exactly why we have these nonsensical issues happening in the first place.

Why?

[Anonymous Coward]

Why was a medical instrument dependent on a PC for a primary part of its function!? I can understand outputting to a monitor or some other piece of PC related hardware, and maybe logging data on a PC for later review (which sounds like part of this devices function) but running one of the devices video functions through a PC software bundle? Foolish is an understatement.

Re:Why?

[gweihir]

In my experience, a design-flaw this fundamental is due to coders that do not understand system administration and networking and have no clue about the failure-modes to be expected. Quite common these days.

Re:Why?

[Anonymous Coward]

Halt and catch fire is acceptable if some bone-headed policy says to install antivirus on everything and configure it to scan every single hour. In fact, the sooner it crashes in these instances, the sooner the solution is found.

What if..

[fluffernutter]

O M G.. Can you imagine if windows update and antivirus ran at the same time? The world would explode!

Re:What if..

[Livius]

I'm imagining an infinite loop in which a PC which upgrades Windows and then the antivirus software correctly identifies Window as a virus and rolls back the 'upgrade'.

Damnit, it is a MEDICAL INSTRUMENT!

[kheldan]

I used to work for a company that built ophthalmic ultrasound machines. It was Windows based (unfortunately). IT departments, being who they are, wanted to put things like antivirus on it. Then the doctors would complain that the MEDICAL INSTRUMENT wasn't performing as advertised. They send it in to us for 'repair'. We remove the shitty antivirus (and all the other crap that IT guys would install on it), then it works perfectly again. We return it.. and IT guys would screw it up again. Rinse, repeat, ad infinitum.

MEMO TO IT GUYS: Stop treating medical instruments like they're desktop computers! Find another solution, or AT LEAST be smart about how you're installing your junk on it, IT IS A MEDICAL INSTRUMENT, DAMNIT!

Re:Damnit, it is a MEDICAL INSTRUMENT!

[gweihir]

The stupidity of some IT people is staggering. We had one case where they put AV on a highly isolated system and then had to compromise its isolation to allow over-the-net updates. When we told them that the system was not isolated anymore and that at the very least the AV vendor could now attack them over the network, they did not even understand what we were talking about. They mumbled something about "all machines must have AV".

Re:Damnit, it is a MEDICAL INSTRUMENT!

[kheldan]

They mumbled something about "all machines must have AV".

That's pretty much the long and the short of it, yes. They don't seem to understand that it's primary function is as a medical instrument, and that compromising that may compromise the health or even the life of a human being. I'm surprised the FDA doesn't get more involved in things like this, since there is extensive testing of any medical instrument before it is allowed to be sold in the U.S., and especially so in the case of anything computerized. Of course I've always thought it was absurd that any medical instrument (or measuring instrument -- Tektronix oscilliscopes run Windows!) would have any version of Windows running on them, too. Most would be better off running Linux tailored for the specific application, which would also more or less preclude the possibility of virus or malware infection in the first place.

Re:Damnit, it is a MEDICAL INSTRUMENT!

[Anonymous Coward]

I used to work for a company that built ophthalmic ultrasound machines. It was Windows based (unfortunately). IT departments, being who they are, wanted to put things like antivirus on it. Then the doctors would complain that the MEDICAL INSTRUMENT wasn't performing as advertised. They send it in to us for 'repair'. We remove the shitty antivirus (and all the other crap that IT guys would install on it), then it works perfectly again. We return it.. and IT guys would screw it up again. Rinse, repeat, ad infinitum. MEMO TO IT GUYS: Stop treating medical instruments like they're desktop computers! Find another solution, or AT LEAST be smart about how you're installing your junk on it, IT IS A MEDICAL INSTRUMENT, DAMNIT!

MEMO TO MEDICAL INSTRUMENT MANUFACTURERS: Stop treating medical instruments like they're desktop computers. If you're looking for life-or-death stability, you should fucking know better than to run Windows.

Common Sense dictates putting antivirus on a Windows machine if you're going to network the damn thing, so big surprise you've found IT departments doing this. They already KNOW medical instrument manufacturers didn't have security in mind when they fucking put Windows on it, so of COURSE there's not even a basic firewall running. Automated updates are also commonly out of the question for devices like this, because "stability".

Re:Damnit, it is a MEDICAL INSTRUMENT!

[Anonymous Coward]

If it's got a desktop OS on it then it gets treated as a Desktop, unless during the SALE of the device you mention that it requires no AV or Updates and requires segregating from the rest of the network

Most of the suppliers seem to mention this either right at the end of implementation or at worst after it has first broken down.

Re:Damnit, it is a MEDICAL INSTRUMENT!

[Mr. Shotgun]

MEMO TO IT GUYS: Stop treating medical instruments like they're desktop computers! Find another solution, or AT LEAST be smart about how you're installing your junk on it, IT IS A MEDICAL INSTRUMENT, DAMNIT!

To you it is a medical instrument, to would be attackers it is yet another unpatched windows box. How IT views something is not the same as how you do and if the organization's perimeter is breached your medical instrument might be a low hanging fruit for exploitation. Granted after exploiting it they may not find anything particularly useful on the device itself but they might find credentials to use elsewhere in the network. You have to remember cases like Stuxnet where the worm targeted Windows boxes used for running centrifuges, everyone else saw centrifuges but the attackers saw windows.

Now I am not defending the IT guys, if the system cannot have antivirus on it then it cannot have antivirus on it. The proper solution would be to keep the device off the network or isolate it inside it's own network away from everything else. They should be able to work with systems that cannot run AV and design security around that, not slap AV on everything and call it secure.

Re:Damnit, it is a MEDICAL INSTRUMENT!

[gweihir]

Read my sig

Re:Damnit, it is a MEDICAL INSTRUMENT!

[Rozzin]

The stupidity of some IT people is staggering. We had one case where they put AV on a highly isolated system and then had to compromise its isolation to allow over-the-net updates. When we told them that the system was not isolated anymore and that at the very least the AV vendor could now attack them over the network, they did not even understand what we were talking about. They mumbled something about "all machines must have AV".

Have you asked them if they approach their kids' schooling with the same keen sense of security?

why are they using an insecure OS?

[lkcl]

why are they using a general-purpose OS, supplied by a company that's known not to care about security (because it costs money and profit), for *life* saving mission-critical software? i don't understand.

Re:So was this out of spec? Yes. Aggregate system.

[some old guy]

I'm sure it was tested...badly. Looks like an epic fail of cGMP validation.

Better antivirus than antivirus?

[Anonymous Coward]

APK Hosts File Engine 9.0++ SR-4 32/64-bit http://www.bing.com/search?q=%...

Antivirus SLOWS YOU DOWN & has tons of exploits (Tavis Ormandy anyone?)!

Hosts add speed (hardcodes/adblocks), security (bad sites/poisoned dns), reliability (dns down), & anonymity (dns requestlogtrackers) natively. Hosts != blockable by ClarityRay (like. souled-out to admen inferior wasteful redundant slower usermode browser addons)

Less power/cpu/ram + IO use vs. DNS/routers/antivirus + less security issues/complexity. Compliments firewalls (w/ layered drivers blocking less used IP addys vs. hosts blocking more used domains) & DNS (lighten dns load). Gets data via 10 security sites.

Works vs. caps & HTTP PUSH ads w/ firewalls.

* Ads rob bandwidth/speed paid for, security (openbid adnetworks abuse), privacy in tracking + anonymity.

APK

P.S. - Safe https://www.virustotal.com/en/... (Verified by Malwarebytes' S. Burn "I've seen the code & yes it is safe" http://forum.hosts-file.net/vi... )

Most of the time RTOS not needed

[bangular]

For equipment like this, it's pretty common that the PC does very little and an RTOS is probably not necessary. Anything important or real-time is done on the equipment itself and the PC is just a dummy terminal. If the PC goes haywire, a watchdog timer probably puts the system into a safe state. I think that's why it's so common to use Windows.

I'm not saying it's right. In fact, it's all the more reason to use as simple and streamlined OS as possible. Something like a customized RHEL that only has enough installed to run QT.

So the manufacturer knew about the problem

[raymorris]

What that says to me is that the manufacturer knew about the problem and shipped it anyway. The usual and customary practice with Windows systems, especially older versions, is to install anti-virus. On critical systems, anti-virus would be considered best practice and not installing AV could be considered reckless. The manufacturer knew that protecting the machine in the ordinary manner would endanger patients and they did nothing to either alleviate the danger (don't CRASH just because an AV scan is running) or prevent it (don't provide administrator access to the OS on a surgical device).

Re:General purpose OS not suitable for critical us

[Grishnakh]

At some point, the developers of computers that are used in critical situations (medical operations, battleships, etc) will soon realize that it is to the detriment of their end users to use a general purpose operating system for systems.

It doesn't matter; the developers have no input about the OS to be used. That decision is made by management.

Design insanity

[Pyramid]

Who in their right mind designs life-critical systems around off the shelf operating systems like Windows? There's a reason aircraft computer systems are custom and highly redundant. Medical equipment of this caliber is no different.

What company produced this system? Their accreditation should be revoked.

Re:Design insanity

[gweihir]

It is a question of cost. A Linux/QNX/BSD coder is expensive. A Windows coder is cheap, hence more profit.

Re:Design insanity

[Anonymous Coward]

A Linux/QNX/BSD coder is going to take forever, bring up pointless things, and all 3 of them are probably already employed elsewhere on some other feel-good project copying Microsoft or Apple OS functionality.

Knowledge

[phorm]

And who is going to know how to do this? You think the guy doing heart procedures knows how to configure the antivirus to that degree, or that the guy setting up the antivirus knows where the heart dude's proprietary software is saving the data files?

Hell, even as a SysAdmin I don't always know 100% what my software is doing in the background and can't account for the crazy shit my AntiVirus has done.

Not only that, but what's the point of having an antivirus if it's not scanning the locations most likely to to be changing. I'm assuming that "vulnerable files" means the OS in this case, but if those are infected your system is already owned and the first thing any good virus does is break the AV...

Re:General purpose OS not suitable for critical us

[Registered Coward v2]

>

Whether it is a weather application being used on live television, or a computer being used in an operating room, Microsoft has shown that Windows is not a proper steward of serious systems programming.

Heh. Go to any major airport with nice big screen monitors showing flight information and some percentage of them will have Windows dialogue box on them informing you of some problem...

maybe

[slazzy]

It found a human virus?

Now doctors will need full admin rights + app admi

[Joe_Dragon]

Now doctors will need full local admin rights + app admin rights to turn off anything that may get in there way. In some settings (more likely with poor IT in place) may even need domain admin rights so they can over ride / block GPO's.

Lack of a good OS for devices.

[jellomizer]

Granted if I were to design a medical device I would probably just use a stripped down version of Linux that just does what it needs to do.
Windows, OS X, and Linux with a full distribution on such a device is adding extra complexity where it isn't needed.
That is why a lot of these devices are still running off of DOS.

The coded application should be the star of the device not the OS which had to be configured to get out of the way,

Such a device should only have port 22 for Administration and updates, and whatever port needed to receive HL7. A Virus shouldn't have any clear path into such a device as it should be locked down.

Need for Virus Scan?

[DarthVain]

Further to this, why do they need a Virus Scan anyway? What is a device like this doing connected to the Internet or even their Internal network?

Re:General purpose OS not suitable for critical us

[Anonymous Coward]

>...not a proper steward...

I really, really like the way you frame this. The proverbial 'we' use products for whatever we're doing. And it IS companies' responsibilities to be less near-sighted and consider their audience.

not to install any software on it = no updates

[Joe_Dragon]

not to install any software on it = no updates and that is bad as well when they get hacked with something that was fixed months ago in an os update.

How about not being online? At the cost we pay why should the system need to link to some E-doctor in el salvador?

Re:not to install any software on it = no updates

[Registered Coward v2]

not to install any software on it = no updates and that is bad as well when they get hacked with something that was fixed months ago in an os update.

No, it means we control any changes to the configuration. We pull the data sets, move them to a new drive and test run the setup to ensure it works. Then we go to the server, pull the old drive, install the new one and reboot into the updated setup. Quite simple.

Re:General purpose OS not suitable for critical us

[Anonymous Coward]

Our office coffee machine runs Linux. Gotta make sure that coffee is flowing.

Don't laugh. Anti virus scans are important.

[140Mandak262Jamuna]

The surgeons should be happy and thankful for these automatic anti virus scans done by these machines. Come on, people, there is a patient on the gurney with chest cavity open, exposed to all sorts of pathogens in the atmosphere. Fungus spores, pollen, bacteria, virus all sorts of things want to get into the body and wreak havoc. It is a good thing, there are machines to do real time anti-virus scans. People complaining about it, being smug and superior over Microsoft, what the hell?!

short-sighted cost cutting

[Anonymous Coward]

A similar incidence occurred to me over 10 years ago. I was using a $200K+ testing equipment running an overnight test. I started the job Friday evening. On Saturday morning, when I logged back on, instead of good progress, there were only a few data files and the equipment had rebooted. I rushed to office and realized that an overnight anti-virus upgrade had forced a reboot: the testing SW was written on top of WinXP. Fortunately there was no real damage: the equipment had to be re-calibrated (took 30 min) and I had to restart the job. However the irony is that a $200K+ equipment was controlled by a $100 piece-of-sh*t from MS*T. The equipment vendor had their own OS/GUI etc. But at certain point, they decided to use the crap from M**T. It's true that many equipment, SCADA, medical equipment need an OS with GUI, file system and internet. However a shot-sighted decision to pick up something cheap can mean read dangers down the road.

Remember folks: open source is "dangerous"!

[Anonymous Coward]

... because anyone could look at the source code!

Running the wrong scan

[Anonymous Coward]

They should be running an antibacterial scan :D

A IBM company?

[Anonymous Coward]

Why on earth didn't the parent company push their own mission critical solutions on the client side?

Re:General purpose OS not suitable for critical us

[thegarbz]

I too am looking forward to the days where people can .... read the details of what happened and stop blaming the underlying OS for a poorly coded program connected to a poorly coded device.

I actually wish that this computer ran Linux. Then maybe we could have a proper discussion about the coding processes used in this specific application rather than the frothing at the mouth while ranting about Windows that every modded up Slashdot post here has become.

But sure carry on writing in bold without having a clue what you're talking about

Re:General purpose OS not suitable for critical us

[Anonymous Coward]

The FIDS systems at the airport are not exactly critical to the safe operation of the Airport.
Stop trolling.
All critical computer systems are flight certified an there are NO windows systems allowed. MS will not support any use of Windows where there could be risk to life.

Disclaimer. I work on the Systems that run airport terminals. These are not flight critical systems and so there is plenty of Windows around (sadly). IT is a Piece of crap but the suppliers very seldom want to use anything else. Windows 10 is at the moment a total no go area. Far too many bits of kit that we have to work with just won't work with that pile of steaming dog poo.

Convenience comes at a price

[Anonymous Coward]

I think an easy solution to this problem, is to make sure the machine that is collecting and performing the process is a standalone machine that is not somehow connected to a workstation/laptop, no matter how nice the idea to add convenience to the doctor(s) might be. When you are investing in a machine that is needing to be relied on during high-risk situations, the one thing you don't want, is for any potential interference of any sort. That means it needs to be non-networked, and have an entire operating system and API to do the entire task right on the one machine. Non-networked is an inconvenience, yes... but it makes sure there's no chances of any malware or any bad updates to corrupt the system and bring it down to its knees. Then, possibly the data could be transferred in a one-way method (never back to the machine) via an SD card, to allow the doctors and technicians to gather all the diagnostic data and present it to the patient and insurance agency.

Poor Host Isolation

[EndlessNameless]

The "need" for antivirus can be greatly diminished if these hosts are simply isolated properly.

If they want it on the domain for manageability, fine. Allow Kerberos/LDAPS/CIFS to domain controllers and Kerberos/CIFS to a file server for data transfer. Run antivirus on the file server.

Block everything else if you can, but make absolutely sure to deny HTTP and SMTP.

Put the damn things on their own subnet and enforce the restrictions via network ACLs so even the "clever" users can't disable it.

Medical and industrial equipment vendors have zero interest in making sure their tools work properly on a normal enterprise workstation. So give them their little sandbox, and keep them as far away from the business network as possible.

And this...

[TheGreatMcCluck]

complete fucking insanity is why IoT needs to be stopped. Here's a fucking crazy thought: don't connect complex medical equipment that is involved in keeping people alive during medical procedures to the internet at all. Ever. Under any circumstances. Use AV to pre-screen shit before you install it. While I'm on the table in a hospital in Canada, cut wide open, I am pretty damn sure no one in Finland needs to know what's up with the machine monitoring my heart. And I know my doctor doesn't need to be looking for kitten videos. Jesus Fuck when will these people get a damn clue?! THIS SHIT is why I don't want a self-driving car; it's not because I don't like the idea, it's because I don't trust an auto maker to get it right. They might be flawless for some time, and then... then some fucking moron is going to push an on-air update while I'm up in the mountains with no guardrail and off I go. Let's leave the internet to the stuff that can't kill us, PLEASE.

Re:And this...

[tibit]

No, IoT is fine, but the execution here was just dumb. I run embedded windows systems in hard realtime applications where even if, somehow, through an exploit of some sort they got owned, the most damage would be to a ram disk holding transient data. You could press the "data reset" button to reboot the system and it'd be back to the factory state, as it boots and operates from read-only media with a ram disk to hold changes to the filesystem.

I think i know which security product to blame

[Anonymous Coward]

Must've been McAfee.
The only security products whose authors want you to die a painfull and horrible death.

That's just stupid :(

[tibit]

Who the fuck uses a non-embedded version of Windows in a fucking invasive medical device *anyway*? It's almost absurd. I use embedded windows in several hard-realtime control systems with 250us cycle times (hard deadlines: you're late and and some expensive metal chunks crash into each other), and it works just fine... Someone dun goofed big time. These systems use built-in firewalls, are not updated willy-nilly, run only necessary services, and the software load is considered to be like firmware: no user and no IT department can mess with it. The OS and our software is running off read-only media anyway, and writable overlay is on a battery-backed, crash-surviving ram disk. This shouldn't be any different in a catheterization system. Once the procedure is done and the data synced with central server, the ram disk should be reset.

Medical staff to blame and not Microsoft Windows

[khz6955]

"the whole incident was nothing more than an oversight on the medical unit's side"

What if the unit had started to record erroneous date in the middle of the operation, would that also make an oversight on the medical unit's side. seriously, what the fuck is Windows even doing in an Operating Theater.

I Know This One!

[davesays]

Never thought I would get to say that on /.! I work in a hospital and set up PCs for Cath-lab, surgical anesthesia monitors and stuff all the time. As far as this goes the hospital usually just buy the seats (or whatever) for the software and you install it. All the software like this comes with explicit and unambiguous instructions for directory and file type exclusions. Right or wrong, the software will run fine if this is done correctly - it will crash if it is not. First, all the posters are right, windows is crap and shouldn't be used. But it is chicken and egg - the hospitals buy it because that is what all the vendors write their apps for, and the vendors code for Windows because that is what hospitals buy. I am stuck with it. There are a few vendors that run Linux end-to-end, but it is rare.

Ehvy Dense

[Tablizer]

Finally, I have proof for the PHB's that McAfee kills!

Have always said

[Anonymous Coward]

Turn drive scanning off, it's a resource hog and completely ignores the user. More to the point, if an infection has worked it's way onto the system, bypassing the active resident scanner, do you think the drive scanner is effective at this point?

Why not air-gapped?

[Anonymous Coward]

Why does this need to be connected to a network?

If ...

[cwsumner]

Running Antivirus or any other unneeded software on a critical medical computer is Wrong.

Connecting a critical medical computer to any network, that would make Antivirus necessary, is Wrong.

Having the Medical software crash because it could not access data is Wrong.

"If Engineers built buildings the way Programmers write programs, the first woodpecker that came along would destroy civilization!"

Beware, it is not going to be long before companys, and even individuals, will be sued for things like this. It already happens in other fields.

WTF are you using Window in Mission Critical apps?

[Anonymous Coward]

The obvious question is who if the irresponsible party who either built or bought a mission critical app that ran Windows????

the app crashed spectacularly

[peawormsworth]

the app...

An app? So it was running on a mobile phone? or tablet

... crashed spectacularly

The crash was spectacular... so how was it different or spectacular? In what way was this so much more special than a regular program crash?

Outsourced

[Anonymous Coward]

Outsourced programming