Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
If a software maker is found negligible and convicted of manslaughter (unintentionaly causing death) due to buggy software, would that void out the whole EULA business since they all claim they can't be held responsible? Or would the burden pass on the poor chap that used it for being irresponsible enough to use something where the maker couldn't be held accountable? Lets's face it, why are only software companies able to make themselves free from accountability when every other industry has to design for it?
Yes it can.
as we see technology inch into out society more in the medical field we will see more incidents of things like this happening. I know many hospitals are moving to computer based medication system, records, etc. A programing error could easily kill someone.
With this happening i think people wouldnt have any issue on arguing how programers need to be accountable for thier programs. So if they should be held responsible, should people who program other things like operating system be accountable to flaws in their programing?
30% Troll, 50% Underrated, 10% Interesting
Score:5, Troll
the therac-25 actually injured a fair number of people in the US 10-15 years ago. yeah, software fucks up sometimes. it's old news. for the article:
Nancy G. Leveson and Clark S. Turner. An investigation of the Therac-25 accidents. Computer 26, 7 (July, 1993) pages 18-41.
-ninjaneer
The software is only one piece of the puzzle, of course. Its killing was enabled by the hubris of its developers and the blind trust of its users.
Not by itself, no.
An autopilot that is consistently 1000 feet off, a poorly written control routine for an MRI, miscalibrated antilock brakes...can certainly cause death.
But ultimately, it comes back to whoever wrote it. Or specced it. Or tested it.
Software by itself is benign.
Human implementation of it may be lacking, though.
Can negligence in any area kill? Yes.
Software is no different from hardware in this aspect. If it is handling mission-critical or potentially-lethal equipment... great care should be taken to ensure its integrity.
Trusting those that make your irraditation software is no different from trusting the those that made your life-support hardware.
Human error, or mechanical, can mean death in both cases. If the error is glaring, it becomes a case of negligence.
Unfortunately in cases of software or even computer hardware operating environment becomes an often overlooked factor. Stress tests are needed... data collisions checked for, line noise, redundancy, etc. When we're talking about people's lives, that extra parity bit can be just as important as a backup-parachute...
There must be a point where software makers can no longer say "DISCLAIMER: IF WE BREAK YOUR MACHINE, IT'S NOT OUR FAULT." If you look at every piece of software's license, you'll see a clause like that. Imagine if every industry took that approach:
DISCLAIMER: IF YOUR CAR'S BRAKES FAIL, IT'S NOT OUR FAULT. TOUGH LUCK!
DISCALIMER: IF THIS MEDICINE KILLS YOU, OH WELL.. NOT OUR FAULT!
etc.
Some laws must be passed and software makers must be held accountable- they should no longer be able to hide under the big umbrella of the disclaimer.
Would an opensource:d project have been more likey to lack these bugs? I mean, compared to proprietary software.
Quidquid latine dictum sit, altum sonatur.
IIRC, Microsoft's license has had since day zero, a clause to the effect that you are not legally allowed to use their software to control nuclear reactors, medical devices, avionics or any other application that could endanger human life. THERE'S A REASON THAT'S THERE!
If you DO have such an application, the software vendor : 1) takes much greater care in design, implementation and testing, 2) carries a godawful ginourmous insurance rider to cover any such failure.
There is a segment of the industry that works in this niche and is well aware of the risks and how to best manage them. It's goddamned clueless PHBs that think Microsoft == Software that don't understand simple goddamned little nuances like this.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
I can't wait until they let software fly passenger airplanes. Remember the Airbus accident several years ago where the software took control of the plane during the landing sequence and flew the plane into forest past the end of the runway. The pilot was trying to pull up to abort the landing but the software was ignoring the pilots control motion because it was apparently programmed to ignore drastic stick movement during the landing sequence. Airbus's mentality was "the software knows better than the pilot". I don't recall if anyone (pilot or co-pilot) was killed in that incident or not. Boeing's philosophy here is that the pilot always knows better than the software.
The problem is that in every other development environment, the legal liability ultimately rests on the engineer who signed off on the quality assurance. But because software developers are not professionals and have no professional code of conduct, their signatures are meaningless. The only way software can become as reliable as other engineered products is to create the profession of software engineering*. And I'm not just talking about giving CompSci students a ring: many CompSci curriculums don't require any engineering techniques at all, and those that do usually devote less time to engineering than they do to sorting algorithms. The software industry requires fundamental changes, and legal liability is at most the catalyst.
* Yes, I know there are a couple of schools out there that offer SoftEng degrees, but until industry distinguishes them from CompScists and requires the engineering designation for key positions they are meaningless.
I agree. I posted the same above, should have looked down further and seen your post first. Heck, I'd have posted it anyway, Grub is funny sometimes, but I had to slap him down on this one.
I write controls software for automated test stands for fluid power component... testing things like valves, cylinders, pumps, motors and in one case, winches that can pull in excess of 150,000 Lbs force. I get chills thinking about the damage one little software fuck-up can do.
To re-iterate, the Therac 25 incidents should be required reading for all programmers.
The Therac
Come to think of it, the power company never warned me that all that electricity the sell me could be dangerous... it was here when I got the house. Maybe it's in the fine print on my electric bill, but I think most of that is how they can track me down to collect money from me if I don't pay...
Seriously - I think it depends on the software. If you use consumer grade software with a EULA that says "this software is good for nothing, use at your own risk"... well fine.
I seriously doubt the firmware for an X-Ray machine has a EULA like that, but then, I didn't RTFA. I am sure critical software doesn't come with the usual global-deny-all-responsibility clauses.
Two final thoughts
For somethings, who would you trust more? Dr. McFeelgood with his finger on a button and a stopwatch, or software control?
If you can't mathematically prove the software is incorrect (by source code inspection), how can you prove it was a particular software in a multi-software environment, an intermittant hardware fault or a cosmic ray flipping the wrong bit?
This issue is a bit more complicated than you think.
The next time you visit the doctor watch the workflow of the office staff. Increasingly chances are they will probably be entering your medical information, and I mean the clinical stuff, not your address into some type of computer system.
I currently work for a small Electronic Medical Records company. At some level I worry about potentially killing someone every day. In fact our bug tracking tool has a special category in it called "Patient Safety" which is the highest priority bug. We deal with things most of you probably wouldn't think of such as a tool for writing Prescriptions, which given the fact that many drugs interact ( potentially fatally) has to catch and alert the physician to such cases. I also deal with lab results which if reported incorrectly could lead to a potentially fatal decision by the doctor and so forth.
Consultants and pundits like to say that computer control reduces the chances of human error and failure, this is said IMO to comfort the masses. To state the obvious I suspect EVERYONE on Slashdot knows that in reality that statement is not true, the human error has just been moved to a different point in the chain. A tired programmer is just as likely to make a mistake as a tired machinery operator. The difference is that that software might be used by 5,000 machines, whearas that operator runs 1.
No, software cannot kill anyone. Only machines controlled by software can kill people.
Now, how to handle the legality or morality of said observation is beyond my level of interest at this time. However, I hope this clarifies things.
At the very least, these things confirm my general posit that "Computers should not be allowed to control things that move."
"There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
Again, from the article:
Every example given was life threatening, yet the author clearly wants you to draw the conclusion that a software author should hesitate to publish a program she wrote to perform a calculation because someone *who thought they knew what they were doing* might plug it into a lethal machine.Next we will be hearing about how someone wrote a spreadsheet in gnumeric to calculate the radiation dose, killed someone because of a bug in gnumeric, and how the authors of gnumeric should be ashamed of themselves, and not the asshole who *thought he knew what he was doing.*
Special Superior Deputy Prosecutor Cristobal Arboleda, unlike the author of the article, is accusing the right people and doing his job well.
They may not be the "code" that most of you know but it's an interesting story...
I've seen code on PLC's and embedded microcontrolers used in industrial safety applications that could have been lethal or at the very least, a finger eater. For about four years, I worked for a company that manufactured light curtains, hardguards and other devices that keep machinery operators from losing limbs.
The most dangerous "code" I've seen was on a large metal stamping machine. The maintenance department was using a PLC to take the anti-tie-down inputs and start the stamping process. The operators had tied one of the palm buttons down with black tape so they could hold the metal with one hand and start the process with the other. The PLC was also counting the inputs and cycling the machine to satisfy the counts whether the operators wanted it or not. Anyway, to make a long story short, our company was called in to install a light curtain and pitch all the other "safety" junk. The techs that mounted the curtain found a part of a hand behind the old physical guard.
Actually, if you check the link in the article, it explains all about the Patriot failoure. It was a "range gate error" caused by clock drift. The patriot was designed as a mobile anti-aircraft SAM and, as such, was never designed to run for more than a few hours at a time. The one at Dhahran had been running for over 100 hours. It was the Israelis who first noticed the clock drift problem and they reported it to Raytheon. The problem was caused by programmers who would "round off" the clock increment before storing it in order to save a couble bytes of memory. This rounding error was inconsequential so long as the system was rebooted every few hours (which a mobile SAM on the move would do), but it could easily grow to cause huge errors if the computers were left running continuously, as they were on SCUD intercept duty. Raytheon's solution was to send out a warning followed by a patch to fix the error. Unfortunately, in classic Raytheon bumbling style, the warning was "'very long run times' could affect the targeting accuracy", with no indication what "very long" was, or how much it would affect accuracy. The Alpha battery at Dhahran only ran so long because the Bravo battery was having radar trouble and Alpha was picking up the slack. The operators had no idea the range gate tracking was off by 600+ meters, otherwise obviously they'd have rebooted to fix it.
If a job's not worth doing, it's not worth doing right.
Amen. Therac-25 was the first thing that came to my mind too. As I understand it, the problem was a shared status variable was improperly handled on some sort of "abort" button, so that the machine would still be operating at full power even though most of the software said it was at low power.
In a former life ( :-> ) I was employed by a large multi-national that worked with utilities. Some of our software used SCADA protocols to remotely switch breakers - not household breakers, these switches control significant segments of the US power grid.
All the software and documentation contained numerous warnings, because if a utility employee manually switched of a segment to make repairs, and switch was remotely turned on, someone could be killed.
There are numerous other software applications that control (potentially) deadly devices - robots, industrial equipment, etc. Failure of the software, or problems with operator headspace, create a potential for death when working with almost any software that controls physical entities.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
We write software for railroad traffic control and crossing warning systems. If it fails we could end up with two trains trying to occupy the same piece of track at the same time (ref. Clapham Junction 35 dead) or gates that stay up when the train comes. Testing is very formal and rigorous and every step is documented.
For every hour we spend making sure the system does what it's supposed to do, we spend eight hours making sure it doesn't do what it's not supposed to do.
None of them can see the clouds; The polished wings don't care.
As tragic as it is, the Panama incident does not stand alone. In all, Baseline has found no fewer than a half-dozen cases in which software has contributed to loss of life.
I'm surprised that they only found a half-dozen cases in which software contributed to loss of life. Back in the old days when I used to subscribe to the comp.risks digest it used to seem that every couple months there was some fatality or near fatality that could be traced to flawed software. If anything, given the increasing use of embedded systems in our society, I would have thought that the fatality rate would have increased.
I believe no discussion of deadly software disasters is complete without mentioning the London Ambulance Service Disaster of '92. Basically, bad project management and other gaffes lead to the Londom Ambulance Service using a computerised dispatch system which was not up to the job. It promptly crashed, and quite a few people died as a result.