Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
Can Software Kill?
Certainly. A complete set of Novell manuals dropped from 40 stories up packs the same kinetic energy as a 10 car freight train moving at 80 km/h.
Trolling is a art,
Apparently it can only kill people in groups of only 28.
No.
Next story please, does it look like I have work to do?
Is 28 deaths the level at which we get concerned?
If software is outlawed, only outlaws will have software.
Don't blame Durga. I voted for Centauri.
One must be very careful when you kill -9!
If a software maker is found negligible and convicted of manslaughter (unintentionaly causing death) due to buggy software, would that void out the whole EULA business since they all claim they can't be held responsible? Or would the burden pass on the poor chap that used it for being irresponsible enough to use something where the maker couldn't be held accountable? Lets's face it, why are only software companies able to make themselves free from accountability when every other industry has to design for it?
Software can kill, just like any other stupid mistakes if left unchecked.
insert open source plug here
Software will only kill people through bad programming.
It is humans that make the underlying mistakes
Tonight on Fox...
WHEN SOFTWARE ATTACKS!
with host Mitch Pileggi
You are in error. No-one is screaming. Thank you for your cooperation.
Anyone who hasn't read this paper, should.
I've had this sig for three days.
... dumb programmers kill!
... but it can make the hardware controlled by it kill.
The Tao of math: The numbers you can count are not the real numbers.
the therac-25 actually injured a fair number of people in the US 10-15 years ago. yeah, software fucks up sometimes. it's old news. for the article:
Nancy G. Leveson and Clark S. Turner. An investigation of the Therac-25 accidents. Computer 26, 7 (July, 1993) pages 18-41.
-ninjaneer
The software is only one piece of the puzzle, of course. Its killing was enabled by the hubris of its developers and the blind trust of its users.
Not by itself, no.
An autopilot that is consistently 1000 feet off, a poorly written control routine for an MRI, miscalibrated antilock brakes...can certainly cause death.
But ultimately, it comes back to whoever wrote it. Or specced it. Or tested it.
Software by itself is benign.
Human implementation of it may be lacking, though.
Can negligence in any area kill? Yes.
Software is no different from hardware in this aspect. If it is handling mission-critical or potentially-lethal equipment... great care should be taken to ensure its integrity.
Trusting those that make your irraditation software is no different from trusting the those that made your life-support hardware.
Human error, or mechanical, can mean death in both cases. If the error is glaring, it becomes a case of negligence.
Unfortunately in cases of software or even computer hardware operating environment becomes an often overlooked factor. Stress tests are needed... data collisions checked for, line noise, redundancy, etc. When we're talking about people's lives, that extra parity bit can be just as important as a backup-parachute...
IIRC, the Patriot missile was never really designed or intended as an anti-missile missile, but a anti-aircraft (ie, a target much lower and slower) missile. It was only pressed into service killing Scuds because there was nothing better available.
So, wouldn't the Patriot missile failure be understandable due to it being used outside its original design? If the Patriot had been really intended and design as a missile killer, then yes, it should have a "failure" because it didn't live up to its original spec.
There's 10 types of people in this world, those who understand binary and those who don't.
A good book that tells how technology can cause death, destruction, and mayhem entitled "Set Phasers on Stun". Includes the Therac radiation machine accidents, nuclear accidents, and many other odd stories.
42 - So long and thanks for all the fish.
So are you saying they INTENDED to kill their patients and this software just did it more efficiently? ;P
Un-news
Bad programming can, just like guns don't kill, people do. An engineer makes mathmatical mistakes designing a bridge and the bridge later collapses, do bridges kill? Seems like a dedundent question, mistakes we make sometimes cost peoples lives, why would software be any different?
Quack, quack.
There was a good discussion of this event some months ago; the current issue has blurbs on topics ranging from computer viruses to aircraft cockpit management.
There must be a point where software makers can no longer say "DISCLAIMER: IF WE BREAK YOUR MACHINE, IT'S NOT OUR FAULT." If you look at every piece of software's license, you'll see a clause like that. Imagine if every industry took that approach:
DISCLAIMER: IF YOUR CAR'S BRAKES FAIL, IT'S NOT OUR FAULT. TOUGH LUCK!
DISCALIMER: IF THIS MEDICINE KILLS YOU, OH WELL.. NOT OUR FAULT!
etc.
Some laws must be passed and software makers must be held accountable- they should no longer be able to hide under the big umbrella of the disclaimer.
Sadly, this is nothing new.
Every software developer needs to read Peter Neuman's book Computer-Related Risks , and keep up with the Risks digest (comp.risks).
Learning from other's mistakes is much less painful.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
You see, if I'm a doctor, and I screw up and overdose you, it isn't a news item. I'll get reprimanded, maybe sued. No one will even notice if it happens many times, because each time it is a different doctor in a different circumstance.
But if I'm a computer software engineer and have a bug in a program that gets 3 people an overdose, then it will be noticed and much howling will be done over it. Even if the total number of errors have gone down, the type of error is new and there is a common factor between all the cases. And so we will complain.
And, I think, rightly. Computers are a tool, not to be trusted, always to be checked. I fear many people believe the computer can never be wrong (because it is so complex as to be indistringuishable from magic, and magic is never wrong) - perhaps this is why there isn't much howling about Diebold voting machines: It's digital, so it must be better!
Fellowship 9/11
IIRC, Microsoft's license has had since day zero, a clause to the effect that you are not legally allowed to use their software to control nuclear reactors, medical devices, avionics or any other application that could endanger human life. THERE'S A REASON THAT'S THERE!
If you DO have such an application, the software vendor : 1) takes much greater care in design, implementation and testing, 2) carries a godawful ginourmous insurance rider to cover any such failure.
There is a segment of the industry that works in this niche and is well aware of the risks and how to best manage them. It's goddamned clueless PHBs that think Microsoft == Software that don't understand simple goddamned little nuances like this.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
This is why I've always thought it's vitally important to have good, precise specifications in place and excellent quality assurance for any life-critical application. It's even better with many eyes overseeing every step of the process -- wait... that smacks of open source, doesn't it?
If you ask me -- and you haven't, but I'll tell you anyway -- what would be the best way to prevent catastrophe, it would be to PREVENT CHANGES TO THE SPEC. In college, our software engineering prof. gave us an assignment, then halfway through, she changed the spec on us. Well, not surprisingly, there wasn't a single project that worked faultlessly, and many of us were doing really well before that.
Software itself doesn't kill people. Bad software written by overworked developers writing to a constantly-changing specification with not nearly enough QA does. That is, people inadvertantly -- we hope -- kill people with software. Yeah yeah, it's cliche, but it works.
If it's not one thing it's your mother.
Good job with the Terminator images in everyone's heads.
Software does not kill. Bad engineering and poor implementation kills. My copy of Windows XP, while still radiating pure evil, has not managed to pop open the gun cabinet.
You might as well ask the question, 'can the old saddlebag gas tanks on Ford Rangers kill? Gasp!'
Microsoft Windows: A thirty-two bit extension and graphical shell to a sixteen-bit patch to an eight-bit operating system originally coded for a four-bit microprocessor which was written by a two-bit company that can't stand one bit of competition
-- author unknown
The problem is that in every other development environment, the legal liability ultimately rests on the engineer who signed off on the quality assurance. But because software developers are not professionals and have no professional code of conduct, their signatures are meaningless. The only way software can become as reliable as other engineered products is to create the profession of software engineering*. And I'm not just talking about giving CompSci students a ring: many CompSci curriculums don't require any engineering techniques at all, and those that do usually devote less time to engineering than they do to sorting algorithms. The software industry requires fundamental changes, and legal liability is at most the catalyst.
* Yes, I know there are a couple of schools out there that offer SoftEng degrees, but until industry distinguishes them from CompScists and requires the engineering designation for key positions they are meaningless.
Last time I checked, we don't have a bunch of kamakazi pilots for our Tomahawk Cruise Missiles. We make software to intentionally kill people all the time.
I was working for a desktop consulting company, and I was the only database developer there.
One of my customers wanted to convert a database, and originally I thought, no problem just convert some tables and redraw some forms.
It turns out this database was also going to store information about blood matching, transplants, and it would also calculate daily drug doses for the nurse to sign off on for kids getting marrow transplants. Success is measured in how many months the kid gets to live.
If I was working on a team using a more robust platform I might have had more confidence to push forward. However, this is Microsoft Access and i'm the only guy who would know how this thing would work. This means it would be very easy for some kid's death to point towards me.
So I quit.
By the way, if anyone has work for a database developer, feel free to contact me at will_spangler@juno.com. I'm quite good with MS Access.
The next time you visit the doctor watch the workflow of the office staff. Increasingly chances are they will probably be entering your medical information, and I mean the clinical stuff, not your address into some type of computer system.
I currently work for a small Electronic Medical Records company. At some level I worry about potentially killing someone every day. In fact our bug tracking tool has a special category in it called "Patient Safety" which is the highest priority bug. We deal with things most of you probably wouldn't think of such as a tool for writing Prescriptions, which given the fact that many drugs interact ( potentially fatally) has to catch and alert the physician to such cases. I also deal with lab results which if reported incorrectly could lead to a potentially fatal decision by the doctor and so forth.
Consultants and pundits like to say that computer control reduces the chances of human error and failure, this is said IMO to comfort the masses. To state the obvious I suspect EVERYONE on Slashdot knows that in reality that statement is not true, the human error has just been moved to a different point in the chain. A tired programmer is just as likely to make a mistake as a tired machinery operator. The difference is that that software might be used by 5,000 machines, whearas that operator runs 1.
If that Patriot missile failure counts as a "software kill" then surely software does kill; Look at the amount of people killed in Iraq for example by different types of bombs and cruise missiles that are guided (and detonated) by software.
Most people in the comments are focusing on actual bugs and crashes in a system causing deaths. While that could certainly happen, those types of errors are more visible and actually a much "better" error to have than some other types. If the system crashes, it may have some immediate effects depending on its purpose, but if it's something that causes its action through an actual user, they are generally harmless, though very annoying. An example of the difference is that if the software designed to run a ventilator has a bug that causes it to crash, since it is directly providing life to a person, when it crashes someone will probably die. On the other hand, systems designed to give information to a clinician, who can then act upon it are going to be very aware when that system is down, and so much less likely to make an error based on that outage.
The more insidious "errors" if you want to call them that are ones that are errors of design and process, and not execution. If a piece of software is designed with certain assumptions in mind, and something happens outside of the parameters of those assumptions, the software will appear to be working correctly when in fact there may be egregious errors. There are a lot of instances of this in everyday practice.
Lastly, what we run across is that clinicians are used to a world of paper, where everything obviously either there or not. You know that there's a problem, and there is transparency to the error, so you can factor that into your decision-making. In a clinical system, the transparency is not there, and a subtle flaw can mislead someone making a clinical decision into making a poor one.
Of course, the above are all gross generalities, as is any discussion of errors in complex systems, but I hope you get the idea.
A software glitch caused the crash of the first F-22 prototype (noone died fortunately) and as someone else pointed out, the Patriot missile failure in the first Gulf War (Software wasn't ENTIRELY to blame there. The bug was known and the folks in the field were given instructions on how to avoid it, but didn't follow them)
The trick is who do you hold responsible? The software person who has no training in mission critical software and who's working 80 hour weeks to meet the deadline the idiot managers are shoving down vis throat?
After 10 years in the industry I'm FINALLY starting to see movement towards software creation as a serious engineering discipline. Schools are starting to offer programs in Software Engineering, the ACM and IEEE have agreed on an official code of conduct (tho IHMO it still has SERIOUS problems), and most importantly companies are starting to listen to their technial folks when they say "You can't do that!".
Liability is just another incentive to head down that road. We need to be sure we pin the liability on the right folks.
In a former life ( :-> ) I was employed by a large multi-national that worked with utilities. Some of our software used SCADA protocols to remotely switch breakers - not household breakers, these switches control significant segments of the US power grid.
All the software and documentation contained numerous warnings, because if a utility employee manually switched of a segment to make repairs, and switch was remotely turned on, someone could be killed.
There are numerous other software applications that control (potentially) deadly devices - robots, industrial equipment, etc. Failure of the software, or problems with operator headspace, create a potential for death when working with almost any software that controls physical entities.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
We write software for railroad traffic control and crossing warning systems. If it fails we could end up with two trains trying to occupy the same piece of track at the same time (ref. Clapham Junction 35 dead) or gates that stay up when the train comes. Testing is very formal and rigorous and every step is documented.
For every hour we spend making sure the system does what it's supposed to do, we spend eight hours making sure it doesn't do what it's not supposed to do.
None of them can see the clouds; The polished wings don't care.