Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
Can Software Kill?
Certainly. A complete set of Novell manuals dropped from 40 stories up packs the same kinetic energy as a 10 car freight train moving at 80 km/h.
Trolling is a art,
Apparently it can only kill people in groups of only 28.
No.
Next story please, does it look like I have work to do?
Is 28 deaths the level at which we get concerned?
If software is outlawed, only outlaws will have software.
Don't blame Durga. I voted for Centauri.
One must be very careful when you kill -9!
If a software maker is found negligible and convicted of manslaughter (unintentionaly causing death) due to buggy software, would that void out the whole EULA business since they all claim they can't be held responsible? Or would the burden pass on the poor chap that used it for being irresponsible enough to use something where the maker couldn't be held accountable? Lets's face it, why are only software companies able to make themselves free from accountability when every other industry has to design for it?
Software can kill, just like any other stupid mistakes if left unchecked.
insert open source plug here
Software will only kill people through bad programming.
It is humans that make the underlying mistakes
Tonight on Fox...
WHEN SOFTWARE ATTACKS!
with host Mitch Pileggi
You are in error. No-one is screaming. Thank you for your cooperation.
Anyone who hasn't read this paper, should.
I've had this sig for three days.
Who makes software??? Blame it on the people who made the software. Its the same as saying guns don't kill, bullets do.
as we see technology inch into out society more in the medical field we will see more incidents of things like this happening. I know many hospitals are moving to computer based medication system, records, etc. A programing error could easily kill someone.
With this happening i think people wouldnt have any issue on arguing how programers need to be accountable for thier programs. So if they should be held responsible, should people who program other things like operating system be accountable to flaws in their programing?
30% Troll, 50% Underrated, 10% Interesting
Score:5, Troll
... dumb programmers kill!
... but it can make the hardware controlled by it kill.
The Tao of math: The numbers you can count are not the real numbers.
the therac-25 actually injured a fair number of people in the US 10-15 years ago. yeah, software fucks up sometimes. it's old news. for the article:
Nancy G. Leveson and Clark S. Turner. An investigation of the Therac-25 accidents. Computer 26, 7 (July, 1993) pages 18-41.
-ninjaneer
The software is only one piece of the puzzle, of course. Its killing was enabled by the hubris of its developers and the blind trust of its users.
Not by itself, no.
An autopilot that is consistently 1000 feet off, a poorly written control routine for an MRI, miscalibrated antilock brakes...can certainly cause death.
But ultimately, it comes back to whoever wrote it. Or specced it. Or tested it.
Software by itself is benign.
Human implementation of it may be lacking, though.
Can negligence in any area kill? Yes.
Software is no different from hardware in this aspect. If it is handling mission-critical or potentially-lethal equipment... great care should be taken to ensure its integrity.
Trusting those that make your irraditation software is no different from trusting the those that made your life-support hardware.
Human error, or mechanical, can mean death in both cases. If the error is glaring, it becomes a case of negligence.
Unfortunately in cases of software or even computer hardware operating environment becomes an often overlooked factor. Stress tests are needed... data collisions checked for, line noise, redundancy, etc. When we're talking about people's lives, that extra parity bit can be just as important as a backup-parachute...
IIRC, the Patriot missile was never really designed or intended as an anti-missile missile, but a anti-aircraft (ie, a target much lower and slower) missile. It was only pressed into service killing Scuds because there was nothing better available.
So, wouldn't the Patriot missile failure be understandable due to it being used outside its original design? If the Patriot had been really intended and design as a missile killer, then yes, it should have a "failure" because it didn't live up to its original spec.
There's 10 types of people in this world, those who understand binary and those who don't.
A good book that tells how technology can cause death, destruction, and mayhem entitled "Set Phasers on Stun". Includes the Therac radiation machine accidents, nuclear accidents, and many other odd stories.
42 - So long and thanks for all the fish.
So are you saying they INTENDED to kill their patients and this software just did it more efficiently? ;P
Un-news
Bad programming can, just like guns don't kill, people do. An engineer makes mathmatical mistakes designing a bridge and the bridge later collapses, do bridges kill? Seems like a dedundent question, mistakes we make sometimes cost peoples lives, why would software be any different?
Quack, quack.
There was a good discussion of this event some months ago; the current issue has blurbs on topics ranging from computer viruses to aircraft cockpit management.
There must be a point where software makers can no longer say "DISCLAIMER: IF WE BREAK YOUR MACHINE, IT'S NOT OUR FAULT." If you look at every piece of software's license, you'll see a clause like that. Imagine if every industry took that approach:
DISCLAIMER: IF YOUR CAR'S BRAKES FAIL, IT'S NOT OUR FAULT. TOUGH LUCK!
DISCALIMER: IF THIS MEDICINE KILLS YOU, OH WELL.. NOT OUR FAULT!
etc.
Some laws must be passed and software makers must be held accountable- they should no longer be able to hide under the big umbrella of the disclaimer.
Sadly, this is nothing new.
Every software developer needs to read Peter Neuman's book Computer-Related Risks , and keep up with the Risks digest (comp.risks).
Learning from other's mistakes is much less painful.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
... but it can instruct hardware to do so.
These issues were already mentioned a year ago in the slashdot article Debug your Code, or Else!.
Every CompSci student I know (disclaimer: most of them are Canadian) learned about the Therac-25 in class. And I'd hope that every engineer building a software-controlled radiation machine would have at least heard of it. Yet clearly its publicity has done nothing to advance the state of software engineering, as this almost identical tragedy shows.
Big scary warnings don't effect software quality. I think we should consider whether legal liability will.
IIRC, Microsoft's license has had since day zero, a clause to the effect that you are not legally allowed to use their software to control nuclear reactors, medical devices, avionics or any other application that could endanger human life. THERE'S A REASON THAT'S THERE!
If you DO have such an application, the software vendor : 1) takes much greater care in design, implementation and testing, 2) carries a godawful ginourmous insurance rider to cover any such failure.
There is a segment of the industry that works in this niche and is well aware of the risks and how to best manage them. It's goddamned clueless PHBs that think Microsoft == Software that don't understand simple goddamned little nuances like this.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Dave: Open the pod bay doors, HAL.
HAL: I'm sorry Dave, I'm afraid I can't do that.
Check out the best P2P sharing website: MEDIACHEST.COM
This is why I've always thought it's vitally important to have good, precise specifications in place and excellent quality assurance for any life-critical application. It's even better with many eyes overseeing every step of the process -- wait... that smacks of open source, doesn't it?
If you ask me -- and you haven't, but I'll tell you anyway -- what would be the best way to prevent catastrophe, it would be to PREVENT CHANGES TO THE SPEC. In college, our software engineering prof. gave us an assignment, then halfway through, she changed the spec on us. Well, not surprisingly, there wasn't a single project that worked faultlessly, and many of us were doing really well before that.
Software itself doesn't kill people. Bad software written by overworked developers writing to a constantly-changing specification with not nearly enough QA does. That is, people inadvertantly -- we hope -- kill people with software. Yeah yeah, it's cliche, but it works.
If it's not one thing it's your mother.
Steam engines have blown up and killed people. The first power machines have killed people. People have died in coal mines. In cars, in airplanes. I'm pretty confident when we first came up with fire there were a few mishaps.
It's called progress, people. The more power we gain, the better we can kill ourselves, as the 20th century showed. Which goes to prove -- with great power comes great responsibility. (HHOS)
I also have to point out the problems with luser errors. A lawyer friend of mine represents a corporation that is sued because somebody lost an arm in one of their industrial machines. The machine is set so you have to keep pushing the button to make it run. That way, if you want to go fiddle with it, it can't be running while you do it and therefore take off your arm. But what are you supposed to do when the guy tells an other guy to help him out and press the button while he puts his fork in the toaster. Did the power machine tear off that guy's arm? No, his stupidity did.
Good job with the Terminator images in everyone's heads.
Software does not kill. Bad engineering and poor implementation kills. My copy of Windows XP, while still radiating pure evil, has not managed to pop open the gun cabinet.
You might as well ask the question, 'can the old saddlebag gas tanks on Ford Rangers kill? Gasp!'
I can't wait until they let software fly passenger airplanes. Remember the Airbus accident several years ago where the software took control of the plane during the landing sequence and flew the plane into forest past the end of the runway. The pilot was trying to pull up to abort the landing but the software was ignoring the pilots control motion because it was apparently programmed to ignore drastic stick movement during the landing sequence. Airbus's mentality was "the software knows better than the pilot". I don't recall if anyone (pilot or co-pilot) was killed in that incident or not. Boeing's philosophy here is that the pilot always knows better than the software.
Microsoft Windows: A thirty-two bit extension and graphical shell to a sixteen-bit patch to an eight-bit operating system originally coded for a four-bit microprocessor which was written by a two-bit company that can't stand one bit of competition
-- author unknown
Case Studies in Information and Comuter Ethics (ISBN: 0-13-533845-X) was required readings for computer ethics course I took.
The book mainly focuses on privacy and computer crimes, but it has a section entitled "Liability, Safety, and Reliability" in which four case studies of dangerous and unreliable software are detailed.
My lack of God, it's Trotsky!
The problem is that in every other development environment, the legal liability ultimately rests on the engineer who signed off on the quality assurance. But because software developers are not professionals and have no professional code of conduct, their signatures are meaningless. The only way software can become as reliable as other engineered products is to create the profession of software engineering*. And I'm not just talking about giving CompSci students a ring: many CompSci curriculums don't require any engineering techniques at all, and those that do usually devote less time to engineering than they do to sorting algorithms. The software industry requires fundamental changes, and legal liability is at most the catalyst.
* Yes, I know there are a couple of schools out there that offer SoftEng degrees, but until industry distinguishes them from CompScists and requires the engineering designation for key positions they are meaningless.
Last time I checked, we don't have a bunch of kamakazi pilots for our Tomahawk Cruise Missiles. We make software to intentionally kill people all the time.
Should read : largest coalition death toll due to a single action in Desert Storm
My editor would beat you with a ruler!
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
I was working for a desktop consulting company, and I was the only database developer there.
One of my customers wanted to convert a database, and originally I thought, no problem just convert some tables and redraw some forms.
It turns out this database was also going to store information about blood matching, transplants, and it would also calculate daily drug doses for the nurse to sign off on for kids getting marrow transplants. Success is measured in how many months the kid gets to live.
If I was working on a team using a more robust platform I might have had more confidence to push forward. However, this is Microsoft Access and i'm the only guy who would know how this thing would work. This means it would be very easy for some kid's death to point towards me.
So I quit.
By the way, if anyone has work for a database developer, feel free to contact me at will_spangler@juno.com. I'm quite good with MS Access.
Come to think of it, the power company never warned me that all that electricity the sell me could be dangerous... it was here when I got the house. Maybe it's in the fine print on my electric bill, but I think most of that is how they can track me down to collect money from me if I don't pay...
Seriously - I think it depends on the software. If you use consumer grade software with a EULA that says "this software is good for nothing, use at your own risk"... well fine.
I seriously doubt the firmware for an X-Ray machine has a EULA like that, but then, I didn't RTFA. I am sure critical software doesn't come with the usual global-deny-all-responsibility clauses.
Two final thoughts
For somethings, who would you trust more? Dr. McFeelgood with his finger on a button and a stopwatch, or software control?
If you can't mathematically prove the software is incorrect (by source code inspection), how can you prove it was a particular software in a multi-software environment, an intermittant hardware fault or a cosmic ray flipping the wrong bit?
This issue is a bit more complicated than you think.
Warfare is not about certainty, it's about playing the odds.
A very similar case could be constructed for them poor fried cancer patients. SQL databases that manage breast/cervical screening programs save thousands of people from cancer each year.
Engineering is the art of compromise.
The next time you visit the doctor watch the workflow of the office staff. Increasingly chances are they will probably be entering your medical information, and I mean the clinical stuff, not your address into some type of computer system.
I currently work for a small Electronic Medical Records company. At some level I worry about potentially killing someone every day. In fact our bug tracking tool has a special category in it called "Patient Safety" which is the highest priority bug. We deal with things most of you probably wouldn't think of such as a tool for writing Prescriptions, which given the fact that many drugs interact ( potentially fatally) has to catch and alert the physician to such cases. I also deal with lab results which if reported incorrectly could lead to a potentially fatal decision by the doctor and so forth.
Consultants and pundits like to say that computer control reduces the chances of human error and failure, this is said IMO to comfort the masses. To state the obvious I suspect EVERYONE on Slashdot knows that in reality that statement is not true, the human error has just been moved to a different point in the chain. A tired programmer is just as likely to make a mistake as a tired machinery operator. The difference is that that software might be used by 5,000 machines, whearas that operator runs 1.
If that Patriot missile failure counts as a "software kill" then surely software does kill; Look at the amount of people killed in Iraq for example by different types of bombs and cruise missiles that are guided (and detonated) by software.
I think we can all agree that yes, software, or rather the result of erroneous software, can kill. However I think the more relevant question is should someone be held responsible if in case it does?
/. was that he deserved punishment. But the case would probably also be made that the original coder should have written code that was less vulnerable to exploitation. For something so important, the code should be guaranteed stable and reliable, right?
If software screws up an automated train switch and two trains collide, or if software used to regulate power transmission goes haywire and causes a blackout that results in a dozen deaths, is the software writer responsible? I say probably not. Obviously sheer negligence and less than satisfactory work would be grounds to punish someone, but that is still subjective. While the blame could be pointed at the coder that wrote the software, he would have as much reason to point to the company that designs power transmission systems and say "Their system should have physical backups to prevent blackouts", and he would be right. They should have physical backups.
Now let's say that someone writes a virus or some exploit that brings down someone else's code and causes a blackout. Should that virus writer be punished? The large majority would say yes. We just saw a story where a guy is getting charged with terrorism for "hacking" a few DirectTV boxes, and the consensus here on
I'm sure that in the case of a few people or a small company writing code, people would demand that they be held liable for writing exploitable code. Seems a bit hypocritical since most people would never blame MS for writing exploitable code.
My $0.02
Most people in the comments are focusing on actual bugs and crashes in a system causing deaths. While that could certainly happen, those types of errors are more visible and actually a much "better" error to have than some other types. If the system crashes, it may have some immediate effects depending on its purpose, but if it's something that causes its action through an actual user, they are generally harmless, though very annoying. An example of the difference is that if the software designed to run a ventilator has a bug that causes it to crash, since it is directly providing life to a person, when it crashes someone will probably die. On the other hand, systems designed to give information to a clinician, who can then act upon it are going to be very aware when that system is down, and so much less likely to make an error based on that outage.
The more insidious "errors" if you want to call them that are ones that are errors of design and process, and not execution. If a piece of software is designed with certain assumptions in mind, and something happens outside of the parameters of those assumptions, the software will appear to be working correctly when in fact there may be egregious errors. There are a lot of instances of this in everyday practice.
Lastly, what we run across is that clinicians are used to a world of paper, where everything obviously either there or not. You know that there's a problem, and there is transparency to the error, so you can factor that into your decision-making. In a clinical system, the transparency is not there, and a subtle flaw can mislead someone making a clinical decision into making a poor one.
Of course, the above are all gross generalities, as is any discussion of errors in complex systems, but I hope you get the idea.
Having said that, it is also obvious that software that is to be used in mission critical systems, where failures would put people at risk; must be certified accordingly.
Take for instance the stringent testing and certification rules for software to be used in NASA's space shuttles. Write Stuff
There is, however, a steep price to be paid. Writing bug-free software is very expensive, and only those that cannot live with the possibility of failure, have to shell out the cash.
http://catless.ncl.ac.uk/Risks/
On the other hand, if you're working with a closed loop system (outputs to equipment and feedbacks to read the state of the equipment), you have a lot more responsibility as a programmer. You should be able to check for most equipment failures (if the feedbacks work correctly) and disable the outputs in the case of a problem. The radiation machine in Panama should have feedbacks to control the output, and those feedbacks could be checked against a "lethal dose" value. If they're not, it's the programmers (and/or designers) fault IMHO. But if the people operating the equipment change the software (disable a sensor...etc) then it's all in their hands. Any modifications outside the original design should release the programmer against any liability because you can't predict changes not made by you.
Again, from the article:
Every example given was life threatening, yet the author clearly wants you to draw the conclusion that a software author should hesitate to publish a program she wrote to perform a calculation because someone *who thought they knew what they were doing* might plug it into a lethal machine.Next we will be hearing about how someone wrote a spreadsheet in gnumeric to calculate the radiation dose, killed someone because of a bug in gnumeric, and how the authors of gnumeric should be ashamed of themselves, and not the asshole who *thought he knew what he was doing.*
Special Superior Deputy Prosecutor Cristobal Arboleda, unlike the author of the article, is accusing the right people and doing his job well.
people acting in a negligent (or intentional) way causes death -- not inanimate objects like cars, guns, and software. Put the blame on the right factor. Poorly designed roads and poorly designed software can both end up causing human death, but the key is that these things were poorly designed by humans. Negligence. Not acting as a reasonably prudent person in similar circumstances. Don't blame the thing, blame the people who made/designed/controlled the thing. Come on, slashdot is better than this.
Stupid people make stupid things profitable.
You wrote:
"If you're ripping around at 150mph on non Z-rated tired, and one blows, it's your own damned fault, not that of the manufacturer."
The problem with your analogy is that tire manufacturers are required to specify the maximum speed rating of their tires between N (87 mph) and Y (186 mph). Note: Z is now effectively a dead designation.
The DOT/NHTSA does not allow Firestone to put a disclaimer on their tires saying that they can't be used when the failure could cause injury or death. And neither should the feds allow software publishers to do that. Specify the limitations of the product, but don't try to weasel out of liability by putting vague warnings about the use in life-critical applications. If it fails to work as advertised, then the publisher should be fully liable, regardless of how someone else used the software.
Do that, and you'll see Microsoft investing a lot more in making a stable OS rather than making bad video editors and sub-par instant messaging clients to bundle with each OS.
They may not be the "code" that most of you know but it's an interesting story...
I've seen code on PLC's and embedded microcontrolers used in industrial safety applications that could have been lethal or at the very least, a finger eater. For about four years, I worked for a company that manufactured light curtains, hardguards and other devices that keep machinery operators from losing limbs.
The most dangerous "code" I've seen was on a large metal stamping machine. The maintenance department was using a PLC to take the anti-tie-down inputs and start the stamping process. The operators had tied one of the palm buttons down with black tape so they could hold the metal with one hand and start the process with the other. The PLC was also counting the inputs and cycling the machine to satisfy the counts whether the operators wanted it or not. Anyway, to make a long story short, our company was called in to install a light curtain and pitch all the other "safety" junk. The techs that mounted the curtain found a part of a hand behind the old physical guard.
In modern systems, where IMRT (intensity modulated radiation therapy) is used, the medical physicist in charge is supposed to verify each field delivered. This tests both the treatment planning software, as well as the accelerator, collimator leaves, etc. Often this is done using film, however, time saving electronic devices (basically diode arrays) are used by those who can afford them. Of course, the verification system has its own software, which requires verification also. Luckily the verification software is fairly simple, relative to the treatment planning and accelerator control systems.
A software glitch caused the crash of the first F-22 prototype (noone died fortunately) and as someone else pointed out, the Patriot missile failure in the first Gulf War (Software wasn't ENTIRELY to blame there. The bug was known and the folks in the field were given instructions on how to avoid it, but didn't follow them)
The trick is who do you hold responsible? The software person who has no training in mission critical software and who's working 80 hour weeks to meet the deadline the idiot managers are shoving down vis throat?
After 10 years in the industry I'm FINALLY starting to see movement towards software creation as a serious engineering discipline. Schools are starting to offer programs in Software Engineering, the ACM and IEEE have agreed on an official code of conduct (tho IHMO it still has SERIOUS problems), and most importantly companies are starting to listen to their technial folks when they say "You can't do that!".
Liability is just another incentive to head down that road. We need to be sure we pin the liability on the right folks.
Isn't that the same question?
In a former life ( :-> ) I was employed by a large multi-national that worked with utilities. Some of our software used SCADA protocols to remotely switch breakers - not household breakers, these switches control significant segments of the US power grid.
All the software and documentation contained numerous warnings, because if a utility employee manually switched of a segment to make repairs, and switch was remotely turned on, someone could be killed.
There are numerous other software applications that control (potentially) deadly devices - robots, industrial equipment, etc. Failure of the software, or problems with operator headspace, create a potential for death when working with almost any software that controls physical entities.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
We write software for railroad traffic control and crossing warning systems. If it fails we could end up with two trains trying to occupy the same piece of track at the same time (ref. Clapham Junction 35 dead) or gates that stay up when the train comes. Testing is very formal and rigorous and every step is documented.
For every hour we spend making sure the system does what it's supposed to do, we spend eight hours making sure it doesn't do what it's not supposed to do.
None of them can see the clouds; The polished wings don't care.
...the software controlling them do!
Four bytes @ 7 bits is enough to store the state of 28 humans: dead(0) or alive(1).
Opportunity knocks. Karma hunts you down.
I know that someone mentioned it already but there was a good paper published on it, it's a good read. There was a similar incident a few years ago with the Therac-25 machine. Supposedly the manufacturer installed a computer and removed hard wired safetys out of the system, running everything by software. They also forgot to put in some interlocks and other good stuff, the net result being that several people died. Or it seems that there is a book here although I have not read it myself.
A lot of people forget that software can have devastating affects on things and even with the best of programming you can't beat a hardware safety for the "just in case."
As tragic as it is, the Panama incident does not stand alone. In all, Baseline has found no fewer than a half-dozen cases in which software has contributed to loss of life.
I'm surprised that they only found a half-dozen cases in which software contributed to loss of life. Back in the old days when I used to subscribe to the comp.risks digest it used to seem that every couple months there was some fatality or near fatality that could be traced to flawed software. If anything, given the increasing use of embedded systems in our society, I would have thought that the fatality rate would have increased.
I write software for automated train switches. If a bug in my software caused an accident, I would most certainly be responsible.
Railway traffic control software is very formally and rigorously tested. The only way I would not be responsible is if the tester neglected to perform a test that would have discovered the bug.
I believe no discussion of deadly software disasters is complete without mentioning the London Ambulance Service Disaster of '92. Basically, bad project management and other gaffes lead to the Londom Ambulance Service using a computerised dispatch system which was not up to the job. It promptly crashed, and quite a few people died as a result.