Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
Software can kill, just like any other stupid mistakes if left unchecked.
insert open source plug here
Software will only kill people through bad programming.
It is humans that make the underlying mistakes
Who makes software??? Blame it on the people who made the software. Its the same as saying guns don't kill, bullets do.
... dumb programmers kill!
... but it can make the hardware controlled by it kill.
The Tao of math: The numbers you can count are not the real numbers.
IIRC, the Patriot missile was never really designed or intended as an anti-missile missile, but a anti-aircraft (ie, a target much lower and slower) missile. It was only pressed into service killing Scuds because there was nothing better available.
So, wouldn't the Patriot missile failure be understandable due to it being used outside its original design? If the Patriot had been really intended and design as a missile killer, then yes, it should have a "failure" because it didn't live up to its original spec.
There's 10 types of people in this world, those who understand binary and those who don't.
Bad programming can, just like guns don't kill, people do. An engineer makes mathmatical mistakes designing a bridge and the bridge later collapses, do bridges kill? Seems like a dedundent question, mistakes we make sometimes cost peoples lives, why would software be any different?
Quack, quack.
Parent is correct. Software no more kills people than guns, or drive thru alcohol stores, or religion. People kill people, the method just varies.
... but it can instruct hardware to do so.
You see, if I'm a doctor, and I screw up and overdose you, it isn't a news item. I'll get reprimanded, maybe sued. No one will even notice if it happens many times, because each time it is a different doctor in a different circumstance.
But if I'm a computer software engineer and have a bug in a program that gets 3 people an overdose, then it will be noticed and much howling will be done over it. Even if the total number of errors have gone down, the type of error is new and there is a common factor between all the cases. And so we will complain.
And, I think, rightly. Computers are a tool, not to be trusted, always to be checked. I fear many people believe the computer can never be wrong (because it is so complex as to be indistringuishable from magic, and magic is never wrong) - perhaps this is why there isn't much howling about Diebold voting machines: It's digital, so it must be better!
Fellowship 9/11
Every CompSci student I know (disclaimer: most of them are Canadian) learned about the Therac-25 in class. And I'd hope that every engineer building a software-controlled radiation machine would have at least heard of it. Yet clearly its publicity has done nothing to advance the state of software engineering, as this almost identical tragedy shows.
Big scary warnings don't effect software quality. I think we should consider whether legal liability will.
I'm not positive, but aren't most of these type of disclaimers saying something along the lines of "We do not give permission for this software to be used in environments where failure could result in loss of life. In the event of such unauthorized use, we will not warranty the product, nor be held accountable for any damages it may cause"? If this is the case, than I have no problem with this, as they are saying the software isn't good enough to use in such a situation, if you do so, you're on your own. Anything that's mission critical to a degree where lives depend on it, should be licensed with that in mind (which I imagine software for nuclear power plants, etc. is).
If the organization that's being entrusted with people's lives cheaps out and uses software in environments it's not rated for, there's no way the manufacturer should be held liable. It's not different than tires on cars. If you're ripping around at 150mph on non Z-rated tired, and one blows, it's your own damned fault, not that of the manufacturer.
This is why I've always thought it's vitally important to have good, precise specifications in place and excellent quality assurance for any life-critical application. It's even better with many eyes overseeing every step of the process -- wait... that smacks of open source, doesn't it?
If you ask me -- and you haven't, but I'll tell you anyway -- what would be the best way to prevent catastrophe, it would be to PREVENT CHANGES TO THE SPEC. In college, our software engineering prof. gave us an assignment, then halfway through, she changed the spec on us. Well, not surprisingly, there wasn't a single project that worked faultlessly, and many of us were doing really well before that.
Software itself doesn't kill people. Bad software written by overworked developers writing to a constantly-changing specification with not nearly enough QA does. That is, people inadvertantly -- we hope -- kill people with software. Yeah yeah, it's cliche, but it works.
If it's not one thing it's your mother.
Steam engines have blown up and killed people. The first power machines have killed people. People have died in coal mines. In cars, in airplanes. I'm pretty confident when we first came up with fire there were a few mishaps.
It's called progress, people. The more power we gain, the better we can kill ourselves, as the 20th century showed. Which goes to prove -- with great power comes great responsibility. (HHOS)
I also have to point out the problems with luser errors. A lawyer friend of mine represents a corporation that is sued because somebody lost an arm in one of their industrial machines. The machine is set so you have to keep pushing the button to make it run. That way, if you want to go fiddle with it, it can't be running while you do it and therefore take off your arm. But what are you supposed to do when the guy tells an other guy to help him out and press the button while he puts his fork in the toaster. Did the power machine tear off that guy's arm? No, his stupidity did.
Software doesn't kill people; programmers kill people.
The bigotry of the nonbeliever is for me nearly as funny as the bigotry of the believer. - Albert Einstein
My point is more in relation to the concept that a EULA should disavow a company of all accountability. Let's look at this in other ways to help illustrate my point.
Car manufacture. This vehicle is intended only to operate withing the bounds of the law and shall be considered out of warranty if operated outside those bounds. - Not a car made would still be under warranty after a week.
Airplane manufacture. This airplane is intended to be flown in by those who choose to accept said risk. - No defect could be held against the manufacturer.
Pharmaceutical company. This drug is intended only to give an increased chance of success to the patient. All risk and responsibility is the patients to accept and the manufacturer cannot be held responsible. - It wouldnt matter if the study was done by baboons instead of on baboons, the drug company would get a walk.
It's a case of accountability, and companies' attempts to use an EULA to get out of accountability. If this precedent stands unbated we will soon have EULAs on everything from TVs to cars with no manufacture ever able to be held accountable for defects. Thats what I have problems with.
Last time I checked, we don't have a bunch of kamakazi pilots for our Tomahawk Cruise Missiles. We make software to intentionally kill people all the time.
Should read : largest coalition death toll due to a single action in Desert Storm
My editor would beat you with a ruler!
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
I was working for a desktop consulting company, and I was the only database developer there.
One of my customers wanted to convert a database, and originally I thought, no problem just convert some tables and redraw some forms.
It turns out this database was also going to store information about blood matching, transplants, and it would also calculate daily drug doses for the nurse to sign off on for kids getting marrow transplants. Success is measured in how many months the kid gets to live.
If I was working on a team using a more robust platform I might have had more confidence to push forward. However, this is Microsoft Access and i'm the only guy who would know how this thing would work. This means it would be very easy for some kid's death to point towards me.
So I quit.
By the way, if anyone has work for a database developer, feel free to contact me at will_spangler@juno.com. I'm quite good with MS Access.
Warfare is not about certainty, it's about playing the odds.
A very similar case could be constructed for them poor fried cancer patients. SQL databases that manage breast/cervical screening programs save thousands of people from cancer each year.
Engineering is the art of compromise.
If that Patriot missile failure counts as a "software kill" then surely software does kill; Look at the amount of people killed in Iraq for example by different types of bombs and cruise missiles that are guided (and detonated) by software.
I think we can all agree that yes, software, or rather the result of erroneous software, can kill. However I think the more relevant question is should someone be held responsible if in case it does?
/. was that he deserved punishment. But the case would probably also be made that the original coder should have written code that was less vulnerable to exploitation. For something so important, the code should be guaranteed stable and reliable, right?
If software screws up an automated train switch and two trains collide, or if software used to regulate power transmission goes haywire and causes a blackout that results in a dozen deaths, is the software writer responsible? I say probably not. Obviously sheer negligence and less than satisfactory work would be grounds to punish someone, but that is still subjective. While the blame could be pointed at the coder that wrote the software, he would have as much reason to point to the company that designs power transmission systems and say "Their system should have physical backups to prevent blackouts", and he would be right. They should have physical backups.
Now let's say that someone writes a virus or some exploit that brings down someone else's code and causes a blackout. Should that virus writer be punished? The large majority would say yes. We just saw a story where a guy is getting charged with terrorism for "hacking" a few DirectTV boxes, and the consensus here on
I'm sure that in the case of a few people or a small company writing code, people would demand that they be held liable for writing exploitable code. Seems a bit hypocritical since most people would never blame MS for writing exploitable code.
My $0.02
Being moderated the old-fashioned way, with a benevolently autocratic editor, it has much higher quality posts than the /. average.
Understatement of the century. The RISK digest has always had very high standards, while even FARK has a higher signal to noise ratio than Slashdot.
the legal liability ultimately rests on the engineer who signed off on the quality assurance
Actually, no. I will paraphrase the legal section of "Mechanical Engineering Reference Manual for the PE Exam" by Lindeburg, here: The courts have generally allowed engineers to make mistakes, short of gross negligence. Engineers are allowed to be human. Engineering firms on the other hand, are not given such allowances. If I design a defective product, my company is far more likely to be held responsible for it than I am, since they're supposed to have a review process to verify my work.
Lindeburg also points out that engineers usually don't carry insurance, so they just don't have pockets deep enough to be of interest to a plaintiff.
That said, safety is a huge part of the engineering culture. We design things so they can't hurt anyone, and so that they won't break. We then put in extra safeties for when they do break, so that they still can't hurt anyone. Look at the number of things on a car that are there simply to protect the driver.
The problem with the engineering approach is that it's slower. With the pace of software development, you can't spend time refining your product, you have to ship it and work on the next version. Maybe as areas mature a little more, they will stabilize enough to allow more care in design. OSS might be an answer, because they aren't motivated to ship a new version by X date to make quarterly revenue, but they also don't necessarily have the motivation to get the customer what they want (in quality or features).
The worst part is that people seem to have learned the computers just don't work. It has become so common that it is accepted, and there are a few complaints, but no outcry. Until people demand more from their software, programmers won't waste the time to do it.
After reading the article, it appears that the software in question was designed specifically for the operation of a piece of medical equipment. Given this case, I would find it to be highly unethical for the EULA governing this piece of equipment to have any kind of statement regarding an application that could endanger human life.
Another point of contention is that everyone here keeps jumping up and showing the Microsoft EULA. I do not see anything in this article that states what OS is running on this equipment. I'm not standing up for Microsoft in this case, I'm just merely stating that we can't blame them without a specific reference. Considering the type of equipment involved, it is most likely using some customized, embedded operating system.
Also, the article specifically states that the Panamanian technicians involved had introduced changes to the software and thought that adding an extra radiation shield to the patient would protect the patient from the higher radiation dosage.
Given my interpretation of the article, this is a case of user-failure versus a software error.
Most people in the comments are focusing on actual bugs and crashes in a system causing deaths. While that could certainly happen, those types of errors are more visible and actually a much "better" error to have than some other types. If the system crashes, it may have some immediate effects depending on its purpose, but if it's something that causes its action through an actual user, they are generally harmless, though very annoying. An example of the difference is that if the software designed to run a ventilator has a bug that causes it to crash, since it is directly providing life to a person, when it crashes someone will probably die. On the other hand, systems designed to give information to a clinician, who can then act upon it are going to be very aware when that system is down, and so much less likely to make an error based on that outage.
The more insidious "errors" if you want to call them that are ones that are errors of design and process, and not execution. If a piece of software is designed with certain assumptions in mind, and something happens outside of the parameters of those assumptions, the software will appear to be working correctly when in fact there may be egregious errors. There are a lot of instances of this in everyday practice.
Lastly, what we run across is that clinicians are used to a world of paper, where everything obviously either there or not. You know that there's a problem, and there is transparency to the error, so you can factor that into your decision-making. In a clinical system, the transparency is not there, and a subtle flaw can mislead someone making a clinical decision into making a poor one.
Of course, the above are all gross generalities, as is any discussion of errors in complex systems, but I hope you get the idea.
I tought we should be concerned at any level above *one* injury.
It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
people acting in a negligent (or intentional) way causes death -- not inanimate objects like cars, guns, and software. Put the blame on the right factor. Poorly designed roads and poorly designed software can both end up causing human death, but the key is that these things were poorly designed by humans. Negligence. Not acting as a reasonably prudent person in similar circumstances. Don't blame the thing, blame the people who made/designed/controlled the thing. Come on, slashdot is better than this.
Stupid people make stupid things profitable.
Isn't that the same question?
ANY wood is good for paper making.
By the way, "hardwood" dosen't MEAN that the wood is hard, any more than "softwood" implies it's soft. It's a stupid thing, whoever invented those terms.
Many so called "softwoods" are harder (more dense, stronger) than some "hardwoods".
Hardwood simply implies that it's from a member of the broad leaf (dicotyledonous) family, and "softwoods" are coniferous.
Fir trees are used for paper mainly because they aren't as valuable (for various reasons) as "hardwoods", and because they're JUST THERE.
Amazon Rainforest wood could make paper just as well as Canadian pine. Grow a fucking brain.