Slashdot Mirror
History's Worst Software Bugs
bharatm writes "Wired has an article on the 10 worst sofware bugs.. From the article 'Coding errors have sparked explosions, crippled interplanetary probes -- even killed people. Here's our pick for the 10 worst bugs ever, but the judging wasn't easy.'"
The last one on the list is this
... to me that sounds like a user not using the software correctly..
"Multidata's software allows a radiation therapist to draw on a computer screen the placement of metal shields called "blocks" designed to protect healthy tissue from the radiation. But the software will only allow technicians to use four shielding blocks, and the Panamanian doctors wish to use five.
The doctors discover that they can trick the software by drawing all five blocks as a single large block with a hole in the middle. What the doctors don't realize is that the Multidata software gives different answers in this configuration depending on how the hole is drawn: draw it in one direction and the correct dose is calculated, draw in another direction and the software recommends twice the necessary exposure.
At least eight patients die, while another 20 receive overdoses likely to cause significant health problems. The physicians, who were legally required to double-check the computer's calculations by hand, are indicted for murder. "
why?
That was a trojan. It was a deliberate attack on their system by an enemy. It didn't even arrive via the now classical "worm" or "virus" route, which would have implied that a "bug let it in the door." No, this one was deliberately planted carefully at the root. It's not a bug, it was an attack.
John
Wonderful article. Twenty years ago I believed that writing software would soon become a licensed profession. (Need alicense to own a compiler, for instance.) I thought that the event that would inevitably trigger this is when a software bug caused a human death.
This is like saying you need a license to operate a Soda Vending Machine because some idiot decided tipping it over trying to get a free soda was a smart idea. You might have to put warnings on compliers like do not code if you have no clue what you are doing, etc but requiring a license won't ever happen. I am sure there will be lawsuits in the future regarding software bugs, but any software being used where an error could cause a human death is going to have a corporation behind it, that can be held responsible.
Seconded.
Both radiation bugs in both cases have killed less people then the shiteware used in Patriot missile system. Ariane and Mariner get an honorable mentioning, Raytheon doen't. Why?
There also no mentioning of power grid system bugs. The recent US blackout was a good example.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
I've read about this instance before, and I think it's attributable to ignorance on both the user and the developer. The software developer in this case knows the life of a human being is resting on his code, so it should have been nigh impossible to "trick" the software into allowing anything other than what the specs said it could do.
Proud member of the American Non Sequitur Society. We might not make much sense, but boy do we love pizza!
http://www.alexisparkinn.com/photogallery/Videos/A irbus320_trees.mpg/
(Let the slashdotting begin! (poor servers))
All things considered, I don't know if the pilots survived.
----------
Any problem can be made unsolvable if there are enough meetings made to discuss it.
I designed and build a diagnostic radiology workstation (in 1997, in Java 1.1, 4x5 megapixel monitors, still in use today). During the development effort we were regaled with stories of software glitches in medical systems resulting in disaster. It really keeps you focused.
In one case, a radiation treatment system had a bug where if you used the backspace key when entering the dose a patient received, the display would show you deleted the last digit, but internally you hadn't. So the patient would recieve 10^backspace times the intended dose of radiation. Not a big deal normally, since the techs would typically shut the machine off between treatments. Until one day when they had two patients needing treatment back to back. The tech knew something was wrong when the machine was running for an unusually long time. The patient knew something was wrong when he died.
On our team a defect that crashed the system was considered severity 2. Severity 1 was reserved for defects that could result in a mis-diagnosis, which most patients agree is worse than a crash.
Here is the Dilbert Strip... Enjoyj pg
http://www.geocities.com/raptorred42/Dilbert0001.
why?
For potential severity, this one's worse than a few they listed.
Basically, the Navy was running critical ship systems on a Windows NT platform, and a divide-by-zero in a database caused a buffer overrun that resulted in a shutdown of the engines, leaving the ship dead in the water for 2.5 hours.
Fortunately, it was on maneuvers off of Cape Charles, and not at war off the coast of Yemen or something. Scratch a billion-dollar destroyer and most of her crew because of an NT bug, in that case.
We've all seen it: the employee who's convinced she's doing a great job and gets a mediocre performance appraisal, or the student who's sure he's aced an exam and winds up with a D.
The tendency that people have to overrate their abilities fascinates Cornell University social psychologist David Dunning, PhD. "People overestimate themselves," he says, "but more than that, they really seem to believe it. I've been trying to figure out where that certainty of belief comes from."
Dunning is doing that through a series of manipulated studies, mostly with students at Cornell. He's finding that the least competent performers inflate their abilities the most; that the reason for the overinflation seems to be ignorance, not arrogance; and that chronic self-beliefs, however inaccurate, underlie both people's over and underestimations of how well they're doing.
I did not become a vegetarian for my health, I did it for the health of the chickens. --Isaac Bashevis Singer
I actually did a research report on the Therac-25 incident while I was in Software Engineering class a few semesters ago (I was also in Technical Writing at the time, so I could kill two assignments with one report!) ;-) The details of the incident(s) are actually quite fascinating and sometimes spine-chilling.
Here's the report in PDF if anyone's interested: reportfinal.pdf
And in HTML for those of you who prefer it: link
From Wiki page:
It also found that FirstEnergy did not take remedial action or warn other control centers until it was too late because of a bug in the Unix-based General Electric Energy's XA/21 system that prevented alarms from showing on their control system, and they had inadequate staff to detect and correct the software bug. The cascading effect that resulted ultimately forced the shutdown of more than 100 power plants.
I think the two worst computer bugs of all time are the two that quite possibly could have wiped us all out. More inforation here.
(Copied from the article:)
* November 9, 1979, when the US made emergency retaliation preparations after NORAD saw on-screen indications that a full-scale Soviet attack had been launched. No attempt was made to use the "red telephone" hotline to clarify the situation with the USSR and it was not until early-warning radar systems confirmed no such launch had taken place that NORAD realised that a computer system test had caused the display errors. A Senator at NORAD at the time described an atmosphere of absolute panic. A GAO investigation led to the construction of an off-site test facility, to prevent similar mistakes subsequently. A fictionalized version of this incident was filmed as the movie WarGames, in which the test system is inadvertantly triggered by a teenage hacker believing himself to be playing a video game.
* September 26, 1983, when Soviet military officer Stanislav Petrov refused to launch ICBMs, despite computer indications that the US had already launched.
If it weren't for two humans who said "fuck what the computer says!", we might be in a very different place right now.
She loves me: 09F911029D74E35BD84156C5635688C0 She loves me not: 09F911029D74E35BD84156C5635688BF
In the words of the old chestnut, "If you're calm and confident when everyone around you is running around in blind panic, you clearly don't understand the situation."
Welcome to the Panopticon. Used to be a prison, now it's your home.
It doesn't matter how highly paid and trained your professionals are, if the environment that produces the software is not conducive to eliminating these types of flaws. Like if they are not given enough resources to test and QA the the projects they are assigned, there is no organizational commitment to take the time and expense to document properly, or leadership overrides technical objections to project timeframes, etc. Most of the cited projects could probably be classified as failures of project management rather than failures of the end product (the software) that these flawed projects produced. Yes, software is hard and the software profession should continue its efforts to improve quality, but that doesn't let the organizational culture, leadership and processes that produced the software in these cases off the hook.
Why is it when the accounting profession makes spectacular mistakes that take down entire Fortune 500 class organizations, there is a critical analysis of the processes that led to these failures, and remedies often comprise prescriptive measures for these processes, but similar analysis for software failures focus upon the software flaw but not the environment that allowed the flaw to emerge? Now sometimes the remedy in the accounting case might not make complete sense (like SOX), but the point here is people don't look at just the end result (the accounting system transactions) of the accounting process.