Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
Anyone who hasn't read this paper, should.
I've had this sig for three days.
I know that you're joking, but there's this little thing called "terminal velocity" that puts a crimp in your little example.
A good book that tells how technology can cause death, destruction, and mayhem entitled "Set Phasers on Stun". Includes the Therac radiation machine accidents, nuclear accidents, and many other odd stories.
42 - So long and thanks for all the fish.
There was a good discussion of this event some months ago; the current issue has blurbs on topics ranging from computer viruses to aircraft cockpit management.
Sadly, this is nothing new.
Every software developer needs to read Peter Neuman's book Computer-Related Risks , and keep up with the Risks digest (comp.risks).
Learning from other's mistakes is much less painful.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
The problem was actually one of training and clueless operators. IIRC. the coordinates of the missile launcher had to be updated several times a day. The technicians went several days without doing so. A Scud flew into the area the Patriot was supposed to be protecting, but the system was so confused as to where it was that it thought it was another batteries' responsibility and did nothing. The Scud crashed into an area with Coalition troops and killed 28, the largest death toll due to a single action in Desert Storm.
If my answers frighten you, stop asking scary questions.
These issues were already mentioned a year ago in the slashdot article Debug your Code, or Else!.
And 64 bit integers converted to 16 bit integers kill, if not people, at least big budgets.
"If you think education is expensive, try ignorance" - Derek Bok
Ha, ha.
It's a serious topic, even more so since the over-radiation shit in Panama happened so recently.
The infamous Therac-25 incidents happened between 1985 and 1987 and should be required reading... too bad the three Panamanian medical physicists cited in the article hadn't paid attention to it.
Case Studies in Information and Comuter Ethics (ISBN: 0-13-533845-X) was required readings for computer ethics course I took.
The book mainly focuses on privacy and computer crimes, but it has a section entitled "Liability, Safety, and Reliability" in which four case studies of dangerous and unreliable software are detailed.
My lack of God, it's Trotsky!
No. By law, the everything-and-the-kitchen-sink EULAs cannot be applied to any application that has to do with medical devices, air traffic control, military weaponry etc. Don't think that just because your word processor has a liability release that the same is true for all types of software.
That said, the software development standards that are required under the FDA essentially enforce standard software lifecycle practices that people should be adhering to anyway, with the exception that the accountability for signing off on a product release is a matter of law, not just good practices.
The problem was actually a programming problem. They used a floating point number for the time counter. Floating point gets less and less accurate as you get farther from 0. So after the missle had been on for a few days, the patriot missles time count would be off, since they were adding .1 ticks to a floating point number. It could have been avoided by using an integer to track the time. The workaround to this problem was to reboot the missle more often, but this wasn't found out until after the incident, when they figured out the problem. So a lesson to all programmers. Floating point is exponentially less accurate as you get farther from 0. At the very top, its only accurate to 1.
The Davis-Besse nuclear reactor in Ohio was running its safty monitoring systems on an NT server. And it got infected by Slammer and crashed. Fortunatly, the system had an analog backup, and the reactor had already been offline all year, after inspectors discovered a 6" hole through the cement in the reactor head, which left the core exposed.
ASCII stupid question, get a stupid ANSI
On the other hand, if you're working with a closed loop system (outputs to equipment and feedbacks to read the state of the equipment), you have a lot more responsibility as a programmer. You should be able to check for most equipment failures (if the feedbacks work correctly) and disable the outputs in the case of a problem. The radiation machine in Panama should have feedbacks to control the output, and those feedbacks could be checked against a "lethal dose" value. If they're not, it's the programmers (and/or designers) fault IMHO. But if the people operating the equipment change the software (disable a sensor...etc) then it's all in their hands. Any modifications outside the original design should release the programmer against any liability because you can't predict changes not made by you.
I'm looking to post my comments after I die, posthumously.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
In modern systems, where IMRT (intensity modulated radiation therapy) is used, the medical physicist in charge is supposed to verify each field delivered. This tests both the treatment planning software, as well as the accelerator, collimator leaves, etc. Often this is done using film, however, time saving electronic devices (basically diode arrays) are used by those who can afford them. Of course, the verification system has its own software, which requires verification also. Luckily the verification software is fairly simple, relative to the treatment planning and accelerator control systems.
A software glitch caused the crash of the first F-22 prototype (noone died fortunately) and as someone else pointed out, the Patriot missile failure in the first Gulf War (Software wasn't ENTIRELY to blame there. The bug was known and the folks in the field were given instructions on how to avoid it, but didn't follow them)
The trick is who do you hold responsible? The software person who has no training in mission critical software and who's working 80 hour weeks to meet the deadline the idiot managers are shoving down vis throat?
After 10 years in the industry I'm FINALLY starting to see movement towards software creation as a serious engineering discipline. Schools are starting to offer programs in Software Engineering, the ACM and IEEE have agreed on an official code of conduct (tho IHMO it still has SERIOUS problems), and most importantly companies are starting to listen to their technial folks when they say "You can't do that!".
Liability is just another incentive to head down that road. We need to be sure we pin the liability on the right folks.
The parent post has incorrect information in it.
"...after inspectors discovered a 6" hole through the cement in the reactor head, which left the core exposed."
Davis-Besse had some pretty severe corrosion on its reactor head, which ate through some of the Alloy 600 (carbon steel) metal. Beneath the approximately 6" of carbon steel, there is a stainless steel shell. This shell was not eaten through by the boric acid, as it is normally exposed to boric acid by design. Part of the stainless shell was exposed when the corrosive boric acid and other wastage was removed.
The core was not exposed. And the reactor head is NOT made of cement.
I know that someone mentioned it already but there was a good paper published on it, it's a good read. There was a similar incident a few years ago with the Therac-25 machine. Supposedly the manufacturer installed a computer and removed hard wired safetys out of the system, running everything by software. They also forgot to put in some interlocks and other good stuff, the net result being that several people died. Or it seems that there is a book here although I have not read it myself.
A lot of people forget that software can have devastating affects on things and even with the best of programming you can't beat a hardware safety for the "just in case."
I write software for automated train switches. If a bug in my software caused an accident, I would most certainly be responsible.
Railway traffic control software is very formally and rigorously tested. The only way I would not be responsible is if the tester neglected to perform a test that would have discovered the bug.