Slashdot Mirror
Can Software Kill?
mykepredko writes "Eweek has an interesting, if somewhat long article titled Can Software Kill? The article focuses on a programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation. The article briefly mentions, but doesn't go into detail, the 1991 Patriot Missile Failure that resulted in the deaths of 28 American service men and women."
Can Software Kill?
Certainly. A complete set of Novell manuals dropped from 40 stories up packs the same kinetic energy as a 10 car freight train moving at 80 km/h.
Trolling is a art,
Apparently it can only kill people in groups of only 28.
Let's log into SKYNET and find out?
No.
Next story please, does it look like I have work to do?
Is 28 deaths the level at which we get concerned?
If software is outlawed, only outlaws will have software.
Don't blame Durga. I voted for Centauri.
if you aim for 28 victims...
One must be very careful when you kill -9!
If a software maker is found negligible and convicted of manslaughter (unintentionaly causing death) due to buggy software, would that void out the whole EULA business since they all claim they can't be held responsible? Or would the burden pass on the poor chap that used it for being irresponsible enough to use something where the maker couldn't be held accountable? Lets's face it, why are only software companies able to make themselves free from accountability when every other industry has to design for it?
Software can kill, just like any other stupid mistakes if left unchecked.
insert open source plug here
Software will only kill people through bad programming.
It is humans that make the underlying mistakes
Tonight on Fox...
WHEN SOFTWARE ATTACKS!
with host Mitch Pileggi
You are in error. No-one is screaming. Thank you for your cooperation.
It's the human error factor.
Short of the software making an intelligent decision to kill, it always comes back to us.
Yes it can.
I thought the magic number was always 42
Anyone who hasn't read this paper, should.
I've had this sig for three days.
Who makes software??? Blame it on the people who made the software. Its the same as saying guns don't kill, bullets do.
No doubt software can kill people. That's why the military, airports and train stations don't use windows.
as we see technology inch into out society more in the medical field we will see more incidents of things like this happening. I know many hospitals are moving to computer based medication system, records, etc. A programing error could easily kill someone.
With this happening i think people wouldnt have any issue on arguing how programers need to be accountable for thier programs. So if they should be held responsible, should people who program other things like operating system be accountable to flaws in their programing?
30% Troll, 50% Underrated, 10% Interesting
Score:5, Troll
... dumb programmers kill!
... but it can make the hardware controlled by it kill.
The Tao of math: The numbers you can count are not the real numbers.
the therac-25 actually injured a fair number of people in the US 10-15 years ago. yeah, software fucks up sometimes. it's old news. for the article:
Nancy G. Leveson and Clark S. Turner. An investigation of the Therac-25 accidents. Computer 26, 7 (July, 1993) pages 18-41.
-ninjaneer
Another example that could have, but apparently didn't.
I suspect this sort of thing is going to become more prevalent as our reliance on tech increases. Assasination by code?
The software is only one piece of the puzzle, of course. Its killing was enabled by the hubris of its developers and the blind trust of its users.
This is old news...
In any introductory Software Engineering course, they will discuss this as one of the first topics.
Humans make mistakes and mistakes can cause an OS to crash, a program to seg fault or for a few people to die. We, like our software, are fallibe.
100% Insightful
Not by itself, no.
An autopilot that is consistently 1000 feet off, a poorly written control routine for an MRI, miscalibrated antilock brakes...can certainly cause death.
But ultimately, it comes back to whoever wrote it. Or specced it. Or tested it.
Software by itself is benign.
Human implementation of it may be lacking, though.
Can negligence in any area kill? Yes.
Software is no different from hardware in this aspect. If it is handling mission-critical or potentially-lethal equipment... great care should be taken to ensure its integrity.
Trusting those that make your irraditation software is no different from trusting the those that made your life-support hardware.
Human error, or mechanical, can mean death in both cases. If the error is glaring, it becomes a case of negligence.
Unfortunately in cases of software or even computer hardware operating environment becomes an often overlooked factor. Stress tests are needed... data collisions checked for, line noise, redundancy, etc. When we're talking about people's lives, that extra parity bit can be just as important as a backup-parachute...
It's a little known fact that the first machine attached to skynet to become self-aware will be a Therac-25 controlled by a pdp 11, although it will be quickly obsoleted by the terminator class machines that have been deployed in Sacramento.
microsoftword.mp3 - it doesn't care that they're not words...
IIRC, the Patriot missile was never really designed or intended as an anti-missile missile, but a anti-aircraft (ie, a target much lower and slower) missile. It was only pressed into service killing Scuds because there was nothing better available.
So, wouldn't the Patriot missile failure be understandable due to it being used outside its original design? If the Patriot had been really intended and design as a missile killer, then yes, it should have a "failure" because it didn't live up to its original spec.
There's 10 types of people in this world, those who understand binary and those who don't.
A good book that tells how technology can cause death, destruction, and mayhem entitled "Set Phasers on Stun". Includes the Therac radiation machine accidents, nuclear accidents, and many other odd stories.
42 - So long and thanks for all the fish.
synopsis: mistakes made can have big and small consequences.
NERDS!!!!
I wrote a buggy web app once with a few nested 'for loops' that generated an infinite loop. After about two minutes the web server exploded killing all of my co-workers.
(jk)
I've been defenestrated by XP several times today, and I'm still here ;-)
When I am king, you will be first against the wall.
Well, sure. Using anything more than the standard lethal dose to kill some one is gratuitous and downright inefficient! Sounds like that code could have been tested more -- think of the savings in power!
Hummmm ... 28 lethal dose, 28 killed in explosing, 28 comments so far. Well i'd better log off before my browser kills me !
So are you saying they INTENDED to kill their patients and this software just did it more efficiently? ;P
Un-news
Bad programming can, just like guns don't kill, people do. An engineer makes mathmatical mistakes designing a bridge and the bridge later collapses, do bridges kill? Seems like a dedundent question, mistakes we make sometimes cost peoples lives, why would software be any different?
Quack, quack.
There was a good discussion of this event some months ago; the current issue has blurbs on topics ranging from computer viruses to aircraft cockpit management.
Comp
Dot
Risks
and an almost fanatical devotion to the pope? truth? blaming society because you're all a bunch of McDonalds swilling fatasses??
hmm... did i think that or did i type it?
There must be a point where software makers can no longer say "DISCLAIMER: IF WE BREAK YOUR MACHINE, IT'S NOT OUR FAULT." If you look at every piece of software's license, you'll see a clause like that. Imagine if every industry took that approach:
DISCLAIMER: IF YOUR CAR'S BRAKES FAIL, IT'S NOT OUR FAULT. TOUGH LUCK!
DISCALIMER: IF THIS MEDICINE KILLS YOU, OH WELL.. NOT OUR FAULT!
etc.
Some laws must be passed and software makers must be held accountable- they should no longer be able to hide under the big umbrella of the disclaimer.
You're stupid.
Sadly, this is nothing new.
Every software developer needs to read Peter Neuman's book Computer-Related Risks , and keep up with the Risks digest (comp.risks).
Learning from other's mistakes is much less painful.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
That means MS product is SAFE.
As part of an ergonomics class the professor made an issue that the radiation incidents like that may not be faulty coding or software, but just poorly designed user interfaces that lacked the proper feedback or something similar. He also made the point that if hundreds of seperate alarms went off at the same time at a nuclear power plant how is the operator supposed to repond to that? Human factors are easily overlooked by programmers and designers.
Linux Resources
The engine control computer on my 1994 Jeep Wrangler decided to shut the engine down in the middle of a busy intersection on a hot Phoenix afternoon. Almost caused an accident that could have killed me. Chrysler never figured out if it was a software, hardware, or sensor error. They replaced the engine computer for free though, because it was still under the emissions warraynt.
-- SKYKING, SKYKING, DO NOT ANSWER.
remember this[cnet.com] crazy event.
apparently windows ce was for some reason intergrated into a BMW that was being used to escort a high ranking offical of Thailand.
That's enough to make one want to kill.
Comparing it to Windows will be a moot point, since El Dorado is going to have a 40% larger code base than XP.
Promise to really kick it up a notch in the commenting area in future if ever asked to code controls for anything emitting radiation. Better yet, refrain from all assignments where control of radiation levels is required.
Why would any radiation-emitting medical device be pemitted to administer a lethal dose? I mean, why was the piece of hardware itself given that kind of upper range?
... but it can instruct hardware to do so.
Wasn't there a recent article about how faulty US software was to blame for one of the biggest non-nuclear explosions the world has ever seen?
Deliberately or otherwise, software can and does kill.
These issues were already mentioned a year ago in the slashdot article Debug your Code, or Else!.
"receiving many times an expected lethal dose of radiation"
"So the dose was expected to be lethal and the software multiplied it"
Actually it says that it delivered many times the dose that would be expected to be lethal, not that the dose was expected to be lethal and this multiplied it beyond that. You know there is not a level of 30Krads or something that kills everyone, while 29Krads won't. The effects of radiation are based on a number of variables, not just the power level or exposure time.
Every CompSci student I know (disclaimer: most of them are Canadian) learned about the Therac-25 in class. And I'd hope that every engineer building a software-controlled radiation machine would have at least heard of it. Yet clearly its publicity has done nothing to advance the state of software engineering, as this almost identical tragedy shows.
Big scary warnings don't effect software quality. I think we should consider whether legal liability will.
Of course, software can be used to kill. What do you think guides cruise missiles, intercontinental ballistic missiles, HARMs, Stingers, Hellfires, etc. Any type of guided weapon is steered by software.
There is even software at use in the cell phones Iraqi insurgents use to trigger roadside bombs.
Haven't heard of anyone dying by ingesting software yet, but those tests are probably still in the stage where they are being conducted on mice first.
Bureaucracy loves company.
I suppose that Darl McBride is trying to further his claims that his enemies are out to kill him, eh?
Wh47 d1d j00 541, 31337 15n't t3h r0xor5 ne m0r3???
... some software forces you to kill yourself.
Remember when software kills don't blame the programmers! Blame QA. They find the stoopid bugs but miss the deadly ones. I best the people who QAd the medical software wrote up 40 spelling errors.
something Orrin Hatch would have thought up
-Phil
Shoot questions, first ask later...
IIRC, Microsoft's license has had since day zero, a clause to the effect that you are not legally allowed to use their software to control nuclear reactors, medical devices, avionics or any other application that could endanger human life. THERE'S A REASON THAT'S THERE!
If you DO have such an application, the software vendor : 1) takes much greater care in design, implementation and testing, 2) carries a godawful ginourmous insurance rider to cover any such failure.
There is a segment of the industry that works in this niche and is well aware of the risks and how to best manage them. It's goddamned clueless PHBs that think Microsoft == Software that don't understand simple goddamned little nuances like this.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
In Soviet Russia, software kills YOU!
Mod +5 Drunk
This reminds me of an episode of Law & Order where the stereotypical 13 year old computer prodigy was put on trial for manslaughter for making a virus that would infect medical computers to increase the perscribed dosage of insulin to diabetes patients. The patients would go into shock and then die. Damn, that was a good episode. I miss DA Adam Schiff.
"programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose of radiation" "Doctor David Banner - physician, scientist; searching for a way to tap into the hidden strengths that all humans have..." Oh, I'm sorry, '70s relapse. Well, at least in this case, there aren't any shirtless green Lou Ferrignos running around Panama...
"We are all in the gutter, but some of us are looking at the stars." - Oscar Wilde
Software doesn't kill people, people kill people.
Just like guns/cars/knives don't kill people, people kill people.
Dave: Open the pod bay doors, HAL.
HAL: I'm sorry Dave, I'm afraid I can't do that.
Check out the best P2P sharing website: MEDIACHEST.COM
Of course. Any system can kill. Software is just a system. If a mechanical system has been used in the software's place it could kill as well. If the nurse has misread the doctors handwriting and administered an overdose it would produce the same effect.
Didn't the Al Qaeda terrorists use IM software to help plan the 9/11 attacks?
Rank Presidents by th
This is why I've always thought it's vitally important to have good, precise specifications in place and excellent quality assurance for any life-critical application. It's even better with many eyes overseeing every step of the process -- wait... that smacks of open source, doesn't it?
If you ask me -- and you haven't, but I'll tell you anyway -- what would be the best way to prevent catastrophe, it would be to PREVENT CHANGES TO THE SPEC. In college, our software engineering prof. gave us an assignment, then halfway through, she changed the spec on us. Well, not surprisingly, there wasn't a single project that worked faultlessly, and many of us were doing really well before that.
Software itself doesn't kill people. Bad software written by overworked developers writing to a constantly-changing specification with not nearly enough QA does. That is, people inadvertantly -- we hope -- kill people with software. Yeah yeah, it's cliche, but it works.
If it's not one thing it's your mother.
Steam engines have blown up and killed people. The first power machines have killed people. People have died in coal mines. In cars, in airplanes. I'm pretty confident when we first came up with fire there were a few mishaps.
It's called progress, people. The more power we gain, the better we can kill ourselves, as the 20th century showed. Which goes to prove -- with great power comes great responsibility. (HHOS)
I also have to point out the problems with luser errors. A lawyer friend of mine represents a corporation that is sued because somebody lost an arm in one of their industrial machines. The machine is set so you have to keep pushing the button to make it run. That way, if you want to go fiddle with it, it can't be running while you do it and therefore take off your arm. But what are you supposed to do when the guy tells an other guy to help him out and press the button while he puts his fork in the toaster. Did the power machine tear off that guy's arm? No, his stupidity did.
...mainly when the software is controlling a pair of 5 ton robotic (human crunching) legs.
Of course software that tells your garbage disposal to turn on at the wrong time would also be bad.
I guess microwave software has a hardware interlock to deal with, so I can safely piss the software off and clean my microwave at the same time ("just stick my head in here to see if the roof is clean..." bRRRRRRrrwttTTTTTT!!!!!!!!!).
Ultimately, I guess it boils down to how many grandmas have tripped over a roomba down a flight of stairs. Though airbus accidents are more exciting.
Someone set us up the bomb, so shine we are!
Did any of them, by any chance, experience dramatic increases in strength when they got really angry?
Good job with the Terminator images in everyone's heads.
Software does not kill. Bad engineering and poor implementation kills. My copy of Windows XP, while still radiating pure evil, has not managed to pop open the gun cabinet.
You might as well ask the question, 'can the old saddlebag gas tanks on Ford Rangers kill? Gasp!'
even waffles. i'm sure someone has died from choking on a waffle. or maybe that person ate a poisoned waffle. or maybe that person didn't know he was allergic to something ;)
off-topic? maybe, but the point is that no matter how good something can be it can inadvertantly cause harm.
I can't wait until they let software fly passenger airplanes. Remember the Airbus accident several years ago where the software took control of the plane during the landing sequence and flew the plane into forest past the end of the runway. The pilot was trying to pull up to abort the landing but the software was ignoring the pilots control motion because it was apparently programmed to ignore drastic stick movement during the landing sequence. Airbus's mentality was "the software knows better than the pilot". I don't recall if anyone (pilot or co-pilot) was killed in that incident or not. Boeing's philosophy here is that the pilot always knows better than the software.
I'm surpirsed nobodies been killed by any of the thousands of killer bugs in Microsoft's code.
Microsoft Windows: A thirty-two bit extension and graphical shell to a sixteen-bit patch to an eight-bit operating system originally coded for a four-bit microprocessor which was written by a two-bit company that can't stand one bit of competition
-- author unknown
Software doesn't kill people.
People kill people.
In the case of the missiles, it seems obvious (off a skimming from an article) that the design parameters were different than the actual use parameters.
Actually, in war, this kind of thing should be expected with a certain percentage. Technology used in wars are very often have little or no testing (like radar in WW2 and the unmanned planes used in Afghanistan). Technology with little or not testing or technology used out of the bounds of design are destined to have higher degrees of error. (Think of it like "just-in-time" software.)
I wouldn't say that software killed those soldiers. It was the scud. No piece of software is 100% bug free (especially when used in situations it's not made for).
"Injustice anywhere is a threat to justice everywhere." - Martin Luther King, Jr.
Don't talk about this people...
or you know who is next in line to get sued... us programmers.
Look at malpractice...its INSANE right now. Yes there are legitimate cases but only about 2% are legit. I read about a family who have successfully SUED for malpractice and were awarded millions that their 18 year old son was delivered wrongly and thus suffers brain damage. How did this come about? His grades weren't good enough to get into a good college. I'm not joking.
Developers will be held 100% accountable soon and its a scary thought. I'm glad I put a warning, void of liability and so forth in my software but those in medical industry will have a hefty amount to deal with from here on out IMO.
While your interpretation is surely correct, the wording in the summary is, indeed, quite strange.
Case Studies in Information and Comuter Ethics (ISBN: 0-13-533845-X) was required readings for computer ethics course I took.
The book mainly focuses on privacy and computer crimes, but it has a section entitled "Liability, Safety, and Reliability" in which four case studies of dangerous and unreliable software are detailed.
My lack of God, it's Trotsky!
A guy I knew on another product team was killed when his project came back from testing with over 200 bugs. When told the news, witnesses said he just sat there, his face turning redder and redder, then his head just exploded.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
The problem is that in every other development environment, the legal liability ultimately rests on the engineer who signed off on the quality assurance. But because software developers are not professionals and have no professional code of conduct, their signatures are meaningless. The only way software can become as reliable as other engineered products is to create the profession of software engineering*. And I'm not just talking about giving CompSci students a ring: many CompSci curriculums don't require any engineering techniques at all, and those that do usually devote less time to engineering than they do to sorting algorithms. The software industry requires fundamental changes, and legal liability is at most the catalyst.
* Yes, I know there are a couple of schools out there that offer SoftEng degrees, but until industry distinguishes them from CompScists and requires the engineering designation for key positions they are meaningless.
But wouldnt "massive dosses of gamma raise" cause all of the cancer patients to turn into the incredible hulk?
Whenever I came across this warning you find in your computer manual, it made me chuckle. Next thing you know, you'll find the same warning on software. This computer system is not intended for use in the operation of nuclear facilities, aircraft navigation or communications systems, or air traffic control machines, or for any other use where the failure of the computer system could lead to death, personal injury or severe environmental damage.
Oh sure. Blame the programmers. ;)
I dont see any cancer survivors thanking the programmers of radiation machines, when they live.
Kinda like when football players or rap guys thank god for winning something. Do you ever hear them blame god for losing?
Software doesn't kill people, people kill people!
Sounds like a Discordian plot breed fear of and contempt for modern software.
"Just because something has an alternate use does not mean the manufacturers should be liable."
You mean, if you decide to try and drink hot coffee with your crotch, you shouldn't sue the place that sold the coffee? That is radical!
When 3 billion people died. Or so the movie says.
risk: tens, hundreds, maybe thousands of people dying because of a software glitch, defective part, or lousy design
reward: a little certificate of appreciation from the company
Mrs. Schroedinger to Mr. Schroedinger: "What the hell did you do to that cat?! It looks half-dead!"
Quidquid latine dictum sit, altum sonatur.
When I could afford to fly I used to enjoy going under the radar. To dissapear, all you had to do was point into the wind and go into slow flight and you would dissapear from the radar. I wonder how many controllers chewed a hole in their seat when the VFR track they were following dissapeared.
Last time I checked, we don't have a bunch of kamakazi pilots for our Tomahawk Cruise Missiles. We make software to intentionally kill people all the time.
Should read : largest coalition death toll due to a single action in Desert Storm
My editor would beat you with a ruler!
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Software can kill, only if the hardware that it controls can kill.... This reminds me a little of Tom Clancy's NetForce. There was a co-ordinated jailbreak which involved some guys in a van, and a hacker. It just made me think to myself "Who is the IDIOT who connected the jail cell door solenoid to the internet?". It's THOSE guys that do the killing, not the software.
I beg to differ. Software quality is not neccessarily that important. It depends on the application. I'm working with some very buggy software right now, but given the choice between having the features it has and waiting another year or 2 for it to reach a level that I would call stable, I will take it's current incarnation. Software quality on a machine that governs radiation matters, but it's less important in non-critical applications. Often, having a product to market faster in exchange for a decrease in quality is a worthwhile trade-off. That's why I often download alpha and beta software, because I want it now, not later, and I don't really care all that much about stability.
Of course software can be used to control the actions of machines that are dangerous, so of course software can kill.
But the people who write stuff like this never seem to understand risk analysis, imho (or at least they choose not to write about their issues in a balanced way). For this to be interesting, you gotta ask, what are the alternatives? How risky would they be?
Example: my dishwasher has a mechanical timer. Last year, for unknown reasons, it decided to get stuck in the "dry" cycle, leaving the heating element on for 9 hours while I was out. When I got back, my house smelled like burned plastic, and the vinyl covering on the racks had been discolored by the heat. Longer, and I'm sure there could have been a fire.
Now I don't run the damned thing unless I'm home. I'd feel a lot safer trusting a microprocessor-controlled timer implemented by some unknown, harried embedded software developer.
So how come there's no Internet email group dedicated to "the risks of cheap mechanical timers and similar devices"? Because that subject, important as it is, doesn't have the glitz or "Frankenstein aura" of worrying about those computers that are taking over the world.
End of rant.
I was working for a desktop consulting company, and I was the only database developer there.
One of my customers wanted to convert a database, and originally I thought, no problem just convert some tables and redraw some forms.
It turns out this database was also going to store information about blood matching, transplants, and it would also calculate daily drug doses for the nurse to sign off on for kids getting marrow transplants. Success is measured in how many months the kid gets to live.
If I was working on a team using a more robust platform I might have had more confidence to push forward. However, this is Microsoft Access and i'm the only guy who would know how this thing would work. This means it would be very easy for some kid's death to point towards me.
So I quit.
By the way, if anyone has work for a database developer, feel free to contact me at will_spangler@juno.com. I'm quite good with MS Access.
The Underwriters Laboratories test electronic devices. Maybe certain fields should have their code reviewed and tested by an independent group before allowing it in the field.
I read this book a couple years ago, and it shares a few anecdotes about software bugs that really fouled things up. Nuclear reactors, airliners, and radiation therapy machines are given particular attention. Not a great read, but somewhat interesting.
Well if Diebold is anything to go on in terms of an average contracter, then basically we're all screwed - imagine planes, millitary hardware, brake systems, life support all designed by companies as incompetent as that whos only goal is profit.
This comment does not represent the views or opinions of the user.
Come to think of it, the power company never warned me that all that electricity the sell me could be dangerous... it was here when I got the house. Maybe it's in the fine print on my electric bill, but I think most of that is how they can track me down to collect money from me if I don't pay...
Seriously - I think it depends on the software. If you use consumer grade software with a EULA that says "this software is good for nothing, use at your own risk"... well fine.
I seriously doubt the firmware for an X-Ray machine has a EULA like that, but then, I didn't RTFA. I am sure critical software doesn't come with the usual global-deny-all-responsibility clauses.
Two final thoughts
For somethings, who would you trust more? Dr. McFeelgood with his finger on a button and a stopwatch, or software control?
If you can't mathematically prove the software is incorrect (by source code inspection), how can you prove it was a particular software in a multi-software environment, an intermittant hardware fault or a cosmic ray flipping the wrong bit?
This issue is a bit more complicated than you think.
Warfare is not about certainty, it's about playing the odds.
A very similar case could be constructed for them poor fried cancer patients. SQL databases that manage breast/cervical screening programs save thousands of people from cancer each year.
Engineering is the art of compromise.
Read 'The way of Z' by Jonathan Jacky. :)
It (briefly) covers the Therac-25 case and offers some interesting insights from someone who works in a field where software WILL kill if not done properly. You might find the rest of the book useful as well, as it covers formal methods, which surprisingly is a method/system for writing better software
the blind trust of its users.
I was thinking this same thing. As we are surrounded more and more by software, is there a point at which we should stop and ask if we can trust it? This not limited to trusting software with our lives directly, but even indirectly. Or what about trusting it with other areas of life (i.e., finances, other material objects)?
I was at Fry's Electronics once, and they had a fridge with a computer built-in displaying the BSOD. I got a laugh of telling the door clerk to reboot his fridge. But, had the computer been controlling the temperature it could have spoiled the food. Sure, most would notice, but not all do.
I love technology, but here needs to be times for to stop and reflect on how it can negatively impact ourselves also. Just my two... er, numerous words.
if I get my hands on some linux documentation writers I won't be responsible for my actions.
(module-init-tools-3 - doc writers, I WILL kill you all)
Sig: Closed for refurbishment.
I would say, it's whoever takes the responsibility to use said software & hardware where human life hangs in the balance. If you write a calculator program, and someone uses it to calcuate oxegen flow rates needed for a life support system, who's responsible when the calculator software fails, and the designer creates an unsafe device based on those calculations? Such as your gun/bullet analogy, the gun is no more responsible than the bullets. It's the individual who decides to put a victim's(as a mugger) or his own life(as a cop) on the line. I would hate to know that my life depends on a Win95 PC's ability to not blue screen. But then, if it was, and I died when it eventually crashed, it wouldn't be Microsoft's fault, it would be the person who decided to base such a critical device on that OS. Someone has to take responsibility for assuring that a device is safe for its purpose. This may be the programmer, the company that makes it, the purchaser, or the end user. This is the way it flows, because blame flows the other way, until it hits the gap where a requirement was not met.
www.facebook.com/DareDefendOurRights
www.fairtax.org
Excellent example! The Therac case is a real classic and highlights one of the common problems in programming -- the assumptions built into use cases. Too often, the creators of software assume that the user will progress through a preconceived set of steps. Variations from the programmer-assumed script often lead to corrupted states. Therac killed when the operators switched machine modes while the machine was in the middle of switching modes (yet the machien gave no indication of being left in an anomolous and danger state).
Although not lethal, this problem is why so many e-commerce sites suck. Hit the back button or try to change your order at the wrong point in the process and you garble the transaction. Most site creators assume that buyers will follow a linear script of actions (browse, ad-item, browse, add-item,...., checkout, fill-in shippng, hit OK) so that any attempt to go back or change something quickly puts the transaction outside the assumed use case.
Two wrongs don't make a right, but three lefts do.
But here's some more wood for the fire:
The QA guys probably did mention it, but development said it was by design and program management said they didn't have time to fix it.
Most likely the programmers didn't have any QA because they believe it to be a waste of time. After all, can't you just find the bugs during beta test? ;)
The next time you visit the doctor watch the workflow of the office staff. Increasingly chances are they will probably be entering your medical information, and I mean the clinical stuff, not your address into some type of computer system.
I currently work for a small Electronic Medical Records company. At some level I worry about potentially killing someone every day. In fact our bug tracking tool has a special category in it called "Patient Safety" which is the highest priority bug. We deal with things most of you probably wouldn't think of such as a tool for writing Prescriptions, which given the fact that many drugs interact ( potentially fatally) has to catch and alert the physician to such cases. I also deal with lab results which if reported incorrectly could lead to a potentially fatal decision by the doctor and so forth.
Consultants and pundits like to say that computer control reduces the chances of human error and failure, this is said IMO to comfort the masses. To state the obvious I suspect EVERYONE on Slashdot knows that in reality that statement is not true, the human error has just been moved to a different point in the chain. A tired programmer is just as likely to make a mistake as a tired machinery operator. The difference is that that software might be used by 5,000 machines, whearas that operator runs 1.
Really, between this current set of Theratronics accidents in Panama and the Therac-25 accidents, isn't it time that these monkeys got shut down? They have a history of irresponsible software development that leads to deaths and injuries. At the very least, they should be gutted for their technology and rebuilt from the ground up.
I can't really say more, but that article misses some major reasons for why that barracks was hit. Just don't take it as the complete story.
Didn't Steve Jobs once berate some Apple programmers once to shave down the boot time a couple seconds. He reasoned that if they multiplied the number of users who had to wait on that boot by the number of seconds they saved, they would "save lives." Is this true or just an urban myth?
postmodernsideshow.com
Namaste
I had to take an ethics in computing class my final semester in college. We learned about a few historical examples, Three Mile Island being the one that stuck wih me best. (Living in Harrisburg PA my whole life and being about 2 years old when it happened).
Apparently there was a valve that was supposed to open to allow water to cool the nuclear reactor. The software that monitored the the valve showed the position that it SHOULD have been in. But, as was the case in 1979, if the valve would get stuck the software would still indicate the valve was open/closed, even if it didn't happen. Yikes, could have made a lot of central PA uninhabitable....
There was an article like this that I got from a software design class in college last year. It was about 50 pages or so how cheapened hardware powered by bad software caused the deaths of many. They were cancer patients like these and the hardware to make it cheaper sesors that once did hard checks to make certain things were checked got moved to software checks. Some of the checks failed resulting in overexposure and radiation burns eventually death.
There exists some positive integer N that you are the Nth person to read this signature.
If that Patriot missile failure counts as a "software kill" then surely software does kill; Look at the amount of people killed in Iraq for example by different types of bombs and cruise missiles that are guided (and detonated) by software.
I think we can all agree that yes, software, or rather the result of erroneous software, can kill. However I think the more relevant question is should someone be held responsible if in case it does?
/. was that he deserved punishment. But the case would probably also be made that the original coder should have written code that was less vulnerable to exploitation. For something so important, the code should be guaranteed stable and reliable, right?
If software screws up an automated train switch and two trains collide, or if software used to regulate power transmission goes haywire and causes a blackout that results in a dozen deaths, is the software writer responsible? I say probably not. Obviously sheer negligence and less than satisfactory work would be grounds to punish someone, but that is still subjective. While the blame could be pointed at the coder that wrote the software, he would have as much reason to point to the company that designs power transmission systems and say "Their system should have physical backups to prevent blackouts", and he would be right. They should have physical backups.
Now let's say that someone writes a virus or some exploit that brings down someone else's code and causes a blackout. Should that virus writer be punished? The large majority would say yes. We just saw a story where a guy is getting charged with terrorism for "hacking" a few DirectTV boxes, and the consensus here on
I'm sure that in the case of a few people or a small company writing code, people would demand that they be held liable for writing exploitable code. Seems a bit hypocritical since most people would never blame MS for writing exploitable code.
My $0.02
programming error that resulted in 28 Panamanian cancer patients receiving many times an expected lethal dose
I've not seen a better argument for open source yet; it's the only known way of getting 100% of the bugs out, especially when the company behind the closed source code has a vested interest in covering the bugs up.
I just want to say software problems that kill people are not glitches, they are fuck-ups.
I cannot comment on this statistically, but it is my opinion that software where the source is open could well be safer.
For example, a climber assures his saftey by inspecting his or her rope before a climb. This is common sense.
Likewise, a pilot makes a huge effort to guage the airworthyness of his or her aircraft by way of a awalkaround and runup checks.
Surley the same could be said about software?
I always learned, both from Star Trek and all my college programming instructors, that programs don't make mistakes... people do. Binary is as simple as on-off, and putzy little electrons just do what they are told anyway. Of course, we only focus on the mistakes. Isn't it the same breed of programmers that digitally mapped DNA to create anti-Cancer treatments? (Oy, I hate when the meat puppets whine!)
"Would you rather be right, or happy?"
and it has always been due to human error."
-- HAL 9000, in "2001: A Space Odyssey"
-- Alastair
Try to repeat the incident under control of a debugger. However, you might have problems to find volunteers to take the roles of the killed people in the reproduction ...
The Tao of math: The numbers you can count are not the real numbers.
Example: a group of 9 '1's and one '10' has an average of 1.9 and 90% of the values are below that average. Or, take 9 '10's and one '1' and you have an average of 9.1 and 90% are above average.
"There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
When the database messed up some data, Vinnie though Sammy was trying to pull a fast one on him and whacked him to death with a heavy steel IBM keyboard. It was really a Visual Basic programming error I made, but the nice folks at the witness protection program say I shouldn't really talk about it.
Skynet. Not only is it a bad idea, it's inevitable. :)
Rember, software doesn't kill people, people kill people.
Never make any mistaeks. -- Anonymous, in a mail discussion about to a kernel bug report
It doesn't get much more funny than that.
500 dollar reward for tip(s) leading to the arrest of the person(s) who stole my sig.
...many high tech weapons work perfectly.
I am very small, utmostly microscopic.
By wasting our time and energy pursuing a fictitious case whose reported results were scientifically unrepeatable, Dateline NBC drew resources in terms of money, time, and mindshare away from very real life-threatening incidents. Can we seek the death penalty for TV shows?
I'm sorry Dave, but this mission is too important for me to let you, or anyone else for that matter, to compromise it.
killall -9 -1
Amen.
Here's the freaking answer to the question. Will somebody mod it up already!
Hackers Can Turn Your Home Computer Into a Bomb and Blow Your Family To Smithereens!
http://www.talknerdy.org
Programmers do
\m/
A few years ago I interviewed at a company developing radiation treatment machines. It was exciting stuff, the machine scans someone to build an exact map of where the cancer is, then generates a computer controlled treatment plan that maximizes the radiation delivered to the cancer while minimizing the amount to healthy cells. The end result would be that patients needing radiation treatment ended up killing less healthy cells while simultaneously killing more cancer cells. Great stuff. But just a tad bit daunting as a software developer. The entire system was computer controlled. The clever algorithms that optimized the irradiation weren't obvious, a tech running the machine would be hard pressed to identify an bad or mis-applied plan. Potentially the difference between life and death was entirely within the hands of the software. I wasn't sure at the time if I was ready for that.
Search 2010 Gen Con events
I think that most people in the industry understand that software quality can be a life-or-death problem. But, as the Eweek article states, the experts in software QA know that "perfection is impossible". Even Slashdot knows it - the random quote of the page I got while writing this reply was:
:) ).
Never make any mistaeks. -- Anonymous, in a mail discussion about to a kernel bug report
Given that fact, what steps can we - the individual programmer - take NOW to help the situation? I say "the individual programmer" because one person usually can't change a whole project's development methodology (or switch the project over to *Ada
*Caveat: I've never used Ada myself, and yes, I know it's got a rep as a real BDSM language, but some respected friends and colleagues who've worked on DoD projects tell me that it does help the programmer avoid many self-inflicted foot wounds.
"If they send someone here, I'll arrange the usual 'accident.'" -- Alice, "Dilbert"
The saddest part of the story to me was the Multidata spokesperson, Conley, saying that the FDA's finding "is wrong." "Given [the input] that was given," he says, "our system calculated the correct amount, the correct dose. It was an unexpected result."
In other words: "functions as designed."
In a strictly technical sense, Multidata might be right. Operators were actually trying to increase safety by adding an additional shielding block. In so doing, they encountered a problem in the software design and created a workaround--a creative way of entering the data--which seemed valid, and which the software did not flag as an error. To me, it is not at all clear from the story whether the other way of entering the data was properly within the scope of the software's spec or not.
But I really wish Multidata had chosen to acknowledge some responsibility for the outcome.
"How to Do Nothing," kids activities, back in print!
Spell cheek you've failed me four the last thyme!
It is evil and sadistic when it kills.. See the movie TRON as an accurate example.
"BEHOLD, CORN!!" - Dr. Weird, ATHF
http://courses.cs.vt.edu/~cs3604/lib/Safety/therac 25.html
"In the end, the massive design flaws resulted in the death or injury to six people receiving treatment for cancer. The costs, it seems, for safeguards independent of the Therac-25 were far too much to be considered for use in the final product.
After the July 26, 1985 incident at the Ontario Cancer Foundation in Hamilton, the manufacturer could not reproduce the problem and ultimately The overdoses have generally been attributed to the flaws in the software that would allow operators to override errors that would arise, many fatal to those patients being treated. The amount of the overdose was, more often than not, many times more that the recommended therapeutic dose that eventually culminated in severe trauma or death."
Most people in the comments are focusing on actual bugs and crashes in a system causing deaths. While that could certainly happen, those types of errors are more visible and actually a much "better" error to have than some other types. If the system crashes, it may have some immediate effects depending on its purpose, but if it's something that causes its action through an actual user, they are generally harmless, though very annoying. An example of the difference is that if the software designed to run a ventilator has a bug that causes it to crash, since it is directly providing life to a person, when it crashes someone will probably die. On the other hand, systems designed to give information to a clinician, who can then act upon it are going to be very aware when that system is down, and so much less likely to make an error based on that outage.
The more insidious "errors" if you want to call them that are ones that are errors of design and process, and not execution. If a piece of software is designed with certain assumptions in mind, and something happens outside of the parameters of those assumptions, the software will appear to be working correctly when in fact there may be egregious errors. There are a lot of instances of this in everyday practice.
Lastly, what we run across is that clinicians are used to a world of paper, where everything obviously either there or not. You know that there's a problem, and there is transparency to the error, so you can factor that into your decision-making. In a clinical system, the transparency is not there, and a subtle flaw can mislead someone making a clinical decision into making a poor one.
Of course, the above are all gross generalities, as is any discussion of errors in complex systems, but I hope you get the idea.
Software is like any other engineering materials; if it is improperly used then it at best it may not work as intended.
There are different software requirements for different applications. Simple things like action of an emgergy stop has different meanings in different applications; halting a production line if there is a dangerous situation in a factory is a good thing, turning off all the engines on a aircraft in flight is not!
The problem begins when the person developing the software either does not talk to the designer of the hardware, or cannot understand what the hardware does. While the engine controller of a jet engine may have enough processing power to write a good video game, programming it in as a "easter egg" within the flight software is an extremely bad idea.
Having said that, it is also obvious that software that is to be used in mission critical systems, where failures would put people at risk; must be certified accordingly.
Take for instance the stringent testing and certification rules for software to be used in NASA's space shuttles. Write Stuff
There is, however, a steep price to be paid. Writing bug-free software is very expensive, and only those that cannot live with the possibility of failure, have to shell out the cash.
http://catless.ncl.ac.uk/Risks/
http://courses.cs.vt.edu/~cs3604/lib/Therac_25/The rac_1.html.
I find it funny that the article talks only about the Panamanian's when this happened in Canada and the States.
Sorry, teleporters just kill you and then make a copy. A perfect, soul-less copy.
mod parent up
On the other hand, if you're working with a closed loop system (outputs to equipment and feedbacks to read the state of the equipment), you have a lot more responsibility as a programmer. You should be able to check for most equipment failures (if the feedbacks work correctly) and disable the outputs in the case of a problem. The radiation machine in Panama should have feedbacks to control the output, and those feedbacks could be checked against a "lethal dose" value. If they're not, it's the programmers (and/or designers) fault IMHO. But if the people operating the equipment change the software (disable a sensor...etc) then it's all in their hands. Any modifications outside the original design should release the programmer against any liability because you can't predict changes not made by you.
Again, from the article:
Every example given was life threatening, yet the author clearly wants you to draw the conclusion that a software author should hesitate to publish a program she wrote to perform a calculation because someone *who thought they knew what they were doing* might plug it into a lethal machine.Next we will be hearing about how someone wrote a spreadsheet in gnumeric to calculate the radiation dose, killed someone because of a bug in gnumeric, and how the authors of gnumeric should be ashamed of themselves, and not the asshole who *thought he knew what he was doing.*
Special Superior Deputy Prosecutor Cristobal Arboleda, unlike the author of the article, is accusing the right people and doing his job well.
I remember reading an article about how the safety system in a plane caused it to crash. It was designed so that if the plane went over a certain height then it would immediately drop to a safe height.
Unfortunately when it was landing in the Netherlands, the run way was below sea level. When the variable storing the height dropped below 0 it wrapped back around and so believed it was above its safety height, thus automatically plumbeting itself in to the ground.
to mourn the loss of time and brain cells for all those who were forced (tricked?) into reading this book, whether in a required computer science course, or not.
If you haven't had occasion to read this, ahem, "literature", consider yourself lucky.
That green slime had it coming.
While I don't know anything about the software failure of the Patriot, I do know that the article about it starts off with assumptions that are blatent falshoods, possibly even urban legend material, which leads me to distrust the rest of the article.
The original patriot was not designed to intercepts missles, nor anything at all. It was designed to get near a target with air-breathing engines, like a fighter jet, and detonate thus filling the engine with shrapnel and possibly damaging the aircraft body.
It was then pressed into service as a missle interceptor, something Ratheon argued was impossible right up to the day when Lockheed's Patriot Advanced Concept 3 (PAC3, which the press also calls "the patriot) intercepted one about 5 years later. Unlike the Patriot, the PAC3 has no warhead -- it uses kinetic energy to vaporize the target and all it's contents. Today, longer range anti-missles like ERINT and others build on this concept.
So why didn't the US admit to the panicky citizens that we couldn't stop a missle? Why didn't the US just announce to Saddam that we were powerless to stop a chemical attack on Israel?
If you have to ask, thank God you aren't in the military.
people acting in a negligent (or intentional) way causes death -- not inanimate objects like cars, guns, and software. Put the blame on the right factor. Poorly designed roads and poorly designed software can both end up causing human death, but the key is that these things were poorly designed by humans. Negligence. Not acting as a reasonably prudent person in similar circumstances. Don't blame the thing, blame the people who made/designed/controlled the thing. Come on, slashdot is better than this.
Stupid people make stupid things profitable.
The Patriot missile system failure did not cause the deaths. The SCUD missile caused the deaths. The Patriot shield failed, yes it was a serious failure but let's not get confused as to cause & effect here.
You wrote:
"If you're ripping around at 150mph on non Z-rated tired, and one blows, it's your own damned fault, not that of the manufacturer."
The problem with your analogy is that tire manufacturers are required to specify the maximum speed rating of their tires between N (87 mph) and Y (186 mph). Note: Z is now effectively a dead designation.
The DOT/NHTSA does not allow Firestone to put a disclaimer on their tires saying that they can't be used when the failure could cause injury or death. And neither should the feds allow software publishers to do that. Specify the limitations of the product, but don't try to weasel out of liability by putting vague warnings about the use in life-critical applications. If it fails to work as advertised, then the publisher should be fully liable, regardless of how someone else used the software.
Do that, and you'll see Microsoft investing a lot more in making a stable OS rather than making bad video editors and sub-par instant messaging clients to bundle with each OS.
They may not be the "code" that most of you know but it's an interesting story...
I've seen code on PLC's and embedded microcontrolers used in industrial safety applications that could have been lethal or at the very least, a finger eater. For about four years, I worked for a company that manufactured light curtains, hardguards and other devices that keep machinery operators from losing limbs.
The most dangerous "code" I've seen was on a large metal stamping machine. The maintenance department was using a PLC to take the anti-tie-down inputs and start the stamping process. The operators had tied one of the palm buttons down with black tape so they could hold the metal with one hand and start the process with the other. The PLC was also counting the inputs and cycling the machine to satisfy the counts whether the operators wanted it or not. Anyway, to make a long story short, our company was called in to install a light curtain and pitch all the other "safety" junk. The techs that mounted the curtain found a part of a hand behind the old physical guard.
Of course software can kill. Just yesterday software reached out with it's cold clammy fingers and killed a gamer in China.
In modern systems, where IMRT (intensity modulated radiation therapy) is used, the medical physicist in charge is supposed to verify each field delivered. This tests both the treatment planning software, as well as the accelerator, collimator leaves, etc. Often this is done using film, however, time saving electronic devices (basically diode arrays) are used by those who can afford them. Of course, the verification system has its own software, which requires verification also. Luckily the verification software is fairly simple, relative to the treatment planning and accelerator control systems.
I always had high regard for comp.risks, but I'm wondering if I should reconsider, seeing that they've stooped to quoting slashdot!
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
"...and the National Rifle Association says, ah, that guns don't kill people; people do. But I think the gun helps."
-Eddie Izzard
When was the last time you saw a military contractor face any penalties for incompetence/waste/negligence?
The US Military effectively has complete immunity from lawsuits related to loss of life or injury sustained by serving soldiers, and this gives a blanket shield to military contractors as well.
The military procurement system is so screwed-up that deaths from equipment failure don't even register... because nobody involved has to take responsibility for them. V-22 Osprey, anyone?
"We have to go forth and crush every world view that doesn't believe in tolerance and free speech." - David Brin
A software glitch caused the crash of the first F-22 prototype (noone died fortunately) and as someone else pointed out, the Patriot missile failure in the first Gulf War (Software wasn't ENTIRELY to blame there. The bug was known and the folks in the field were given instructions on how to avoid it, but didn't follow them)
The trick is who do you hold responsible? The software person who has no training in mission critical software and who's working 80 hour weeks to meet the deadline the idiot managers are shoving down vis throat?
After 10 years in the industry I'm FINALLY starting to see movement towards software creation as a serious engineering discipline. Schools are starting to offer programs in Software Engineering, the ACM and IEEE have agreed on an official code of conduct (tho IHMO it still has SERIOUS problems), and most importantly companies are starting to listen to their technial folks when they say "You can't do that!".
Liability is just another incentive to head down that road. We need to be sure we pin the liability on the right folks.
Isn't that the same question?
... the programmer giveth then taketh away.
However, you might have problems to find volunteers to take the roles of the killed people in the reproduction ...
Enter the crash test dummy plus radiation meter.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
They were tampering with the internals of equipment they didn't design or test. They are shining examples of why the FCC has such stringent guidelines for medical hardware in the US. Heck, here in the US they'd be facing jail simply for admitting they tampered with the machine and used unapproved settings!
In a former life ( :-> ) I was employed by a large multi-national that worked with utilities. Some of our software used SCADA protocols to remotely switch breakers - not household breakers, these switches control significant segments of the US power grid.
All the software and documentation contained numerous warnings, because if a utility employee manually switched of a segment to make repairs, and switch was remotely turned on, someone could be killed.
There are numerous other software applications that control (potentially) deadly devices - robots, industrial equipment, etc. Failure of the software, or problems with operator headspace, create a potential for death when working with almost any software that controls physical entities.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
What about when meteorites kill people?
We write software for railroad traffic control and crossing warning systems. If it fails we could end up with two trains trying to occupy the same piece of track at the same time (ref. Clapham Junction 35 dead) or gates that stay up when the train comes. Testing is very formal and rigorous and every step is documented.
For every hour we spend making sure the system does what it's supposed to do, we spend eight hours making sure it doesn't do what it's not supposed to do.
None of them can see the clouds; The polished wings don't care.
GPL'ed software would NEVER have been used for this in the first place, whether or not it was better.
Read Paragraphs 11 and 12 of the GPL. There is NO WARRANTY. Nobody in their right mind would use software without any form of warranty in a situation where lives can be lost
I am Monkey, the Great Sage, equal of heaven!
The moral of the story, software safe, hardware dangerous, powered hardware running software - teotwawki.
I'm not saying that software is never on time or reasonably error free. I am saying that software is so often late and error prone that it cannot fairly be described as engineering. My favorite analogy is the state of cathedral building in the middle ages. They would make their best guess and then start building. If they guessed wrong then the building would fall down, often before it was done, and they would change the plan. This is how we do software today. Despite the hype, no "software engineering" technique has achieved the repeatability standards of civil, electrical or chemical engineering. We're not even close.
Slashdot had a discussion about Programming Gone Wrong in the past.
It mentioned, among others, the Ariane 5 Failure, the infamous Therac-25 accidents, loss of Mars Orbiter, Hi-tech toilet swallowing woman, AT&T Switch failure, a bunch of things literally crashing, etc. And here is yet another article on miserable Patriot failure.
For professional assessment of risks, there is a Usenet group for RISKS Digest (Google groups) that describes all kinds of situations where technology has gone wrong.
Future Wiki -- If you don't think about the future, you cannot have one.
Dr. Pib? Mr. Pibb got jealous of Dr. Pepper and went to med school too?
All employees must wash hands before seeking equitable relief.
Software kills like guns do.Someone had to pull the trigger. Software is only as good as the way it is writen
...the software controlling them do!
For any Software Engineer or architect, read "Fatal Defect", this is old news for the article.
It's scary what no documentation and an upgrade can do. Stories of Nuclear Power Plant control and radiation systems.
But I read this a few years ago, so this isn't anything new.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
It is recorded that the ancient Roman generals required the engineers
to stand under the bridge as the army marched across it.
What would happen if software engineers had to stand behind
their software in a similar way?
It's not that software kills, just that people die. It's an unavoidable consequence of being alive. Sometimes things happen that you can't do anything about. Accept it and move on.
Je fume. Tu fumes. Nous fûmes!
Freedom isn't free; its price is the well-being of others.
there is a whole ton of law on this. there are many considerations, but as a general rule you can't protect yourself from all liability by burying a clause in a contract. it is a question of policy how much latitude to allow. unsurprisingly, trying to escape liability for personal injury is looked upon more skeptical than for economic losses.
there is a general reluctance to allow liability for gross negligence or intentional misconduct to be bargained away, as opposed to ordinary negligence
food for example carries strict liability -- all you need to show in that the food hurt you, not that the supplier did anything wrong.
there are also the warranties of fitness and merchantability, meaning basically that the product should do what you say or imply it does.
Four bytes @ 7 bits is enough to store the state of 28 humans: dead(0) or alive(1).
Opportunity knocks. Karma hunts you down.
they arent immune to bad press, which usually caused congressional investigations. Believe me, the military is paranoid about how powdered its nose is in the public media.
It's the profit motive.
It's the rush to get a product out the door.
It's the refusal to allow a programmer to hold up the delivery of a [product while looking for an error that he is not sure is there.
It's the idea that reducing the bottom line is an allowable business practice when working on critical systems.
It's the idea that programmers are an expense rather than an asset.
It's the practice of hiring accountants to calculate "acceptable risk" from wrongful death lawsuits while making engineering decisions.
There are many many places in the software development world where profit is a truly acceptable basis for making decisions, but there are clearly situations where profits should be secondary, and it is difficult to bring a business culture around to acknowledging when and why profit must take a back seat. Cheaper is not always better. Computerized is not always superior (abs systems). "Time to Market" is almost never a correct way to determine when a project is ready for release.
For-profits will almost always be more concerned about return on investment than they will be about the quality and saftey of a product, especially when they are prodicing goods in a market with a limited number of suppliers (specialty medical equipment), or an outright "granted monopoly" (military defense manufacturing). In those cases there is litle motivation to provide the better product and there is even less opportunity for the client to be sure that they are getting what they've paid for.
But it's OK, cause the stock valuation just increased, and we all know that that's the end-all, beat-all, purpose of our existance.
Read, L
From the article: Otherwise, unanticipated uses will lead to unintended consequences.
I for one welcome our new software overlords!
Flying is easy, just throw yourself at the ground and miss. -Douglas Adams
Ones who expect to predict and eliminate every possible point of failure.
One (and two, and even a team) just can't do that!
Thus, we'll always have software glitches with a varied level of problems caused by these glitches.
Tigers respect lions, elephants and hippos. Maggots respect no one. (C) S. Dovlatov
Don't blame us, it's the compiler.
Learn something new.
The article on the Patriot failure is interesting. However, I'm having trouble picturing how cumulative clock drift over days of operation could cause cumulatively increasing error in a range-gate check. After all, range-gating is done by taking t0 at the initial contact point, and then waiting delta-t before taking the t1 observation. Absolute time doesn't figure into it at all.
Does anyone have any insight on this?
When all you have is a hammer, everything looks like a skull.
Dr. Nancy Leveson, cited in the original article, has a book called "Safeware" which discusses the Therac tragedies and others.
She argues that the Therac incidents were unusual in being caused by bugs. Most horrible accidents, in her analysis, involved software doing exactly what the user asked for.
She teaches viewing real world problems as intersecting sets of circular dependencies (feedback loops), instead of neat linear pipelines. For instance: if you put a cooling unit into a room, will the room cool down? Not if you put the cooling unit next to the thermostat. No amount of QA on the cooling unit will save you from mishaps like that.
Looking at the world from her point of view, you might fix problems like the radiation overdoses by putting a dosimeter on the patient's body that would cut the radiation unit's power supply when it started overdosing. Simple, direct, controls the problem.
Another relevant feedback loop is one for humans to use for detecting chronic problems. Therac didn't have a good reporting/logging system for safety-related incidents. If they'd known there was a trend of patients getting zapped, they might have acted sooner or more appropriately.
Boiling her message down to a single badly distorted sentence, "If you're not looking at it, it's out of control."
have epillepsy.
Software doesn't kill people. People kill people.
mix_master_mike
vafrous
Unfortunately, the proper answer to that question is "Yes, it most certainly can." That is why before testing or reconfiguring, you should always mount a scratch monkey.
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."
perhaps they need a QC
but please dont blame the programmer
That's documenation.
Now, if course, if you broke the CDs up, and then taped them onto the books so they had sharp pointy bits, then maybe you could claim software.
Build it, and they will come^Hplain.
Read the risks of computer systems..
http://catless.ncl.ac.uk/Risks
The newsletter has been going out for over 15 years now. The idea of software programmers not having the big picture view of all the interrelated components of a complex system isn't new. That's why system engineering is important, why engineers do Failure Modes and Effects Analysis (FMEA) and why big complex things are done by teams of multi-disciplined people. When you put a whole pile of software specialists in a room you get things like Windows...
brainwashed humans turned into machines
depleted of all humanity, they crash on demand
Kibo's Fake Dr Pepper Roundup
{Note: There is no period in "Dr Pepper"}
I know that someone mentioned it already but there was a good paper published on it, it's a good read. There was a similar incident a few years ago with the Therac-25 machine. Supposedly the manufacturer installed a computer and removed hard wired safetys out of the system, running everything by software. They also forgot to put in some interlocks and other good stuff, the net result being that several people died. Or it seems that there is a book here although I have not read it myself.
A lot of people forget that software can have devastating affects on things and even with the best of programming you can't beat a hardware safety for the "just in case."
I took basically the same class at UC Berkeley. It must be a UC thing.
Only kills when 28 users are vulnerable. Soon to be featured in the documentary "When Software Attacks".
Come on /. editors, you've got to try harder this has to be the lamest headline ever. Of course software can kill, its just another failure mode for hardware.
Bitter and proud of it.
As tragic as it is, the Panama incident does not stand alone. In all, Baseline has found no fewer than a half-dozen cases in which software has contributed to loss of life.
I'm surprised that they only found a half-dozen cases in which software contributed to loss of life. Back in the old days when I used to subscribe to the comp.risks digest it used to seem that every couple months there was some fatality or near fatality that could be traced to flawed software. If anything, given the increasing use of embedded systems in our society, I would have thought that the fatality rate would have increased.
I can see two possible outcomes:
- The software engineers get given the time and resources they need, and high-quality software (eventually) results. (I believe this already happens in certain high-risk environments.)
- Management sees similar software getting made for less, calls the estimates bloated, and insists on getting it made quicker and cheaper. 99% of the time, they'll seem right; the other 1%, they'll blame the engineers for not doing their job properly.
Which do you think more likely?Ceterum censeo subscriptionem esse delendam.
" Software doesn't kill people; programmers kill people."
And we would have gotten away with it, if it wasn't for you meddling kids.
and take it from an old embedded systems programmer, bits get flipped and shit happens.
anyone play .hack?
:P
nuff said
If this were a virus
You would be dead now
Fortunately it's not
Slashdot is a dangerous place;
How's your security?
Call Hiro Protagonist Security Associates
For a free initial consultation.
--Stephen
Deep structures, anyone?
Did you ever notice that *nix doesn't even cover Linux?
You better believe it can.
ôó
for longer than there's been a slashdot. Of course it kills.
I write software for automated train switches. If a bug in my software caused an accident, I would most certainly be responsible.
Railway traffic control software is very formally and rigorously tested. The only way I would not be responsible is if the tester neglected to perform a test that would have discovered the bug.
I believe no discussion of deadly software disasters is complete without mentioning the London Ambulance Service Disaster of '92. Basically, bad project management and other gaffes lead to the Londom Ambulance Service using a computerised dispatch system which was not up to the job. It promptly crashed, and quite a few people died as a result.
I thought if there was any risk of human injury, the expense of applying Formal Methods was justified.
Formal Methods is the application of Mathematical Proofs to software.
FM can actually eliminate all Functional risk,
which still leaves Integration risk (interfaces with outside systems), and Environmental risk (e.g. differences between development and production).
I wonder if these companies used FM, and to what degree. The cost of FM is horrendously expensive, so I could see them skimping where they thought they could get away with it.
I also thought FM was government mandated for Military & Health systems.
Over the last decade greedy contractors in Iran decided to use less steel in reinforced concrete high-rise buildings than they knew they were supposed to. After the earthquake, the contractors were rounded up and put in prison for murder.
Not software engineers, but "Systems Engineers" have the position of the Civil Engineer and the Contractor in the building of a computerised system.
http://www.ucl.ac.uk/syseng/pages/sedef.html
"Systems engineering is the branch of engineering concerned with the development of large and complex systems, where a system is understood to be an assembly or combination of interrelated elements or parts working together toward a common objective."
"Systems engineering focuses on: the real-world goals for, services provided by, and constraints on such systems; the precise specification of system structure and behaviour, and the implementation of these specifications; the activities required in order to develop an assurance that the specifications and real-world goals have been met; the evolution of such systems over time and across system families. It is also concerned with the processes, methods and tools for the development of systems in an economic and timely manner."
For an example consider the Challenger Space Shuttle, which self-destructed in the late 1980's. The avionics system consisted of sensors, actuators, computers, and code. These systems were built by Software Engineers, EE's, Physicists, Mechanical Engineers, and Systems Engineers.
An O ring on an engine failed so that one engine started loosing thrust and the Shuttle started veering off course. The sensors notified the software, and the software decided to increase the thrust. And so on, in a postive feedback fashion. The Software Engineer could have noticed the feedback problem, and might have, but there was no provision in the System for making any other decision but to press on. The Shuttles do not have an abort and crew recovery feature.
Who killed the crew? The one responsible for the overall system is not the Software Engineer, but the Systems Engineer. Althought the Systems Engineers probably argued for a recovery system and did not get one approved by management, they still officially bear the responsibility (they should have quit.)
Worrying, complaining, arguing, studying, and quitting is what systems engineers do.
(This is sour grapes; I'm unemployed right now because -- you know -- I quit...)
http://www.onlinelawyersource.com/personal_injury/ pl/slindex.html
Strict liability mandates that responsibility for some accidents automatically rests with the defendant rather than the plaintiff. Strict liability laws shift the burden of proof, increasing the social accountability of the defendant. Situations requiring strict liability include blasting and keeping dangerous animals. Under the regulations of strict liability, anyone who engages in an activity known to endanger others assumes responsibility for resulting damages.
Strict liability applies regardless of any precautionary measures taken by the defendant. For instance, a strict liability case exists if blasting damages property-regardless of any measures the blasting company took. Under strict liability law, some activities carry such high risk of damage that responsibility for the outcome is absolute. Strict liability in product law, for instance, holds accountable everyone from the manufacturer to the reseller for dangerous products. Strict liability on the part of everyone who profits from the product increases consumer safety. In a strict liability case, if a product injures someone, even if the injury results from improper use, the strict liability for the injury rests with the manufacturer, wholesaler, and reseller, not the consumer. Strict liability assumes that the producers and sellers of products anticipate any injury from its use or misuse.
See this article for examples.
A different title or a different curriculum takes us no closer to reliable software.
First, there is no proven way to design applications correctly or to write good code. We have formal methods, CMM, and ISO, but none of those can guarantee that the whole system works as expected. Even if it works as expected, it may still not be "correct."
Second, even if we teach people to use better tools, or to use tools better, every software project is a group effort. Ususally these groups are very diverse, bringing together people with different skill levels and experience. What good would it be if we had a "Sofware Enginner" signing the project if one incorrect line of code could make the whole system crash? Would you sign a "Software Project" and be personally liable for it?
If methods and tools can't make more reliable software, then what the hell can? If Engineers don't make reliable stuff because of their curriculum, then what is it that makes them do it?
The job of the chief engineer is to ensure that the product is developed properly, beyond a reasonable doubt. If that means they have to write every line of code themselves, then I guess we'll just have to write mission-critical software in high-level languages.
If I was confident in my team's abilities and used formal methods without cutting corners, you're damn right I'd accept liability -- provided I got paid the way PEngs do, at least.
Most of the admin systems use Windows NT. Much of the message traffic is interface with Windows NT (though Outlook). Chat is used a lot now between ships for basic coordination. However, there are still plenty of comms systems (mostly voice) that go through their own old school, propietary systems. Navigation is being done more with electonic plotting on Windows computers, however the QM's still plot primarily on paper charts. Heck, most still do periodic celestial shots to compare to GPS data. The critical weapons, sensors and engineering systems are mostly run on older, non-Windows systems. There are some newer systems that are being tried with off the shelf components using Windows, but all of these have backups. The Navy has learned to have a back up system for everything.
The headline was "Two spammers killed". The very first comments were "Which software does this?"
When millions of people get treated with a software based cancer treatment, a minor bug may kill thousands and no one will notice. In these cases there is no death due to the treatment, but because of worse than optimal treatment quality. Still, the latest optimization systems (IMRT) have already saved tens of thousands of people that could not be treated using manual techniques. Even if someone dies because of bad software, there is still enough quality software around to make it is an overall lifesaver.
-- Imperial units must die --
When you Pay 100000$ for software, the companies say in their EULA that they will be responsible for losses. In chip design the EDA Software vendors charge a lot per license but if your chip screws up due to faliure, they pay.
My Aurora : http://www.youtube.com/watch?v=o91ZsGwJYyg
FB : https://www.facebook.com/TanveersPhotography
That's why big pharma companies spend *millions* on validation and testing, as required by the FDA.
Do we really need to ask this question?
At the same time I have seen an app recently written just a few years ago that used floats for stock exchange transactions. Also a double-plus ungood situation. Hell, we were taught at school that floats were approximations and only good for a certain number of digits, and the accuracy can collapse over repeated calculations.
See my journal, I write things there
Drugs get administered based on the information in those records. A good doctor will always double check by asking the patient about allergies before prescribing. Of course if the patient has been admitted unconscious after a road accident and the data dowloaded from their medical insurance card, that's a little difficult.
See my journal, I write things there
I've been working in a similar area (in hardware) to prove that our designs work as expected with limitted success. ... Ok, I lied... Lack of success is more appropriate. So I switched to just teaching it. ;-P
Software is an unprofessional engineering field. When someone created a software, s/he wrote a disclaimer right away. I have not seen a disclaimer in a bridge / building written by a civil engineer.
-- br
ninjas kill people!
I work at a place with a large ammonia refrigeration system. (Ammonia is a toxic chemical with an extremely pungent smell. 3ppm is detectable by humans. If inhaled it can cause death by tissue damage to the lungs and contact with the liquid form will 'burn' the skin and eyes. We have a very rigorous OSHA mandated program for minimizing the risk of ammonia release)
A bug in a blast freezer control program sent a slug of vapor propelled liquid ammonia down a stretch of pipe bursting the end cap and sending ammonia steaming out into the plant. Fortunately the usually busy area was vacant. Had any person been standing in the area it most certainly would have killed them.
Simple people talk of people, better people talk of events, great people talk of ideas.
Plain and simple: It's Skynet. We're all dead, we just don't know it yet. Nice knowing you!
The Satori Effect
Joe Ganley
This course is interesting in that it was a bit like the "civics" course in Heinlein's "Starship Troopers"; you don't get a grade for it (it's pass/fail), and it's required, but it's mostly discussion and reading.
Nathan's blog
I think the other show was about software piracy on the TRS-80.
Fnord.
Please mod parent up: +5, Funny!