Slashdot Mirror
When Bad Software Can Kill
bhoman writes "A wrist computer that tracks and calculates safe diving times and
limits for SCUBA divers had a dangerous software bug that may have been covered
up by company executives.
This SF Chronicle Article
details the problem, product, company, and some of the lawsuits.
According to the Chron article, company execs tried to cover up and
deny the problem for years, but their
official website
makes it look like they did a voluntary recall."
And it has extensive safety measures built into it to prevent insulin over delivery. Obviously, when you put your life in the hands of a machine, you want to make sure it works, and that when it doesn't, you're notified. If a company is guilty of covering up a problem like this, I hope they get sued out of existence and the people guilty spend some time in jail.
There are two major diving computer companies with "original" systems-- UWATEC and SUUNTO. Uwatec (named in the suit) has been known for less conservative systems; they let a diver stay down longer.
This is attractive to people who do decompression diving, because it means that they don't have to hang out shivering at 5-10m with nothing to see as long at the end of the dive.
Suunto takes a different approach, has a more conservative model, and makes it easier to force your computer to be more conservative still. Most divers don't use that function, because it is contrary to their desire to have maximum bottom time.
Proper diving procedures recommend using two different computers, and always relying on the more conservative unit for your decompression limits. (Assuming that you are doing a computer-only dive and not a table dive.) When your life is at stake, you have to assume that equipment has problems, and act accordingly.
I had a friend in the US who underwent LASIK surgery. He told me that his wife, who was computer-savvy, and was watching him being operated on, saw a Win95 box dedicated to controlling the laser and the mount's stepper motors, and that the operator was repeatedly hitting ENTER to make that recurring message box with a red X disappear. She got worried but the surgery was already under way, so she didn't say anything.
...
Fortunately, his LASIK succeeded. Later on however, he went back to the hospital and asked about the operator's behaviour : the response was "well, we were worried at first, but that error message comes back every five minutes and the machine always works anyway".
Scary
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I have dealth with Healthcare Software for Pharmacy and Lab systems where a delay or missed processing of an order can be fatal to a patient. One thing I found before leaving that industry was that there was a massive migration of these systems from reliable-high uptime servers (VMS, Unix, Mainframe) to Windows client server enviroment. If you think that the Klez virus is bad in a regular office, try working in an enviroment where it brings down a server critical to patient care.
For this specific problem, it was more that they modified an existing product to get it to market (in order to sell the company).
Your nitrogen uptake in diving is a function of the nitrogen partial pressure (79% in air, 60-75% with oxygen enriched air (nitrox). Making a computer calculate based on nitrox rather than air should be as simple as changing the O2 percentage. The problem is that at the surface, this computer still assumed you were still breathing O2 enriched air.
Good diving practices dictate a minimum 1-hour surface interval, and 24-hours before flying when using nitrox. Professional divers often push these limits, especially at the end of a trip when it is most dangerous.
Oooooh, you're so wrong!
You can prove the correctness of a bit of code, but it's very hard, takes a long time and is highly skilled work. It becomes especially hard if you're trying to do it with a grammar for any real-world programming language and the code is anything approaching complex. For most real world cases it's simply imppossible.
Mathematical proof of a program's correctness is simply too hard and costs too much money to be applicable in the real world.
If you did a real CompSci course, such as the one I did at Cambridge University, you'll discover that "CS people" are very very far from being "code nerds". I was supervised by a couple of mathematicians for some courses who could code no Java, C++ or Perl (although one of them knew much ML). Proper Computer Science folk are seriously academic and embrace the mathematical side of the field. You can't write an optimising compiler without doing so, to name but one thing.
People don't "ignore this issue" - it's just virtually inapplicable to real world problems. Exteme programming is not "management rubbish". If you'd ever actually read a book about it and tried some of the methodology you'd appreciate that. You forget that the driving force for commerical products is pretty much how much it costs, against the feature set and speed. Provided it doesn't crash very often, that's Good Enough. Unless you're the sort of type who doesn't pay for Windows, I'd suspect most people would rather have a version for $100 that crashes once a month than a version for $1000 that crashes only once a year.
Get down off your pretentious high horse and get a clue.
Six or seven years ago, I worked with a fellow with the very British
name of Ken Appleby. He had a Spitfire, I had my '74 B, and we used
to motor out to Pickwick's Pub and throw darts after work on occasion.
Ken used to work for Lucas in the UK, specifically for a division
of Lucas that did military electronics. My favorite of his stories
was about the time he had been working on a computer-controlled
torpedo. It used magnetic core memory to store the programs, which
had the advantage of being very non-volatile as well as not susceptible
to EMP discharge.
So Ken got to ride on the boat for the first test of the torpedo that
used the computer with his program in it. Somewhere out in the North
Sea, on an R. N. cutter, Ken and his crew launched the first ever run
of this new weapon, and Ken learned a new respect for debugging...
The program was supposed to make the torpedo shoot off the boat, dive
to a depth at which it couldn't be easily detected, then circle
toward the target, climb to striking depth, and hit the target. There
were on-board sensors to detect sea level, and the torpedo was supposed
to travel at a preset distance below sea level, with constant feedback
keeping it on track.
Somehow, somewhere, Ken had multiplied one of the 3D coordinates by
a negative number, and this error soon propagated through the
transformation matrix (the mathematical construct that models 3D
space), with predictable results.
Within instants of hitting the water, the torpedo -- instead of
sinking out of visible range -- blasted up and out from the water in
a great silver fountain, then continued skipping across the surface of
the blue like some sort of deranged wingless flying fish. Worse yet,
instead of circling toward the target, it circled all right, but began
to return to the ship that launched it. Fortunately it was not armed,
but they still detonated the self-destruct on it rather than let it
slice through their ship at 50 knots or whatever rate it travelled.
Because of the non-volatile core memory, Ken was able to debug the
program from what the Royal Navy frogmen could recover from it, and
he fixed the problem for Rev 2.0.
But I must admit that the image of the torpedo, splashing happily
above the surface of the water like an aroused porpoise, is one that
returns to me in idle moments such this. What else would a Lucas
torpedo do but try to fly?
I had a scuba instructor for my first certification, Di Dieter, who had experience diving with Coustou (hope I spelled that right), he also dove the Andrea Doria on several occasions, and basically has been around. I'm sure he's had thousands of dives, perhaps approching or even exceeding the ten thousand mark, under his belt (close to forty years of diving, multiple daily dives, several hundred dives a year, including a grueling dive schedule with Coustou, and he's a dive instructor). He's a no-nonsense guy with a good dose of common sense, and has little patience with screwups.
He did it right. He taught us to dive the navy dive tables, one up, one over, plus a safety margin. This was when the recreational tables had just come out. My friend and I dove for some years after that, and never had a problem. At that time, dive computers were out for a few years, and all the dive shops, through their "train with really expensive gear so you buy it" training programs had all their students diving with computers throughout their training.
Di Dieter did it right. He trained us with the old fashioned, and RELIABLE mechanical guages, waterproof clocks/watches, and tables on waterproof material. No computers.
While computers can extend dive times because you don't spend all your time at maximum depth, you also increase risk in doing so. Whether you decide to use the dive computers or not, you should ALSO have the mechanical pressure and depth guages, and manually calculate your dives.
Solely relying on a computer for diving is sheer stupidity and absolutely reckless. The minimal increase in bottom time is not worth the risk of an embolism, or the bends, which can be a debilitating condition for the rest of your life, or even fatal.
Do it right. Manually calculate your dive, and rely on your brain, not a computer, to stay safe and not risk your life. Bring that fancy computer with you if you want, but don't trust it over basic guages.
And Di, if you're reading this, this is the dude with the 43 lbs of lead on his weight belt! Hope you're still diving. And enjoying life. Peace.
Attacked by mathematical methods?! I've just spent another 4 hours revising Z, a horribly evil formal language designed to almost exactly the system you propose. Unfortunately, it doesn't work, at least not for the vast majority of software programmed.
I don't know about most other CS students, but the reason I despise mathematical formal proofs in computer is because they are infeasible, and, frankly, you're more likely to make a mistake with the math then you are with the code, once the project gets big enough.
Have you ever used formal methods such as Z to prove your software? Believe me, it's not something you'd do willingly. The only possible uses for mathematically proved software is with simple, but important code, like in an ATM, for example. Anything larger and it all falls apart.
My point is that mathematical methods aren't the answer. Even my lecturer on this subject, who co-wrote the language, would use such techniques on a large software project. You'd have to be insane.
BTW, moderators who modded down the original post, doesn't this qualify as "When Bad Software Kills"?
Companies are out to make money.
Take the expected number of products that customers have that will fail and harm/kill someone, then multiply that by the average settlement. You end up with what your company can expect to pay from all the court cases from people dying with whatever product a company sells.
If this is cheaper than doing a recall, the company won't do a recall. Even when the company knows people will die from their shitty products
That's what Fight Club says, though I think most companies these days will do a recall anyway, in an effort to avoid bad PR as well.
Ford/Firestone didn't do too well by not doing a recall for a long time. Yeah, they might have expected to lose less money by not doing a recall, but the massive amout of bad PR that came around (people started noticing they were more likely to die on the things) ended up doing a lot worse damage to the bottom line than a recall.
I would love to find some attribution to this, I believe I remember reading it in Computerworld in the mid-'80s:
A manufacturer of particle accelerators for treating cancers had a unit, that due to a software bug, would occasionally blow a fuse. It wasn't considered important enough to track down, since you could just reset the machine, and it'd be fine.
Until they upgraded the equipment for a higher power unit, with the same software. The radiation dose killed a patient.
This came up originally under the subject of software malpractice.
Design for Use, not Construction!
More government control doesn't necessarily help that much.
Some of the most serious problems with defective products in recent history have occured when government was entirely in control. In some cases they screw up because, like business executives, they want to cut costs (providing HIV/AIDS infected blood for example). Sometimes they wind up killing people because they are too cautious. Scandals usually occur when actions kill people, not so much when inaction kills people (delays in FDA approval for new treatments cost thousands of lives).
If you think this is a problem with Capitalism then you should take a look at the sorts of things that went on in Communist countries like the USSR and still go on in places like Communist China.
Cover-ups make me sick.
I think that the only effective remedy for this sort of problem is greater transparency in both business and government. These kinds of problems thend to occur when the people involved think that they can get away with a cover-up.
I actually had a little trouble with an ABS system a few years back, on a '94 Pontiac Grand Am. The system failed in such a way that once in a while when I would apply the brakes, the pedal would sink all the way to the floor without doing anything to slow the vehicle... the brakes were just plain not there. I would immediately let up on the pedal and reapply the brakes, and then they would work.
Luckily, the first time this happened I was slowing from about 25mph to turn into a parking lot, with no other traffic around-- otherwise things might have been more, shall we say, interesting.
I was stunned when the service people told me that the failure of the ABS could take out the brakes entirely. One can just imagine the kind of lawsuit that could have been unleashed, had my brakes gone out at a truly inopportune time-- like if a little kid ran out in front of my car, or I were unable to stop at an intersection and ended up getting t-boned by a speeding 18-wheeler as a result.
~Philly
In my experience as a diver for the last 15 years, I have seen many divers who rely on dive computer technology to get closer to the edge and get more bottom time, longer dives, deeper dives, etc. The original "paper" dive tables were based on the experience and testing by U.S Navy diver's and are VERY conservative. The advent of widespread use of dive tables for recreational diving resulted in diving being a lot safer. The advent of computerized tables has promoted a false sense of security to the diver (kind of like having a radar detector in car - you might avoid more tickets, but you may speed more also) - I myself have dove profiles I would never have attempted based on the dive tables but the computer "said" it was OK so we did it. Here is a story about a dive computer specifically designed to be used with mixed gas diving (nitrox) adding yet another element of risk over regular diving. I think that dive computers should come with a waiver that says "if you trust you life to this device you do so at your own risk". Based on what the story said I would never have gone flying so soon after diving using regular dive tables - they threw the dice and lost, and now they want to pretend that what they were doing was risk free and the dive computer caused all the problems - Its nice that the dive computer maker is recalling the units to make them more conservative. Too bad those divers didn't buy the "common sense" computer too.
> We don't want government getting too involved with businesses, With the current administration and congress, we don't have to worry too much about that, do we? Actually, I do want the government getting *more* involved with the state of business practices. A laissez-faire attitude always results in lack of business responsibility, liability, and concern for the consumer, which results in pain, death, and suffering. Any study of history should acknowledge this(read: The Irish Potato famine could have been averted of the imperialists in question had espoused a less laissez-faire attitude). Moreover, the amount of penetration corporations have within our government is sickening; corporations vicariously have more power here than in any other country. The consumer needs to be protected more vigorously, and to do that, the goverment needs to be more committed to checking corporate behavior. Things like this - blatant corporate oversights and resulting cover-ups - should never happen. >but we want them to kick them in the ass hard when they >do something that not only can hurt/maim/kill someone, >but also creates an adverse environment for corporations >who DO act responsbibly. Agreed. Unfortunately, corporations, at their highest levels, seem to be fraught with irresponsibility. What happened last year with ENRON, Worldcom, AOL, etc, made me quite angry. Executive greed is dangerous.
Don't forget that, for years, the PADI dive tables were Navy Dive Tables. For a very good history of dive tables, click here. Dive tables are based on the same theories and technology as dive computers. The biggest difference is that they are more prone to human error than are dive computers.
It's great to say "plan your dive and dive your plan", but people are fallible. Your buddy may go a little deeper than he/she intended. You may, because of narcosis, get confused about the maximum time or depth. You or they might have problems that slightly delay your ascent. If you plan to go to 90 feet and you drop a video camera to the bottom at 110, unless you're Bill Gates, you're probably going to go get the camera and cut the dive short. So much for the plan. (I've been a diver for over fifteen years. I've seen everything from divers getting entangled in wrecks to outright equipment failures. Don't reply with some macho explanation of how you can foresee and prevent every error. You can't and if you believe that you can, then you need to stop diving until your ego and testosterone stop affecting your reasoning.)
My buddy and I each bought dive computers at the same time. I made it a point to choose dive computers from different manufacturers so that a sofware flaw in one would not put us at undue risk. We stick close together and always use the more conservative of the two computer readings. Had one of us had a UWATEC computer, we would have noticed the problem immediately when comparing the two computers.
As long corporations can figure out when it's cheaper to just go-ahead and let a few people die, some will. There needs to be a 3 strikes your company is dissolved law.
No more company, all assets sold, stockholders get whats left over, after all debts payed (as usual). Corporate officers and board members prohibited from serving in either capacity in any corporation for a period of at least 2 years. Don't worry if they don't actually have enough cash to cover that, they can always get real jobs...
At one point in our history, it actually required an act of Congress to incorporate, it isn't a right its more like a drivers license, the only thing Congress would need to do is care.
or as we call them ABS.
These are software driven just like vehicle stability systems (in fact VSS is just ABS hardware with a few more sensors).
We don't seem to have a plague of rogue software killing occupants because, guess what, we really try to be careful with this stuff. The FMEA for ABS software is huge, and detailed. How many of you guys ever have to do an FMEA? or even know what one is?
Here's a funny war story (I know everyone involved, this happened).
The wheel speed is picked up via a sensor on a 40 tooth tone wheel. Cars without ABS don't have tone wheels. Someone asked, what will happen if a non-ABS wheel gets onto an ABS car?
Mechanic fits non-ABS wheel to car. Development engineer drives off, hits brakes at 10 kph, no prob.
same at 20, same at 30
same at 40
same at 50
same at 70
Drives up to 80 kph hits the brakes, one wheel locks, the car slews sideways into a bank and rolls up on two wheels.
Engineer needs new undies.
What had happened was that the sensor could pick up the back of the wheelstuds. There are 5. The software assumed that the car was travelling at the speed of the slowest wheel, so the non-ABS wheel was the vehicle speed. Our ABS is disabled below 9 kph to ensure that you can stop the car, on some surfaces it will only inch to a halt unless you lock the wheels.
So, at 80 kph the ABS realised it was supposed to work, activated, saw one wheel almost stopped, released the brakes on that wheel, and locked the other one. Hilarity ensued.
I routinely do dive profiles that my dive tables say I should get bent on. My computer knows better, because it knows the actual depth profile I dive, and not just the max depth and total dive time.
Almost any dive profile on a wall will do this. You start deep, and drift slowly more shallow as you go, and you can do a nice hour-long dive with a max depth of about 80ft, and an average noticably shallower than that.
Yes, you can do this with a multi-level dive table, with a wheel or similar. I've done that. You know how much trouble that is? And how difficult it is to know, for a sloping wall dive, exactly how long you'll spend at any particular depth looking at coral? Yes, plan your dive, and dive your plan. But realize that you aren't a robot, and don't dive like one.
Multilevel dive planning is for deco diving where your computer can't handle it, and, incidentally, you *have* to know deco times ahead of time so you can hang stage bottles at the right depth.
But that dive profile above, for a normal set of dive tables, diving with a computer, will almost always end with your tables telling you that you went into deco. Because all it uses is max depth and total bottom time.
Yeah, and the tables are approximations, too. Actually, they are statistical representations, and state that 98% of divers that stay within these guidelines will not get DCS, with some confidence bound. Yes, diving tables, you can still be that unlucky 2% that does everything right and gets bent anyway. Sometimes it's just not your day.
No. You do research, you find out what algorithm your computer uses, how conservative or liberal it is, how it was modified from standard industry-published algorithms, and you pick a computer that works the way you want it to. And then you dive within the bounds the computer sets, so long as those bounds pass your internal bullshit detector. (You *do* have an internal bullshit detector, right?) But diving close to those bounds is not "dumb" it is simply using your equipment to the limits you are comfortable with.
Can't disagree with that at all. Finishing a dive at 10pm and flying at 6:30am the next morning is not safe.
This is my sig. There are many like it but this one is... Oops. Frank, I've got your sig again! Where's mine?