When Computers Go Wrong

Barence writes "PC Pro's Stewart Mitchell has charted the world's ten most calamitous computer cock-ups. They include the Russians' stealing software that resulted in their gas pipeline exploding, the Mars Orbiter that went missing because the programmers got their imperial and metric measurements mixed up, the Soviet early-warning system that confused the sun for a missile and almost triggered World War III, plus the Windows anti-piracy measure that resulted in millions of legitimate customers being branded software thieves."

Comments

Computers do what they are told to

[adosch]

TFA article should have been named the 'Worlds ten most calamitous logic cock-ups' instead. Because in the end, malformed, ill-tested or and unforeseen logic compensation(s) caused those issues, not computers themselves.

Re:Computers do what they are told to

[the_humeister]

I'm surprised they didn't mention incidents where people actually died, such as the Therac-25 incident.

Re:Computers do what they are told to

[hairyfeet]

Yeah there really wasn't much computer related there. If you wanted computer related I would have added WinME, aka "what idiot thought mixing WDM and VXD drivers was a good idea?" along with Vista Capable, aka "We've got to let the OEMs dump their crappers on Best Buy, so pretend it runs, okay?" and finally the early Athlon without thermal monitoring aka "Heat problem? What heat problem?".

And of course if you wanted some real old time badness there was Bonzi Buddy, also known as "Kill that GODDAMNED MONKEY DEAD!!" and Geocities with the ever popular "WTF? Why is there a pocketwatch hanging off my mouse like a ball of snot and who thought pink OMG Ponies! text on a lime green background with sparkles and GIFs was tasteful?" and of course MSFT Bob, an OS made for the clueless that needed a fricking gamer rig just to run and spawned the electronic son of Satan known as Clippy.

Finally on the hardware side I'd add the Pentium 4, also known as "Mr Piggy Super Space Heater", the Geforce 5xxx Hoover Edition, which was famous for not only filling your PC with the sounds of sucking but thanks to cheating by Nvidia on rendering actually gave you REAL sucking as well! Quite an accomplishment that, the Seagate "I hope you didn't actually NEED your data for anything" bug in the early 1.5TB drives, the early Phenom "watch this patch suck away your performance" TLB bug, the iPhone 4 which gave us such lovely phrases such as "WTF do you mean I'm holding it wrong?" and finally to show they can still make incredible mistakes the Nvidia bumpgate, also known as "We do NOT have a problem with our GPUs, its a power saving feature! See it makes your computer shut down and everything!!". These I think would have been a little more computer centric than stolen code and a screwed battery on a Volvo.

Re:Computers do what they are told to

[crunchygranola]

And of course there is the Patriot missile software clock issue - that led to a failure to engage a SCUD on February 25, 1991 at Dhahran, Saudi Arabia, killing 28 soldiers.

This failure is rather similar to the Soviet defense and NORAD errors mentioned in the article in that it was a weakness designed into the system that did not account for the range of operational condition and issues. In the Petrov Incident case - a natural condition, in the NORAD case an easy to make operator error, in the Dhahran barracks Patriot incident it was a failure to consider that a unit might be operated for weeks without a restart.

Re:Computers do what they are told to

[fyngyrz]

No, he clearly meant "The Fucking TFA Article."

Kids today.

Re:Computers do what they are told to

[rarel]

it's okay, you can breathe now.

Re:Computers do what they are told to

[TheLink]

It's not confirmed that the gas pipeline blowup was due to computers going wrong.

http://en.wikipedia.org/wiki/Siberian_pipeline_sabotage#Hoax_Theory

Here are a few more "logic cock ups":

http://en.wikipedia.org/wiki/Ariane_5_Flight_501
http://it.slashdot.org/article.pl?sid=07/02/25/2038217

And Wired's list: http://www.wired.com/software/coolapps/news/2005/11/69355

Re:Computers do what they are told to

[jc42]

Another aspect to this is a common property of most "digital" computations. I've seen it expressed as "Digital errors have no order of magnitude". Another phrasing is "Getting one bit wrong is generally indistinguishable from randomizing all of memory". So when a digital calculation goes wrong, a tiny, inconsequential error is just about as likely as a total meltdown of the entire system.

Programmers tend to get familiar with this phenomenon very early in their career. They write a small chunk of code that does a simple calculation, and the result is orders of magnitude wrong. When they investigate, they discover it was caused by a one-character typo, perhaps an "off by one" error such as using '<' instead of '<=', or vice-versa. This quickly leads to what many "normal" people consider the major character failure of software geeks, the insistence that everything be exactly right, no matter what, and the willingness to spend long hours discussing insignificant minutiae as if they mattered. In their work, it's usually such insignificant minutiae that brings the whole house of cards tumbling down.

If you're unwilling to take the difference between a comma and a simicolon seriously, you have no future as a software developer. This is often why something goes badly wrong and we have events like those described in this story.

OTOH, it is interesting that, despite all the software disasters like the metric/imperial-units story, the software world has never insisted that programming languages include units as part of variables' values. It's not like this is anything difficult, and it has been done in a number of languages. But none of the common languages have such a feature. It is a bit bizarre that we can get into long discussions of complex, obscure concepts such as type checking or class inheritance, when our calculations are all susceptible to unchecked unit mismatches (without even a warning from the compiler or interpreter). There's a lot of poor logic when the topic is the relative importance of various sources of bogus calculations.

Re:Computers do what they are told to

[Opportunist]

To be fair, the PATRIOT manufacturer didn't think it would stay assembled for weeks without falling apart, thus requiring a restart.

Re:Computers do what they are told to

[hitmark]

https://secure.wikimedia.org/wikipedia/en/wiki/Ada_(programming_language)

I think the problem is that most of the hobby, and perhaps even commercial, programming happens on a "scratch itch" basis. Once it does what the programmer set out to do, the job is done no matter how nasty the code may look. And any language that allows the programmer to get there quickly get instant love. Then there are situations, mostly on the bare metal level tho, where doing things in crazy ways is the only way to get it done.

Re:Computers do what they are told to

[kennykb]

A system using proper fractions can actually get exactly the right answer every time OR it will overflow and we will know for a fact the answer isn't exact.

What theory of numeration are you using, that has all numbers rational? I'm sorry, but even the humble square root is something I don't want to give up, to say nothing of transcendental functions. The theory of exact arithmetic on the reals is not all that well developed. Bill Gosper makes a start, and a handful of researchers take it somewhat further, but actually using exact arithmetic for everything you'd want to do remains a mirage.

Re:Computers do what they are told to

[kennykb]

"Units are parts of variables" usually comes along with systems in which there is no escape. Dimensional analysis is fine up to a point, but when you get into weird quantities like dBm/sqrt(Hz) (seriously: ten times the log-base-10 of a quantity measured in milliwatts, over the square root of another quantity measured in hertz), the systems that enforce units tend to fall apart, and often it turns out that they simply lack the notation you need. (By the way, "dBm per root hertz" was a unit that I used in daily work at an earlier time in my life. And I still use weirdness like neper-coloumb per square micron.)

Re:Computers do what they are told to

[PPH]

Don't anthropomorphize computers. They hate that.

Re:Computers do what they are told to

[sjames]

None at all. I just presumed it was understood that my statement applied to rational numbers.

You could take it to the next step and handle irrational numbers symbolically, but that's probably best left to software rather than hardware. You could keep a hardware function called squareish root though if you like that returns a fraction matching the current approximation. You won't actually lose anything that way.

I'm pretty sure we will at least be improving matters by not losing on simple division.

therac 25

[Anonymous Coward]

List fails without the therac 25

Re:therac 25

[cratermoon]

The list fails for a many reasons. Too many reasons to calculate accurately on a Pentium. On the first page, while describing the bug on said Intel CPU, the author defines floating-point numbers as "numbers too large to be represented as integers".

Imperial - Metric

[Anonymous Coward]

Due to the imperial-metric mash-up, the sums were so far askew that when Ground Control initiated boosters to secure the pod in orbit, all they succeeded in doing was firing it closer to the planet, where it burnt up in the atmosphere.

When I see the Imperial-Metric confusion shit, I just want to slap the shit out of someone. That waste because some engineers are incapable of using Metric or some vendor just doesn't want to spend the money to modernize their machinery. I know of an aerospace contractor that is using machinery from the 50s - yep, they're constantly being recalibrated and sometimes they don't notice - ooopsie!

And when I see that we, the US, are one of two countries still on Imperial - one is some Third World non-industrial country, I want to barf.

And then, when I have to buy two sets tools to work on a car, I wish for the entire US auto industry to go bankrupt and be replaced with some modern companies.

I love Metric. It makes measurements and calculations much easier - quick! What is the mass of 329 mL of water? You'd need a calculator to do something similar in Imperial.

Ariane 5 missing on the list

[Anonymous Coward]

It isn't smart to assign a 64 bit floating point to a 16 bit integer - unless you want to crash you first flight of the heavy Ariane 5 rocket... (http://en.wikipedia.org/wiki/Ariane_5#Notable_launches)

Re:Ariane 5 missing on the list

[owlstead]

Actually, those kind of conversions should be banned from any managed programming environment. It's fine that you need to work with bytes, shorts etc. or heck maybe even machine words, but lets only do that when absolutely required, shall we.

It amazes me that the many programming languages still don't define acceptable ranges, accept null pointers, and use round robin two-complement numbers etc. etc.. It's just asking for errors just like these. Sure they have their uses for lower level functions, but I would certainly like to have something better for API's and general use business logic. They are just another pointer arithmetic or GOTO waiting to be erased from mainstream programming (and for sure, in many newer languages, they indeed are).

1 Page Read

[DevConcepts]

http://www.pcpro.co.uk/features/363580/when-computers-go-wrong/print

It's a simple rule

[Anonymous Coward]

As a fellow programmer I worked with years ago was fond of saying, "Computers don't make mistakes. They do, however, execute yours VERY carefully."

Re:It's a simple rule

[TheRaven64]

Unfortunately, they don't just execute your mistakes, they execute the mistakes of everyone involved in the toolchain. If you want to write bug-free software, then you also need a bug-free compiler, bug-free libraries, and a bug-free OS. The most you can say about most software is that it doesn't contain any bugs that are both serious and obvious.

Re:It's a simple rule

[jc42]

"Computers don't make mistakes. They do, however, execute yours VERY carefully."

That's a good way of phrasing it. But it does miss the fact that not all "computer errors" are due to software mistakes.

One example, of course, is the Pentium FDIV failure. That was a hardware failure, "programmed" into the CPU by Intel's experts in solid-state hardware design. There wasn't a whole lot that any software developer could do to defend against that failure.

Another, more subtle one, came up when I was a grad student back in the 1970s. At that time, most of the campus research computing was done on the big mainframe in the campus Computer Center. After discovering a number of (published ;-) results that turned out to be wrong, some researchers investigated, and found that they were due to undetected overflows in the calculations. Yes, the hardware could and did test for overflows, and set a status bit when they occurred. Almost all this calculating was done in Fortran, and the Fortran compiler had a run-time flag that could turn the status-bit checking on or off. It defaulted to OFF. They did a bit of analysis, and concluded that about half the runs of Fortran programs on that machine produced output that included numbers that were incorrect due to undetected overflow.

So why didn't they make the overflow-detection flag default to ON? Well, they did a little survey of the users. They found that the overwhelming response was that, if enabling overflow checking made the program run slower, then overflow checking shouldn't be done. Somewhere around 90% of the people asked said this. They weren't mathematically ignorant people; they were the people using the Fortran compiler for the data in their professional publications.

This told us a lot about the way such things are done. Since I left academia and worked in what passes for the Real World, I've found that this is a nearly universal attitude. Faster and cheaper is always preferable to correct. This is still true even when we have computers in commercial aircraft and hospital operating rooms. And you can't call this sort of thing a "human error". People don't decide to disable overflow checking by accident; they do it knowing full well what the effect will be. When the computer fails in such cases, it wasn't executing a human's mistake; it was doing what the human wanted it to do.

The creation of the EFF

[Anonymous Coward]

The "Switchboard meltdown" problem sounds like the incident which led to the creation of the EFF.

Basically, someone forgot to include a ";" in a C program, which led to the problems at ATT. Originally, they thought it was due to "hackers", and called in the Secret Service.

The Secret Service in turn busted a gaming outfit called "Steve Jackson Games". Who was completely innocent, of course, but that has never mattered to the Secret Service when they need to look like they are actually useful. The SS confiscated the computers, all illegally.

The ACLU refused to get involved, so John GIlmore (formerly of Sun, and who worked with Richard Stallman to get out an open Operating System around that time) created the EFF to fight the unconstitutional raid on Steve Jackson Games. The EFF trounced the Secret Service in Court, and was thus born. I believe if you google for "Steve Jackson Games", you can still find the original story around.

So, in a way, you can say that the EFF was created due to the single misplacement of a semicolon in a C program. Would that all of our bugs have such results. :)

Re:The creation of the EFF

[not-my-real-name]

Should this be modded "+1 Too Informative"?

Re:Imperial - Metric

[Rob the Bold]

Due to the imperial-metric mash-up, the sums were so far askew that when Ground Control initiated boosters to secure the pod in orbit, all they succeeded in doing was firing it closer to the planet, where it burnt up in the atmosphere.

When I see the Imperial-Metric confusion shit, I just want to slap the shit out of someone. That waste because some engineers are incapable of using Metric or some vendor just doesn't want to spend the money to modernize their machinery. I know of an aerospace contractor that is using machinery from the 50s - yep, they're constantly being recalibrated and sometimes they don't notice - ooopsie!

And when I see that we, the US, are one of two countries still on Imperial - one is some Third World non-industrial country, I want to barf.

And then, when I have to buy two sets tools to work on a car, I wish for the entire US auto industry to go bankrupt and be replaced with some modern companies.

I love Metric. It makes measurements and calculations much easier - quick! What is the mass of 329 mL of water? You'd need a calculator to do something similar in Imperial.

I'd prefer to slap someone for saying "Imperial vs. Metric" when they're talking about US standards vs the SI -- which one certainly is when talking about the mars spacecraft failure. After all, the US system -- while derived from the Imperial System -- is not the same thing. Quick: how many l in a gal? Well, it depends, doesn't it? Did you mean Imperial gallon or US gallon? How many m^2 in an acre? What's the mass of a ton(ne)? And as I like to point out to people -- because I'm a pedantic nerd like everyone else here -- the US system is a metric system . . . see what I did there? I didn't use a capital "M" or say SI there?

Re:Imperial - Metric

[Sycraft-fu]

Well I have to support part of what you've said, and contradict part.

I support you in that it is stupid NASA uses Imperial ever, anywhere. Metric is the method for science and with good reason. So it is stupid that they wouldn't use it 100% of the time. Any chemistry or physics class I ever took was all metric all the time. It wasn't even a "We do this to make you learn it," kind of thing, it was just the way it was, it was assumed.

However I have to contradict you on the "OMG the US is so stupid for not going Metric," thing. It doesn't really matter. What matters to normal people in every day life is having a feel for what a unit is, not inter-unit conversions. Your example is something people do not do. It does not matter the ability to do fast conversions on units of volume, it matters that you have a feeling for what they are. You can stick with a system that is not neat and regular and it works just fine.

Also if you think metric rules all in other countries you've just not looked. I have the occasion to visit Canada once a year and the imperial system is alive and well, lurking in the shadows. In some cases it is explicit, you find various food items sold in pounds, rather than kilograms. In some cases it is more hidden. Soda is sold in 12 ounce cans. Yes, they say 355mL on them as well (as they do in the US) but it is a 12 ounce can. 355mL was not the unit used to design it, 12 oz was. Sometimes people don't even know it. Alcohol is sold in units frequently referred to as "fifths". It is 750mL but why the the term? Because it is a fifth of a gallon (well 5.04 is you want to get technical).

That is why there's the apathy in forcing a change. You really gain very little for most people in every day operation. I'm not saying it would be a bad thing for a change to happen, but there isn't the incentive many geeks seem to think there is.

I work comfortably in both systems. I've done plenty of science so I've no problem with any metric units, but I also bake which is extremely imperial dominated. Doesn't matter to me. I can even work in both at the same time. If a recipe calls for 3 cups of bread flour, I know my chosen flour is 155 grams per cup. So when I weigh it out on my scale I weigh out 465 grams. I could do ounces instead wouldn't matter, my scale just reads grams. Likewise it wouldn't matter if the recipe instead called for 700mL of flour. Metric doesn't make it any easier because the nice "all units are 1" factor only applies to water. My flour converts volume to weight at about 0.664, of course that depends on how dense it gets packed. That conversion factor is no more, or less convenient than 155.

Really, working in the screwy imperial system just isn't a big deal to normal people. You don't do anything that needs inter-unit conversion which is where metric shines.

MS London stock exchange crash

[flyingfsck]

We got to commend MS for the most expensive computer cock-up.

And sometimes it isn't the computers...

[BrokenHalo]

(See title.)

Any of us who have been in a sysprog or sysadmin role for a significant amount of time (by which I mean double-digit years) will often have at least one anecdote of some monumental cockup we've perpetrated.

My worst case in point is where I managed (IIRC after a long liquid lunch) to delete the :per directory (more or less equivalent to /dev on a *nix box) on a Data General mainframe machine running AOS/VS. While hundreds of users' processes disappeared off the system (which took about 90 minutes), I found it expedient to simply make my confession to the boss.

Fortunately, in this case, the escapade was more or less written up as "Shit Happens", which I thought was generous...

1982 explosion did not happen

[mike449]

Te Soviet pipeline explosion seems to be an urban legend, traced to a single source: At the Abyss: An Insider's History of the Cold War, by Thomas C. Reed.
There is no mention of this explosion anywhere else, either in Russian or Western sources. If you can read Russian, some debunking is here:

link
One of the facts mentioned there is that there was no SCADA on Soviet pipelines until late 80-s. All control was still pneumatic in 1982, with no software involved.

Re:Imperial - Metric

[mikael_j]

From your post it sounds like you've been living somewhere that used to belong to the british empire, those people still tend to think of their weight in "stones" and various other oddball measurements but there are definitely countries where imperial units are barely used.

Here in Sweden the only people who use imperial units seem to be carpenters who call a 5x10 cm piece of wood a "tvåtumfyra" ("twoinchfour") but even they don't actually assume the actual size of it is 5.08x10.16 cm, it's just that "tvåtumfyra" is faster to say than "fem gånger tio centimeter".

As for degrees, most people tend to use degrees in everyday conversation (when it comes up) but degrees are not an "imperial" measurement, it predates most imperial units by centuries. And most people I've met who have taken "advanced" high school level math or college level math tend to use radians when actually doing any kind of math related to angles.

Also, you tell someone here in scandinavia that you're 5'10" tall and weigh 176 lbs and they're likely to either not understand you or they'll go "So, a foot is like, 30 cm, right? and how many inches are there in a foot? I know it's not ten but like, fifteen or something, right? And a pound's like, 0.5 kg? or was it less? maybe more? And aren't there two types of pound? Or was that pints?".

Basically, if you tell someone around here that something is "n <imperial unit>" they will have no clue no matter how "natural" you think it is because you happened to grow up with it.

Also, as for easy unit conversions, people do use them, just not in the uncommon ways you described, most people just aren't familiar with some of the less common prefixes but milli-, centi-, deci-, hecto- and kilo are all commonly used (and most people know that mega and giga are millions and billions, they just don't have much use for them, so rather than saying 1.5 megameters you say 1500 kilometers).

"Black day for power programmers" Windows virus

[Locutus]

to comments, I thought the deal with the big blackout was that the network(TCP/IP) was flooded with a Windows virus infection and if you know TCP/IP, it's not very good with lots of traffic. There was so much traffic that the computer( a UNIX box ) sending status messages to the control room display system could not get messages out of it's buffers. TCP/IP does this thing where the message isn't put on the network if there's going to be a collision and it waits some before trying again. With the network flooded with Windows based computers trying to infect each other, the warning messages were stuck in the UNIX box and eventually the buffers filled up as more and more warning messages queued up. They seem to be blaming the UNIX box software because the software ended up crashing because they didn't catch the situation where they buffers overflowed. IMO, that was caused by Windows and it's ability to be a great petri dish for viruses and the idiots who keep putting Windows systems on critical networks.

The second comment I have on this is about missing the LAX Communications system software crash which caused multiple near misses on the tarmac and in the air when air traffic controllers could not communicate with pilots because of the crash. The cause of the software crash was a UNIX system was replaced with a Windows based system which had a known flaw. The flaw was that the OS could not run for more than 39 days no matter what was running on it. The system and software was still approved and put inplace with a maintenance instruction of rebooting the computer every 30 days. In comes a new employee who sees things are working fine so he/she doesn't reboot the computer and 9 days later the system crashes. The backup does the same and both are unable to recover and it takes hours to get the system back running again. That should have been in the list IMO.

There was also the CSX Railway situation when lots of its signals go offline because they are run by Windows and their Windows computers got a virus.

It would be nice to see a more complete and more accurate list of these kinds of computer software failures.

LoB

Re:Don't tell me this stuff, please...

[Reziac]

I'd not heard of it nor the fellow involved (who as it turns out is still alive), so I went and looked it up, and learned all sorts of interesting stuff:

http://en.wikipedia.org/wiki/Stanislav_Petrov
http://www.brightstarsound.com/world_hero/article.html
http://www.armscontrol.ru/start/publications/petrov.htm

Re:Imperial - Metric

[Reziac]

My college physics and chemistry classes went as you describe -- for classwork, metric was assumed and no one thought anything of it. For everything else, Imperial was used. So you might hear something like (making up absurd example to shoehorn it all into one sentence) "I had to move my desk twenty feet just to get a measurement of less than one millimeter!" and it sounded perfectly natural to us. We're measurement-bilingual. ;)

Re:The best parts of the article were...

[kennykb]

One way to calibrate your respect for the press: listen to them when they're talking about something you know about. Assume that they have the same depth of understanding when they're talking about something you don't know about.

Re:Imperial - Metric

[lahvak]

Yes, getting a decent kitchen scales in the US is a pain. In Europe, every reasonably equipped kitchen has a set of kitchen scales on the counter.

On the other hand, measuring certain ingredients by volume is better. For example, the specific weight of flour changes quite a bit with humidity, while volume stays pretty much the same.

Therac-25 deaths and airport body scanners

[Anonymous Coward]

I'm surprised they didn't mention incidents where people actually died, such as the Therac-25 incident.

Radiation dosage mistakes like this make you wonder how well and how often
airport body scanners will be calibrated as machines remain in service for years.