Not All Bugs Are Random

CowboyRobot writes "Andrew Koenig at Dr. Dobb's argues that by looking at a program's structure — as opposed to only looking at output — we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on an aspect of its input such as size, we should verify that it works properly as close as possible to the decision boundary on both sides. I've seen quite a few programs that impose arbitrary length limits on, say, the size of an input line or the length of a name. I've also seen far too many such programs that fail when they are presented with input that fits the limit exactly, or is one greater (or less) than the limit. If you know by inspecting the code what those limits are, it is much easier to test for cases near the limits.'"

Comments

Is this news for anyone?

[JeffOwl]

We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.

Re:Is this news for anyone?

[AmiMoJo]

Even if you think you "know" the input will always be a certain length or less than x characters you had better check it anyway and test what happens if it isn't. Otherwise one day you can guarantee it won't match your expectation.

About 60% of the random, occasional failure bugs I fix are people making these kinds of assumptions. Clocks never run backwards, sensors never fail, data never gets corrupted. In reality they do with alarming regularity.

Bounds test?

[jklovanc]

Has testing degraded so far that people don't now what a bounds test is?

Re:Bounds test?

[girlintraining]

Has testing degraded so far that people don't now what a bounds test is?

It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense. It may be sufficiently complex so as to appear random, but it isn't. There's very little "random" about a computer, or a computer program. Every now and then a single bit error creeps into the I/O but the rate this occurs in computers not in space and thus subjected to extreme radiation is so low you could run most tests for many thousands of hours before encountering one.

Hell, creating randomness is so problematic in a computer that they've had to create specialized circuits to create entropy; And at that, it's still been shown to fall short of being random enough for some cryptographic algorithms... If creating entropy is this hard in a computer, then why do people suddenly point at the sky and say "It was teh randomz!" whenever it crashes? It wasn't random: Something caused it. It just may be beyond your ability or comprehension to know what it was, but it's a deterministic system. It did not just up and decide to fail because it felt like it.

No shit Sherlock

[cruachan]

You know, I remember writing test plans to to test input that were one below, at, and one above, some arbitrary limit when I was a trainee programmer coding up COBOL on mainframes back in the mid 80s.

How on earth does this drivel make it onto Slashdot? This is 30 year old news at least (which makes it straight out of the 17th C in internet years)

Re:No shit Sherlock

[Tanuki64]

Sadly, I still see developers not testing, and are practically afraid of writing test scripts.

Afraid? Less afraid than not paid for. 'Write your code correctly, then you don't need to test'. A bit simplified, but this is more or less what I heard more than once from a customer.

Look at commit time stamp.

[140Mandak262Jamuna]

Talking from personal experience, all the bugs I ever committed to production were coded in between 1 PM and 2 PM, my sleepiest time, when my stomach digesting the lunch was competing with the brain for blood supply.

If I am a columnist with some modest name recognition I could have converted this mildly amusing (to me at least) observation into a column. But alas, I am not one. So he gets to repeat the age old advice given by old Prof Mahabala, teaching Intro to Computing 201, (Fortran programming, in IBM 365/155 using punch cards no less ) back in 1980 into a column, and all I get to do is to bitch about it.

Re:Obvious, but worth restating.

[uncqual]

I think in common usage, white box testing does not mean "debugging" or "diagnosing" the code - it means more like "testing that takes into account the internal design and implementation of the system under test rather than just the inputs and outputs".

There's a need for both types of testing. However, as long as white box testers don't get too wrapped up in the details to step back and view the system as a whole (as developers sometimes do), they can do both. But there are so few good testers in the business, you don't want to waste a good white box tester on developing black box tests.

Sadly, I suspect there are quite a few "average" developers out there who, had testing as a career been given more respect by the industry over the decades, would have been "great" white box testers. While there is significant overlap between the skills/personality attributes needed to be a great developer and those needed to be a great tester, there are a bunch of skills/personality attributes that are quite important in one field and not nearly as so in the other.

Re:Obvious, but worth restating.

[icebike]

Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.

We are so busy giving new names to old concepts, and barfing up new programming languages that we end up re-inventing and re-naming long known concepts with shiny new trendy names that we fail to notice this was well known decades ago.