Not All Bugs Are Random

CowboyRobot writes "Andrew Koenig at Dr. Dobb's argues that by looking at a program's structure — as opposed to only looking at output — we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on an aspect of its input such as size, we should verify that it works properly as close as possible to the decision boundary on both sides. I've seen quite a few programs that impose arbitrary length limits on, say, the size of an input line or the length of a name. I've also seen far too many such programs that fail when they are presented with input that fits the limit exactly, or is one greater (or less) than the limit. If you know by inspecting the code what those limits are, it is much easier to test for cases near the limits.'"

Comments

Obvious, but worth restating.

[Animats]

Fairly obvious statement, but one some newer programmers don't know. Koenig is talking about white-box testing, which is well understood.

Re:Obvious, but worth restating.

[Anonymous Coward]

Yep, fairly obvious, but where I work there's this whole class of people who hide their incompetence behind the phrase "black box testing". At first I didn't understand why they insisted so much on doing black box testing, until one day we basically ran into what the article describes: a subsystem behaved poorly when one of its inputs was exactly on the boundary that that very subsystem had defined. In that case I saw the black box bunch crash and burn because they weren't able to even design a good test case, much less diagnose the cause of the problem.

To me Andrew's article basically says: if you want to be able to test code, you need to be able to read code. There is a whole breed of testers nowadays who are not able to do that.

Re:Obvious, but worth restating.

[uncqual]

I think in common usage, white box testing does not mean "debugging" or "diagnosing" the code - it means more like "testing that takes into account the internal design and implementation of the system under test rather than just the inputs and outputs".

There's a need for both types of testing. However, as long as white box testers don't get too wrapped up in the details to step back and view the system as a whole (as developers sometimes do), they can do both. But there are so few good testers in the business, you don't want to waste a good white box tester on developing black box tests.

Sadly, I suspect there are quite a few "average" developers out there who, had testing as a career been given more respect by the industry over the decades, would have been "great" white box testers. While there is significant overlap between the skills/personality attributes needed to be a great developer and those needed to be a great tester, there are a bunch of skills/personality attributes that are quite important in one field and not nearly as so in the other.

Re:Obvious, but worth restating.

[icebike]

Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.

We are so busy giving new names to old concepts, and barfing up new programming languages that we end up re-inventing and re-naming long known concepts with shiny new trendy names that we fail to notice this was well known decades ago.

Re:Obvious, but worth restating.

[Joce640k]

Are any bugs truly "random"? I always thought computers were deterministic machines.

OK, maybe process scheduling bugs are fairly random...the point is that the headline makes it sound like bugs just happen all by themselves.

Is this news for anyone?

[JeffOwl]

We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.

Bounds test?

[jklovanc]

Has testing degraded so far that people don't now what a bounds test is?

No shit Sherlock

[cruachan]

You know, I remember writing test plans to to test input that were one below, at, and one above, some arbitrary limit when I was a trainee programmer coding up COBOL on mainframes back in the mid 80s.

How on earth does this drivel make it onto Slashdot? This is 30 year old news at least (which makes it straight out of the 17th C in internet years)

Re:No shit Sherlock

[Tanuki64]

Sadly, I still see developers not testing, and are practically afraid of writing test scripts.

Afraid? Less afraid than not paid for. 'Write your code correctly, then you don't need to test'. A bit simplified, but this is more or less what I heard more than once from a customer.

Look at commit time stamp.

[140Mandak262Jamuna]

Talking from personal experience, all the bugs I ever committed to production were coded in between 1 PM and 2 PM, my sleepiest time, when my stomach digesting the lunch was competing with the brain for blood supply.

If I am a columnist with some modest name recognition I could have converted this mildly amusing (to me at least) observation into a column. But alas, I am not one. So he gets to repeat the age old advice given by old Prof Mahabala, teaching Intro to Computing 201, (Fortran programming, in IBM 365/155 using punch cards no less ) back in 1980 into a column, and all I get to do is to bitch about it.

It's good to test the corner cases

[Black Parrot]

News at 11.

Who might know about this?

[Anonymous Brave Guy]

That depends. If you're are a die-hard TDD fan, for example, then you'll be writing your (unit) tests before you necessarily know where any such boundaries are. Moreover, there is nothing inherent in that process to bring you back to add more tests later if boundaries arise or move during refactoring. It's not hard to imagine that a developer who leaves academia and moves straight into an industrial TDD shop might never have been exposed to competing ideas.

(I am not a die-hard TDD fan, and the practical usefulness of white box testing is one of the reasons for that, but I suspect many of us have met the kind of person I'm describing above.)