Not All Bugs Are Random

CowboyRobot writes "Andrew Koenig at Dr. Dobb's argues that by looking at a program's structure — as opposed to only looking at output — we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on an aspect of its input such as size, we should verify that it works properly as close as possible to the decision boundary on both sides. I've seen quite a few programs that impose arbitrary length limits on, say, the size of an input line or the length of a name. I've also seen far too many such programs that fail when they are presented with input that fits the limit exactly, or is one greater (or less) than the limit. If you know by inspecting the code what those limits are, it is much easier to test for cases near the limits.'"

Comments

Obvious, but worth restating.

[Animats]

Fairly obvious statement, but one some newer programmers don't know. Koenig is talking about white-box testing, which is well understood.

Re:Obvious, but worth restating.

[deconfliction]

Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

Re:Obvious, but worth restating.

[Anonymous Coward]

Yep, fairly obvious, but where I work there's this whole class of people who hide their incompetence behind the phrase "black box testing". At first I didn't understand why they insisted so much on doing black box testing, until one day we basically ran into what the article describes: a subsystem behaved poorly when one of its inputs was exactly on the boundary that that very subsystem had defined. In that case I saw the black box bunch crash and burn because they weren't able to even design a good test case, much less diagnose the cause of the problem.

To me Andrew's article basically says: if you want to be able to test code, you need to be able to read code. There is a whole breed of testers nowadays who are not able to do that.

Re:Obvious, but worth restating.

[uncqual]

I think in common usage, white box testing does not mean "debugging" or "diagnosing" the code - it means more like "testing that takes into account the internal design and implementation of the system under test rather than just the inputs and outputs".

There's a need for both types of testing. However, as long as white box testers don't get too wrapped up in the details to step back and view the system as a whole (as developers sometimes do), they can do both. But there are so few good testers in the business, you don't want to waste a good white box tester on developing black box tests.

Sadly, I suspect there are quite a few "average" developers out there who, had testing as a career been given more respect by the industry over the decades, would have been "great" white box testers. While there is significant overlap between the skills/personality attributes needed to be a great developer and those needed to be a great tester, there are a bunch of skills/personality attributes that are quite important in one field and not nearly as so in the other.

Re:Obvious, but worth restating.

[icebike]

Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.

We are so busy giving new names to old concepts, and barfing up new programming languages that we end up re-inventing and re-naming long known concepts with shiny new trendy names that we fail to notice this was well known decades ago.

Re:Obvious, but worth restating.

[Joce640k]

Are any bugs truly "random"? I always thought computers were deterministic machines.

OK, maybe process scheduling bugs are fairly random...the point is that the headline makes it sound like bugs just happen all by themselves.

Re:Obvious, but worth restating.

[Lord Crc]

Are any bugs truly "random"? I always thought computers were deterministic machines.

If your code contains a state machine and the code has a bug where it in some conditions references invalid or uninitialized memory to determine it's next state, then the bug may appear quite random. In fact, it may appear to be several different bugs.

This is particularly fun when doing cross platform programming, where some platforms always zero initializes memory upon allocation causing programmers who develop on those platforms to think everything is peachy, only to have the code promptly crash or worse when run on platforms which does not zero initialize memory.

Re:Obvious, but worth restating.

[buchner.johannes]

Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.

I haven't read TFA, but the newer programming paradigms, such as agile/XP, are heavily test-based. Instead of writing a design, implementing it, and then testing it in production, we start with writing test cases (unit & integration tests, and hopefully some behaviour tests). When they are all fulfilled, we can stop programming.

But how do you know which tests -- beyond functional ones -- to write? Writing and evaluating is already non-trivial for realistic quality tests (throughput, security). Surprisingly, random input tests work quite well (generated either completely random or from regular expressions).

Maybe TFA is suggesting to consider the building blocks a test element is made from, and take into account their limitations to craft test cases (white-box testing when doing test-based programming). I'm not saying it's new, but it makes some sense to me to re-iterate from that point of view.

Is this news for anyone?

[JeffOwl]

We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.

Re:Is this news for anyone?

[Jane Q. Public]

I don't even understand why this is being considered as news. If you don't know this already, you aren't the kind of programmer I've ever worked with.

Now... there are exceptions, too. For example if you know your input will always be less than X characters, you have only one boundary condition to check (no characters). If you know it will always be exactly Y characters you have no boundary conditions. On length, that is... using the same kind of example OP was using.

Re:Is this news for anyone?

[Cley Faye]

Only if your very confident that B will never ever get it's input from anywhere else, or that if it happen, the "anywhere else" will also properly check it's input. Oh, and you should know for sure that A will never change and suddenly spit something that B won't accept. Better document your code, and hope that the future maintainer will actually read what you wrote.
In large, long-lived project, these assumptions are all hard to make.

Re:Is this news for anyone?

[AmiMoJo]

Even if you think you "know" the input will always be a certain length or less than x characters you had better check it anyway and test what happens if it isn't. Otherwise one day you can guarantee it won't match your expectation.

About 60% of the random, occasional failure bugs I fix are people making these kinds of assumptions. Clocks never run backwards, sensors never fail, data never gets corrupted. In reality they do with alarming regularity.

Re:Is this news for anyone?

[uncqual]

In many cases, the "data corruption" case is just impractical to test for very well unless every datum has a hash or something associated with it or another copy exists and every function verifies the hashes or for both copies matching on entry.

Also, I've seen code that was so filled with validation of stuff that was already validated or produced by code we wrote that the bugs were mostly in the validation code not the logic associated with the core functionality.

There is a happy medium. It's important that an appropriate point on the continuum is picked based on the type of task (such as controlling the engines on a commercial jetliner engine vs. making a smart recommendation on what other articles the viewer might want to look at on a news site) and the implementation environment (such as assembly vs. Java). It's also important the everyone on the team grok where that appropriate point is for the task.

This is not new

[Anonymous Coward]

http://en.wikipedia.org/wiki/Boundary-value_analysis

Bounds test?

[jklovanc]

Has testing degraded so far that people don't now what a bounds test is?

Re:Bounds test?

[Evil Pete]

I actually read the article. You know what it is? It is a simplified tutorial on white box testing for people who don't know, that is, amateur programmers. Yeah, that would have once been new and interesting to me, when I was learning to program. Not when I was doing it for a living though. Bad title and summary.

Re:Bounds test?

[dbIII]

It was one of the points of difference between the engineering programming classes and the maths department ones. On the engineering side we were expecting things to go wrong and out of bounds while on the maths side it was all just supposed to work first time.

Re:Bounds test?

[gnasher719]

It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense.

In the 68020 processor manual, it says that determining the exact execution time of an instruction is very difficult, even if you know all the details of the processor implementation. That was 20 years ago. Today, precise timing is basically unpredictable. Then you have multi-threading, where the order of operations is basically unpredictable. Then your code is waiting for external events that are totally unpredictable. Anything that has anything to do with timing is essentially random.

No shit Sherlock

[cruachan]

You know, I remember writing test plans to to test input that were one below, at, and one above, some arbitrary limit when I was a trainee programmer coding up COBOL on mainframes back in the mid 80s.

How on earth does this drivel make it onto Slashdot? This is 30 year old news at least (which makes it straight out of the 17th C in internet years)

Re:No shit Sherlock

[Tanuki64]

Sadly, I still see developers not testing, and are practically afraid of writing test scripts.

Afraid? Less afraid than not paid for. 'Write your code correctly, then you don't need to test'. A bit simplified, but this is more or less what I heard more than once from a customer.

Look at commit time stamp.

[140Mandak262Jamuna]

Talking from personal experience, all the bugs I ever committed to production were coded in between 1 PM and 2 PM, my sleepiest time, when my stomach digesting the lunch was competing with the brain for blood supply.

If I am a columnist with some modest name recognition I could have converted this mildly amusing (to me at least) observation into a column. But alas, I am not one. So he gets to repeat the age old advice given by old Prof Mahabala, teaching Intro to Computing 201, (Fortran programming, in IBM 365/155 using punch cards no less ) back in 1980 into a column, and all I get to do is to bitch about it.

Re:No bugs are random - computers are deterministi

[msobkow]

Apparently you've never dealt with race conditions and multi-threaded code.

It's good to test the corner cases

[Black Parrot]

News at 11.

Look at the comments

[tomhath]

A study by the company I worked for a few years ago only found one thing that correlated with the number of bugs - the number of comments. It wasn't so much that the algorithm was complicated and needed explanation; more often it was just bad programmers tossing in a bunch of commented out print statements and comments that didn't accurately describe what the code was doing.

Who might know about this?

[Anonymous Brave Guy]

That depends. If you're are a die-hard TDD fan, for example, then you'll be writing your (unit) tests before you necessarily know where any such boundaries are. Moreover, there is nothing inherent in that process to bring you back to add more tests later if boundaries arise or move during refactoring. It's not hard to imagine that a developer who leaves academia and moves straight into an industrial TDD shop might never have been exposed to competing ideas.

(I am not a die-hard TDD fan, and the practical usefulness of white box testing is one of the reasons for that, but I suspect many of us have met the kind of person I'm describing above.)

Re:Who might know about this?

[Anonymous Brave Guy]

FWIW, what you're describing seems more practical to me. However, if you start from a position that "You should not be doing almost any programming until the design is done" then you're already a long way from how many TDDers practise and from the process that many who advocate TDD, including big name consultants and authors, describe.

Re:Who might know about this?

[gl4ss]

you can't write the tests before knowing what the program is supposed to do first and for that you need to know the bounds it needs to adhere to...

or well, you can write them, but then you're doing them to just fill a tickbox and to input hours into timetracking. which is fucking useless. still happens a lot though, because someone on the management chain is a tdd fan but is also an agile fan. furthermore, tests for the ui could be testing against boundary values xb while the back end was developed against unit tests with values ya - which is really what the blurb of the article is about anyways, you could have 100% unit tests for all the pieces of the code with the problem being that since nobody knew the real values at the time of development they could be different..

Re:No bugs are random - computers are deterministi

[TechyImmigrant]

Not if the different threads are clocked from different PLLs.

Re:No bugs are random - computers are deterministi

[PCM2]

It is not random. If you have enough knowledge and the ability to comprehend that knowledge, you can predict what will happen. Nothing is random.

Sure, as long as you start a program and let it run all by itself without touching anything. As soon as you introduce human input, the system may still be deterministic, but the output of the program is in effect random because the behavior of the operator cannot be predicted. The kind of "knowledge and the ability to comprehend that knowledge" that you describe is known as omniscience, and most IDEs currently don't support it.

Re:QA needs to be testing coders are poor QA / tes

[plopez]

QA ignorance is not a bad thing. QA should not be working from designers POV but from a requirement/user story POV. If it does not satisfy the requirements or the user story, it is a bad design. Period. The less QA knows of the design the better.

Worst headline of 2013?

[khb]

Bugs are never RANDOM. Bugs are, by definition, an error (human blunder ... incorrect design, improper code, etc.)

Re:The 80's called...

[Alain Williams]

And always test for the null set.

I once tried to get the following into some coding standards, unfortunately I failed: Every program and function should do nothing correctly

Re:The 80's called...

[rvw]

And always test for the null set.

I once tried to get the following into some coding standards, unfortunately I failed: Every program and function should do nothing correctly

You didn't fail, you just failed to notice that you correctly did nothing!

Re:not to a developer, programmer, or comp scienti

[DarkOx]

Even in the computer science sense the article is using random correctly. It's arguing bugs from programming errors are NOT randomly distributed through the code base but cluster around code that does certain implementation tasks.

There article is IMHO vapid, but does use it's vocabulary correctly.