Test, Test and Test Again

snikkersnak writes "Richard Collins has written a piece about developers and testers; the article is arguing that in closed development these two roles have to be chained together one-on-one in order to reproduce the 'release early and often' effect of open source development."

Comments

ABsolutly

[Ingolfke]

I am in 100^ agrement hear. Far to often developer dont test and check they're code before they release it This resultsi n a poor edn-user experience. Tey wrok two fast always trying to git on to to the next thing.

Test test tst!!$

one-to-one

[brenddie]

one tester for each developer?? bah, I have around 300 users^Wtesters.....

Re:one-to-one

[soloport]

I have around 300 users^Wtesters.....

And there's what's wrong with OSS in a nutshell.

No, the GP was referring to portion of code the GP developed for Vista.

:-p

One tester per developer?

[ErGalvao]

The article is very good, but I disagree with the part of "one tester per developer". I don't believe there is a formula to use, since each system is/can be a different case. I don't have a good theoretical background about the subject, but for me it always seemed a good idea to divide the system in specific parts like 'login routine', 'add foo routine', 'database connection' (...), and then test them.

Too amateur?

Re:One tester per developer?

[aadvancedGIR]

That's the kind of testing you can actually do fast and close to the development team, but the real problems come from the impact your code could have on parts you barely know they exist in the full system.
Parts testing is just step 0 of the testing process.

Re:One tester per developer?

[eric76]

I always test everything in detail as I write it. A good tester would be left with checking for something I missed, but that wouldn't take a full time tester.

At one time, I had a tester assigned to me half time. That was when I was given someone else's code to fix. That was the worst code I've seen outside of a classroom. The writer, for some bizarre reason, had been promoted to project manager in spite of the fact that he had only been a software developer for a couple of years or so. Prior to that, he was an insurance salesman!

I asked time and time again to let me rewrite the application instead of the useless task of fixing it. Instead, they assigned a tester to find bugs for me to fix and hinted that if I didn't get the code polished, they would fire me. He had no problem finding bugs but I had already seen the futility of trying to fix them.

I left a short time later for a new job. The task was handed over to a guy in the office next to mine who went hog wild fixing bugs for a week or two before realizing that it just didn't matter. Instead of asking permission as I had done to rewrite it, he just did it on his own. When someone would holler for a bug fix, he'd fix that bug for them and then get back to work on rewriting it.

About the time he finished the application, the project had been sold to the company for which we were developing the code. They had been sitting in on our meetings for a while and so one of the first things that company did was replace the project manager with someone who knew what he was doing. So when the other guy showed off the new code, instead of getting fired for insubordination, everyone thought it was great.

In short, the one time I had a tester assigned to me, I didn't want a tester. I wanted to start over. The rest of the time, I've had to beg to get anyone to test the code instead of just using it for its intended purpose.

Re:Wrong.

[DrSkwid]

welcome to the brave new world of 'anything will do' user driven content

Automated unit tests

[tcopeland]

I think the trick here is for the developers to write lots of automated unit tests. This catches the vast majority of obvious bugs, and lets testers become more like "power users" or something. That is to say, rather than the testers writing bugs like "I did xyz and it crashed" they can write bugs like "it'd be great if this screen had a 'quick user lookup' field". The testers then begin to think of ways to improve and streamline the application, the end users get a better experience, and the developers have more fun because they're actually adding features rather than constantly fixing easy bugs.

Andy Glover has a good blog on testing and QA in general. He uses FitNesse and TestNG and various other Java testing tools so he's pretty up on all the latest junx.

Re:Automated unit tests

[Anonymous Coward]

Oh, good idea! Have the developers do testing, which frees up the testers to do design work.

Uh...

If you think that the work of testers can be replaced by unit tests, you either have really stupid testers or really stupid developers. Or both.

Unit tests don't fix bugs. A unit test tests a unit (hence the name), so if 'xyz' is a complex setup, it's likely your unit test won't have caught it. If a developer thought it could happen, he would have fixed it to begin with. Testers are there to find bugs that developers *didn't* think of, and unit tests written by developers can only catch bugs developers *did* think of.

And counting on testers to do design work is just stupid. It takes a special kind of person to design a good user experience, and (the vast majority of) testers aren't that person. You'll end up with a frankenstein app, where each tester's pet feature is assembled into one app. If that's better than what you have now, then you're in pretty sorry shape.

Re:Automated unit tests

[ChunderDownunder]

Well... Testers can't be replaced by unit testing, nevertheless unit testing does have some merit.
When a tester spots a bug, a unit test might be written to document the behaviour, that the developer didn't think of, thereby reducing regressions.

In additional to 'bugs', testing is useful in checking whether an application performs a task as dictated by a specification, i.e. missed functionality. Unit testing won't catch bugs in code that hasn't been written in the first place!

As for the 'design work' matter, the development ought to be a collaborative effort between the Business Analyst, UI Designer/Usability specialist, Tester and Coder. The tester is often the first person to see the application. If a user interface isn't usable in the eyes of a tester it probably won't be to the end user either. But rather than delegate enhancements to the coder, the tester ought to refer these thoughts back to the usability person. As in the example, if the UI specialist agrees that yes, "it'd be great if this screen had a 'quick user lookup' field" then that could be written as a requirement in the specification in consultation with the BA. - e.g. 'the system shall provide quick user lookup'.

Agreed, it's not the role of testers, and hence developers, to go adding features just for coolness sake. That's how projects never get finished. But in testing that the software meets requirements in a specification they're also, to some degree, testing that the specification is complete and workable. And if the spec is set in stone then at least suggested enhancements can be documented for a future revision.

Uh, from my experience...

[Blakey Rat]

From my experience, open source software may be released early and often, but it sure isn't tested thoroughly. Sometimes it seems to me that it's not tested at all, when extremely blatant bugs appear in releases. (One of the recent ones I found is a Import dialog in Inkscape that doesn't alphabetize correctly. This isn't 1985, open source developers, alphabetization of a list of files is a solved problem!)

Of course, I also don't think "release early and often" is a good philosophy. If you release early, by definition you're releasing something that isn't yet ready for the public, and when the public uses it they're going to be disappointed and a lot less likely to try your next version.

Re:I stopped reading...

[ErroneousBee]

Just re-read his opening comment. I was imagining a class of physics students licking butter off of bread, and it may have led me astray.

As for the CS/Physics thing, I find physics grads (me, BTW) have a big weakness when it comes to 'correctness', and being able to simplify algorithms. If they can fix those weaknesses, they become better than ordinary CS grads. The same for CS grads not being able to break out of problems, if they can broaden their narrow focus on 'code elegance', they to can do great things.

See the original Netscape browser for an example of the awfulness of Physicist code, and regexp for an example of CS theory beating usability.

Of course, neither class of programmer have anything on an MBA implementing the company payroll as an Excel macro.

Re:Closed source does not work like open source

[TheRaven64]

Many Free Software projects (the BSDs in particular) split the tree into three sections:

1. The active development 'this may compile, and if it does, then it may work, but don't expect it to be stable' branch.
2. The 'this stuff seems to work, but may still break' branch.
3. The 'we think this stuff is stable now' branch.

On FreeBSD, for example, these three branches are called -CURRENT, -STABLE, and -RELEASE. Typically, only active developers run the -CURRENT branch, but a lot of people run -STABLE. People running either can file bugs, which make -RELEASE more stable, and people running important services run -RELEASE (and possibly keep a -STABLE box around for testing).

Contrast this with closed development. In this case, the first branch will be internal-only, and snapshots of the second my be released as betas. This reduces the number of people testing both of these branches, which then has an impact on the stability of the third.

Moo

[Chacham]

Don't forget The Career Programmer which discusses testers at length.

Thorough self-testing

[c0d3h4x0r]

I've been a professional software developer for over ten years. Based on my experience, I can say with certainty that there are three main sources of bugs.

In order, they are:

1. Lack of sensible high-level design and architecture. This causes the bugs that require massive amounts of churn or ugly hacks on top of hacks to fix.

2. Source code that was written without regard for human maintainability/understandability/readability. This causes the bugs that take the longest time to investigate and which are the trickiest to fix without breaking something else without realizing it.

3. Lazy developers who neglect to thoroughly self-test their own work before calling it finished. This causes the most bugs in gener, but fortunately most of them are easy and quick to fix as they are mere oversights.