Test Driven Development Examples?

esnyder queries: "I find the pragmatic/agile/XP hype about test driven development compelling, but find it hard to see how to test first (or even unit test at all) in some situations. I would like to explore some extended examples of it in a moderate to large scale real world codebase to improve my test design skills. Can anyone recommend some F/OSS software projects that consistently use test driven development processes that I could check out? Preferable over 50K lines of code, but I'd welcome pointers to anything that people think would be helpful."

Comments

eating your own dogfood

[anomalous cohort]

Nothing reality tests the usability of a proposed API then writing unit tests against it.

For those who develop in Java, may I propose JUnit? If you want to test the GUI of a web server, then try HTTPUnit although the value of writing unit tests to this is less since GUI is usually subject to a lot of changes over time.

Darn good question.

[Godeke]

None of the projects I have looked at appear to have test first design. Some (especially CPAN items) have good testing of the functionality in place, but these are all modules inside a programming language. I have never seen a full on stand alone project with such testing built into the core.

I find that odd really, considering that the point behind "write tests first" is to create an executable specification for functionality. If a new piece is desired, surely it would be easier for someone else to pick up the module with the tests and write code to pass the tests than it is to try to pick up the flow of a project from out of date documents, etc, and just contribute. I look forward to those who know of such projects, because I plan to examine them for help on how to implement these techniques myself. (My current project uses regression tests driven by an external test application... I have had this same question for some time, especially in regards to web applications).

Good idea, but a time-sucker

[Cranx]

My experience has been that "unit testing" slows development to a painfully slow crawl. I find that the "gain" in quality I get is offset heavily by the reduction (or slow implementation) of features (usability, requirements, etc.). I did unit testing (once upon a time), and even developed my own test suite for C++, but I find that it catches VERY few bugs and I end up spending time writing unit tests AS WELL AS hunting down bugs the same old ways I always have. I just stopped bothering; I personally got little or nothing out of them.

They are, however, sort of fun to write. Like picking lint from your belly-button.

Re:Good idea, but a time-sucker

[eukreign]

Unit testing my code has saved me hundreds of hours of debugging time, unit tests are also great documentation. In a big system where you can't keep track of all the code and how it works in your head, unit tests act as little documentation beacons showing how some piece of code works.

Re:Good idea, but a time-sucker

[Nevyn]

I did unit testing (once upon a time), and even developed my own test suite for C++, but I find that it catches VERY few bugs and I end up spending time writing unit tests AS WELL AS hunting down bugs the same old ways I always have.

Sorry to let you know but, you didn't write good unit tests and probably did waste your time. I've found very close to 100% of the bugs in Vstr a network IO string library using unit tests. That includes a couple of ones that would have been damn hard to track down otherwise.

However it's been over a year since 1.0.0 which had a unit test for every function and every function option, to the last release which had over 99% code coverage found a couple of weird corner case issues (not just bugs, but optimizations that could never be reached for some reason). And going from 98% coverage to 99% coverage took a significant time investment, and required significant thinking about how the test should be written.

As with much software development, it's easy to write simple tests that don't show much and aren't very useful. It's much harder to write tests that find bugs (and you have to appraoch writing the tests with a very different mindset to how you approach writting the code you are testing. This is not even close to being "Like picking lint from your belly-button."

Re:Good idea, but a time-sucker

[rlowe69]

My experience has been that "unit testing" slows development to a painfully slow crawl.

This is a common miconception about test-driven development. Sure, when you look at the speed at which new features go in, "development" seems to be slowing to a crawl.

But this is a very short-sighted and limited view of the whole project lifecycle. I've found that having great unit tests for my code actually help in the long run for adding new features. And that's not just adding the features to the code base, that's all the way to the end -- making sure they work properly.

Nevermind the benefit they give while refactoring in an agile development process: the unit tests become a sort contract you can guarantee will work for objects that use your code, no matter the internals of a function. For refactoring this is invaluable.

In short, you're not alone. Very few developers think long term like this and it's a shame. But once you drink the unit testing/test driven design (TDD) kool-aid I guarantee you'll never go back. I wouldn't think of writing code without unit tests any more and quite often it's TDD.

Re:Good idea, but a time-sucker

[Anonymous Coward]

Then you're not writing the tests correctly. In my experience the quality of code that has been developed in conjunction with (or better, after the tests are written) is consistently better than code that is written and never tested except as part of the greater context.

The real win for unit tests is when you modify the code later on. If the unit tests pass, then you can be pretty darn sure that any code that depends on the modified code will continue to work. This has saved me hundreds of times.

Read Testing Extreme Programming and Fowler's Refactoring.

Re:Good idea, but a time-sucker

[Mr. Shiny And New]

Where I work, unit tests are a requirement, not a luxury. WIthout fail, the projects with the most unit tests have the fewest bugs, and the projects with the fewest unit tests have the most bugs. I'm not exagerating.

My question to you is, if you have no unit tests, how can you be sure that the code works at all? Manual testing? Do you cover all the code paths with manual testing? It must take forever, or your application is rather trivial.

Re:Good idea, but a time-sucker

[Cranx]

Well, let's just say that the OVERWHELMING MAJORITY of software you use right now was developed WITHOUT the benefit of unit testing.

Have you never developed anything of significance without unit testing? I ask because it seems like you can't understand how it could be done. It not only CAN be done, it IS done, and some very solid, worthwhile code has been developed with absolutely no unit testing framework whatsoever.

But in answer to your question, (please understand that whatever I say cannot be construed as a template for ALL cases) in general, my development goes along these lines:

A rough blocking of the modules involved takes place. Basic interfaces between modules are planned tentatively. Development usually starts from the whole to the part, but sometimes modules are developed separately and integrated in later. At various points, re-factoring and code clean up. Interfaces are refined as things progress. Code is tested at intervals during development, and the tests are usually thrown away, but sometimes saved (depends on how stable the interface design is). At later stages, betas are delivered to the client and/or testers and the software is run through tests described on paper. After a certain period of time, when the software is deemed stable enough, it is considered "shipped" and checks are cut.

Re:Good idea, but a time-sucker

[Cranx]

I wholeheartedly disbelieve that the time you saved skipping unit tests was eaten up at QA. I've done it both ways, and it's much, much faster developing in the requirements. Unit tests are good quality assurance, but they are NOT a time-saver.

You also touched on another problem I have with unit testing: it influences the design pattern too much. I don't believe that the design which best fits with the unit testing framework is the best design pattern for the project. If your only goal is to make lots of happy unit tests, yeah, then alter your design to suit the tests...but that's WAY too restrictive for me. There are lots and lots of design patterns that are tough to write unit tests for, and the last thing I need is the wrong wheel squeaking for grease. I want my patterns to focus on the needs of the project, not the needs of a unit testing framework.

Re:Good idea, but a time-sucker

[okock]

Others have already commented on their varying mileage with this topic, I too believe that TDD (done correct) will rather save time than cost more time. At least in the long run. (I'll leave the "done correct" part to somebody else, books and articles have been recommended in other comments).

There's one misconception that I always see with TDD and Unittesting: Both are not the same.

In my eyes, TDD is rather a design strategy than a testing strategy. It certainly gives a test suite as a very nice "byproduct", but first and foremost it forces you to think about what your code is supposed to do and write it down. Usually TDD code looks completely different from non-TDD code.

So you'll have to add some portion of time taken for upfront design to the time you take to write code in order to compare writing testdriven with non-testdriven code.

This doesn't mean, that the statement commented was wrong: I also believe that you have to manage some more or less steep learning curve in order to be able to really do testdriven development, just as you have to learn a lot about each process, language or framework you work with. I'd certainly be quicker writing a small java tool than writing equivalent five lines of PERL because I've never really (wanted to) manage the PERL learning curve.

A more positive veiw

[oo_waratah]

I wrote a series of tests for code I have written for what I would term a trivial application. Took me about 3 months to develop. The series of tests for about 50% of the code and missed 50%. The vast majority of the bugs came from the second half of the code.

Second. The tests themselves act as documentation. I am anoyed that I pick up generic routines like a CSV file reader and there is subroutines but no way to figure out how to use them. A test suite acts as sample code as well.

This process is balance, you don't test every detail but you do codify some tests. When you find a bug you codify a test and then you NEVER reintroduce the same bug. I find that in general I reintroduce bugs about once for every month of coding. Since this application only gets worked on in random blocks of a week here and there I am not focussed on the whole application and I simple forget. Use the computer to validate that you have not made a mistake.

These sort of tests saves my reputation, does not frustrate my users.

I keep junit around

[ReciprocityProject]

I'm sure a lot of people have grounds to object, but I suggest, depending on the nature of the project, it's a good idea to be intelligently selective about what does and does not need to be unit tested. Consider:

Do write a test case if:

+ a failure could introduce data propagation issues
+ it performs some intricate mathematical or logical function whose result must be precise
+ you're writing test cases to hunt down bugs that you know are in you're code; keep those test cases
+ you're uncomfortable about the quality of your code
+ an error might kill someone, or otherwise be Really Bad

Don't bother writing a test case if:

- you can use a guard clause or assertion instead
- a failure in the code will otherwise be immediately obvious
- the code generates massive amounts of data which need not be mathematically precise (i.e. graphical output)
- you don't feel it

I should probably write more test cases, according to my own guidelines. ::sigh::