Slashdot Mirror
Test Driven Development Examples?
esnyder queries: "I find the pragmatic/agile/XP hype about test driven development compelling, but find it hard to see how to test first (or even unit test at all) in some situations. I would like to explore some extended examples of it in a moderate to large scale real world codebase to improve my test design skills. Can anyone recommend some F/OSS software projects that consistently use test driven development processes that I could check out? Preferable over 50K lines of code, but I'd welcome pointers to anything that people think would be helpful."
Nothing reality tests the usability of a proposed API then writing unit tests against it.
For those who develop in Java, may I propose JUnit? If you want to test the GUI of a web server, then try HTTPUnit although the value of writing unit tests to this is less since GUI is usually subject to a lot of changes over time.
Take a look at Zope3 it has some 5,000 unit tests:
http://cvs.zope.org/Zope3/
Free Talk Live: Talk Radio where YOU ar
Where I work we're coding a dating site. Currently, it's up to about 180K lines of scripts written in PHP. However, not a lot of it was written with unit testing (and test-driven development) in mind. I'm a big believer in unit testing but it's been hard to get the other members on board as they complain how hard it is to write unit tests for web pages. Bah.
So, most of the library functions that I wrote (stuff like except an integer, return a text string from a list) have been unit tested by myself, and every time I change a function or a class, I try and write a unit test for it.
Seriously, you just need to dive right in and think about how you make your code easy to test. I use the SimpleTest testing framework (it's PHP), and I always feel good when my array of tests all run correctly when I make a major change to the code that impacts a huge portion of the site.
If test-driven development has done one thing, it's forced me to carefully examine my code to create a way to make sure it is actually working according to the business logic we've been asked to implement.
None of the projects I have looked at appear to have test first design. Some (especially CPAN items) have good testing of the functionality in place, but these are all modules inside a programming language. I have never seen a full on stand alone project with such testing built into the core.
I find that odd really, considering that the point behind "write tests first" is to create an executable specification for functionality. If a new piece is desired, surely it would be easier for someone else to pick up the module with the tests and write code to pass the tests than it is to try to pick up the flow of a project from out of date documents, etc, and just contribute. I look forward to those who know of such projects, because I plan to examine them for help on how to implement these techniques myself. (My current project uses regression tests driven by an external test application... I have had this same question for some time, especially in regards to web applications).
Sig under construction since 1998.
The Python programming language and all of its standard library modules make extensive use of unit tests and TDD.
It would have been helpful if you'd mentioned the language you're using, and the types of applications (since of course both of those make a huge difference).
If you're using Java for web development, I'd suggest reviewing the Struts Applications Project on SourceForge.
It's a collection of documentation and applications using Struts that are really "done right" -- with documentation, sensible and scalable design, fully implemented testing (unit level AND on the HTTP level). I'm currently roaming through the AppFuse source in there -- it's basically designed to give you a complete setup to start building your app on, with common functions already built-in.
From the site:
AppFuse
An application for starting your Struts-based applications. Checkout from CVS and execute "ant -Dapp.name=yourApp -Ddb.name=database" to instantly be up and running with a Tomcat/MySQL app. Uses Ant, XDoclet, Hibernate, JUnit, Cactus, StrutsTestCase, Canoo's WebTest, Struts Menu, Display Tag Library, JSTL and Struts (including Validator and Tiles). Features include CMA, Remember Me, Self Registration, Password Hint. The fuse to start your apps.
There are only 10 types of people: those who understand decimal, those who don't, and, uh, 8 other types I forget.
http://www.kernel.org/
Dialectician. Archology.
I'm far from convinced that TDD is actually a good approach. Although it's pretty obvious that without testing the code is often trivially buggy, and unit testing is the cheapest way to perform testing. For instance this kind of thing is all too easy to do with TDD.
For unit tests you want to write your code, and then look at the best set of unit tests to do complete code coverage. For an OSS e3xample of that you can look at Vstr string library and the code coverage for that project.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
Unit testing my code has saved me hundreds of hours of debugging time, unit tests are also great documentation. In a big system where you can't keep track of all the code and how it works in your head, unit tests act as little documentation beacons showing how some piece of code works.
Free Talk Live: Talk Radio where YOU ar
I wrote a series of tests for code I have written for what I would term a trivial application. Took me about 3 months to develop. The series of tests for about 50% of the code and missed 50%. The vast majority of the bugs came from the second half of the code.
Second. The tests themselves act as documentation. I am anoyed that I pick up generic routines like a CSV file reader and there is subroutines but no way to figure out how to use them. A test suite acts as sample code as well.
This process is balance, you don't test every detail but you do codify some tests. When you find a bug you codify a test and then you NEVER reintroduce the same bug. I find that in general I reintroduce bugs about once for every month of coding. Since this application only gets worked on in random blocks of a week here and there I am not focussed on the whole application and I simple forget. Use the computer to validate that you have not made a mistake.
These sort of tests saves my reputation, does not frustrate my users.
Sorry to let you know but, you didn't write good unit tests and probably did waste your time. I've found very close to 100% of the bugs in Vstr a network IO string library using unit tests. That includes a couple of ones that would have been damn hard to track down otherwise.
However it's been over a year since 1.0.0 which had a unit test for every function and every function option, to the last release which had over 99% code coverage found a couple of weird corner case issues (not just bugs, but optimizations that could never be reached for some reason). And going from 98% coverage to 99% coverage took a significant time investment, and required significant thinking about how the test should be written.
As with much software development, it's easy to write simple tests that don't show much and aren't very useful. It's much harder to write tests that find bugs (and you have to appraoch writing the tests with a very different mindset to how you approach writting the code you are testing. This is not even close to being "Like picking lint from your belly-button."
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
I'm sure a lot of people have grounds to object, but I suggest, depending on the nature of the project, it's a good idea to be intelligently selective about what does and does not need to be unit tested. Consider:
::sigh::
Do write a test case if:
+ a failure could introduce data propagation issues
+ it performs some intricate mathematical or logical function whose result must be precise
+ you're writing test cases to hunt down bugs that you know are in you're code; keep those test cases
+ you're uncomfortable about the quality of your code
+ an error might kill someone, or otherwise be Really Bad
Don't bother writing a test case if:
- you can use a guard clause or assertion instead
- a failure in the code will otherwise be immediately obvious
- the code generates massive amounts of data which need not be mathematically precise (i.e. graphical output)
- you don't feel it
I should probably write more test cases, according to my own guidelines.
Upstairs Dog, Downstairs People.
My experience has been that "unit testing" slows development to a painfully slow crawl.
This is a common miconception about test-driven development. Sure, when you look at the speed at which new features go in, "development" seems to be slowing to a crawl.
But this is a very short-sighted and limited view of the whole project lifecycle. I've found that having great unit tests for my code actually help in the long run for adding new features. And that's not just adding the features to the code base, that's all the way to the end -- making sure they work properly.
Nevermind the benefit they give while refactoring in an agile development process: the unit tests become a sort contract you can guarantee will work for objects that use your code, no matter the internals of a function. For refactoring this is invaluable.
In short, you're not alone. Very few developers think long term like this and it's a shame. But once you drink the unit testing/test driven design (TDD) kool-aid I guarantee you'll never go back. I wouldn't think of writing code without unit tests any more and quite often it's TDD.
----- rL
The first wow, that's cool moment I had with test driven development was from an article on Object Mentor called The Bowling Game. It also highlights pair programming a bit.
The unique thing about the article was that it was presented as a discussion between two developers pair programming doing agile test driven development of the game. It was like watching over their shoulders.
If you want to get an idea of what extreme programming is like, I suggest reading this article AND writing the code and tests along with it, either in Java (and JUnit) or C#/VB.NET (and NUnit) or another language with a xUnit unit testing framework. Most object oriented languages have them now so you don't have to roll your own framework.
----- rL
I've given it enough of a chance that I wrote unit tests for about 3 small projects and developed my own framework for it.
I had three primary problems with it.
#1: My biggest problems always revolve around the user interface. Users encounter FAR more bugs than my internals do, and I never found a good way to unit test GUIs.
#2: I spent many hours developing tests for things that ended up rarely/never choking my tests, meaning they rarely/never caught any bugs. Yet, I still had bugs to work out. I found that it just wasn't worth the time.
#3: Whenever I re-factor, unless I'm Mr. Perfect and all my interfaces are perfect from the get-go, I end up having to edit a bunch of unit tests to match up with the new, re-factored code.
I know the party line on this, but I think it's just the current fad.
Where I work, unit tests are a requirement, not a luxury. WIthout fail, the projects with the most unit tests have the fewest bugs, and the projects with the fewest unit tests have the most bugs. I'm not exagerating.
My question to you is, if you have no unit tests, how can you be sure that the code works at all? Manual testing? Do you cover all the code paths with manual testing? It must take forever, or your application is rather trivial.
Creating mock objects is much simpler than creating a "text X server", although admitedly a wm is slightly harder than normal as you can't take the easy route of running a seperate version on your main display.
Howewver taking a quick look at blackbox, textPropertyToString() is the only thing in Util.cc that couldn't trivially be unit tested and at least all of i18n.cc and Timer.cc. That's 3% with basically no changes.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
SCons is a next-generation build tool, or make tool, written in Python with strong cross-platform support, integrated autoconf-like functionality, a lot of stable features, and a growing user community. We're currently at 14K+ lines of non-comment, non-blank source code, and 32K+ lines of non-comment, non-blank test code.
We use a combination of two different testing methodologies: 1) Individual modules all have PyUnit unit tests (similar to JUnit, but in Python, of course). 2) The SCons application itself is tested using a custom testing module that manages creation of temporary directories and files, execution of the application, and checking against expected results. This custom module is actually a wrapper around a generic "test any script/command" infrastructure module that could be easily used to test other scripts and/or commands. (The command under test could be implemented in any language, not just in Python.)
I use the Aegis change management system to manage the SCons development and testing cycle. Aegis' primary value add (for me) is its management of the test cases and the testing methodology it enforces. By default, all Aegis changes must have one or more modified or new tests. The new/modified tests must not only pass when run against the new code, but must (by default) fail when run against the old code. This helps guarantee that your tests are good, and that your code isn't passing because you made a mistake in your test and forgot to call the new feature.
By testing in this fashion from day one, we've built up a very strong regression test base--284 test scripts at last count, each script containing multiple individual tests. This test base has become crucial to our ability to refactor (and refactor and refactor...) the internals as we add more features. Sometimes it takes longer, of course, to make a rewrite satisfy all of the regression tests, but when you're done, you can be pretty sure you haven't broken anything. And if you did break something, then you have to add or modify a test when you fix it, and that becomes another part of the regression test base.
The key to getting going with this kind of test-driven development (in my opinion) is making writing and executing the tests as simple as possible (but no simpler!). If writing a test is too difficult, then a lot of developers will simply avoid it. But if you can get them over the initial hump by making it easy to write tests, it gets downright addictive because you get all of this positive feedback when your tests show you that your new code works.
We'd be glad to have you check out the testing infrastructure we've developed for SCons, either for code you can actually use, or simply as a source of ideas. Feel free to contact the SCons development team if you have any questions.
I have developed code of significance without unit testing, and I can assure you that the quality of that code was far inferior to the quality of the code developed with more attention paid to unit tests. In the projects I've worked on, sometimes the manager has made the decision to skip unit tests in order to concentrate on features. It's true that those features can be worked on at a faster rate, and seemingly completed sooner. But inevitably, the number of bugs found in QA, and the amount of time spent revising the code eats up any savings. Even if we break even, in terms of time (which, in my experience, would be a generous assumption), then the non-unit-tested project finishes at the same time but has no suite of tests to carry forward to the next version.
Furthermore, the benefits of testing are not just related to finding bugs sooner in the initial version. The tests can help influence the design of the code. I've written code where I ended up refactoring it completely while writing the tests, because the initial version, while it worked, didn't wasn't easily testable or had too many redundancies in the code. The quality of the run-time product was the same, but the source code improved dramatically just because I wrote tests for it.
Also, having the unit tests is a good way to ensure that bugs aren't introduced later. In short, the benefits are so high that, while I can imagine (and have actually done it) working on a project with no tests, it's not a task that I would welcome. It's similar to asking me to work on a task with no debugger, or no source-code control system. Yes, it's possible, and heck, the Linux kernel was written without either of these things, but realistically these are vital tools for software development. To me, unit tests are another of those vital tools.
#1: My biggest problems always revolve around the user interface.
But what would you rather debug: 1) A problem in the user interface you know has to be in the user interface because you have confidence that it's not below it or 2) the whole thing all the way down all of the method calls? If you have unit tests in the rest of the code, debugging the UI is much easier.
#2: I spent many hours developing tests for things that ended up rarely/never choking my tests....
Making tests for situations that work is still valuable. If you make such a test and it fails later on, you'll know you had a regression in the code somewhere. This is very handy to have when refactoring as well.
#3: Whenever I re-factor, unless I'm Mr. Perfect and all my interfaces are perfect from the get-go...
Interfaces can be refactored too, and of course you'll break tests and have to rewrite things because now you're expecting a new set of behaviour.
I've found that this has the side benefit of forcing people to really think about changes they make to their code and not just hack away at it. If people know they have to write tests too they seem to put more thought into it ahead of time.
Anyway, that's just my personal experience with unit testing.
----- rL
#1: No, the bugs aren't in the UI itself, the bugs originate through the UI. Users can do things in such a way that you simply can't predict.
#2: But you know, if you have time, there are LOTS of things you can do that might help, or might not. I stopped writing unit tests thinking that they one day *might* catch a bug. If I don't have an immediate or at least imminent benefit from it, I don't do it. It's simply a waste of time.
#3: I don't hack away at code, so I don't think unit tests will save me from something I don't do. My interfaces are developed exactly at the level of quality called for; no more and no less. So in the sense that unit tests can help me "think"...I never found myself lacking in that area. Perhaps it's something that helps less experienced programmers ramp up...I don't know. I've been programming for 20+ years and I'm alright in that regard.
I wholeheartedly disbelieve that the time you saved skipping unit tests was eaten up at QA. I've done it both ways, and it's much, much faster developing in the requirements. Unit tests are good quality assurance, but they are NOT a time-saver.
You also touched on another problem I have with unit testing: it influences the design pattern too much. I don't believe that the design which best fits with the unit testing framework is the best design pattern for the project. If your only goal is to make lots of happy unit tests, yeah, then alter your design to suit the tests...but that's WAY too restrictive for me. There are lots and lots of design patterns that are tough to write unit tests for, and the last thing I need is the wrong wheel squeaking for grease. I want my patterns to focus on the needs of the project, not the needs of a unit testing framework.
And check out the Boost unit test libraries. http://boost.org/libs/test/doc/index.html
Try the book called "Test Driven Development: By Example" by Kent Beck!
.. using the framework he is writing to write the framework! Here he shows you how to bootstrap a new testing framework, which you probably won't ever need to do, but it was cool. He starts with a single test that prints "1" or "0" on the screen for pass/fail, and builds on that.
It's a neat little book. The first third demonstrates writing a Money class in Java using test-driven development. It's kinda like you're sitting next to him and he is demonstrating. In that example he shows a lot of what comes up in an average TDD session. If you're new to it, you might not realize how fluid it is, and how tiny the steps can be. Even if you *think* you know what TDD is about from reading a description, this will really show you how it works.
Then the next chapter is really cool.. here he writes a unit test framework in Python
Final third of the book is patterns, example, theory, philosphy, somewhat interesting.
This is a book you'll only read once or twice, honestly, but you will benefit greatly from it. In my opinion.
Software Development Magazine has been running a series for about the past year called The Craftsman written by Robert C. Martin. It focuses on a young apprentice writing a java app with his mentors. Nothing is done that doesn't involve a test first.
While the series itself is kind of slow, it is a good introduction to TDD, and I really enjoy Robert's writing style. Might be able to lead you to some more examples.
Random Musings
It's not *exactly* test driven, but gcc has a very extensive test suite, and adds regression tests for every new bug that is uncovered. The project certainly passes your 50kloc specification!
http://gcc.gnu.org/install/test.html
I bought this book, and it was really useful in the sense of basic training, but it does not tackle any of the difficult questions, such as mock objects, user interfaces, etc.
I am left still wanting a nice non-trivial working example to look at.
Most writers regard truth as their most valuable possession, and therefore are most economical in its use - Mark Twain
First, IMO, html is the UI ... and the best way to test that isn't with a unit test (that I've seen). You want all your UI functions to just glue sql/data commands to the UI. Then you can "easily" test the sql/data commands with a test database and a bunch of function calls ... then you'll know that any bugs you find in the html part must be the fault of the UI code (which should be fairly small).
As far as the testing DB, I'd recommend having testing data outside the DB and use something to init. the database just before you run your unit tests. This should be easier to keep upto date, and solves problems of testing alterations to the DB.
Having an external representation of the DB that can be reloaded should solve this problem.
The only DB thing I've seen is for Java that was on testdriven.com.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
Others have already commented on their varying mileage with this topic, I too believe that TDD (done correct) will rather save time than cost more time. At least in the long run. (I'll leave the "done correct" part to somebody else, books and articles have been recommended in other comments).
There's one misconception that I always see with TDD and Unittesting: Both are not the same.
In my eyes, TDD is rather a design strategy than a testing strategy. It certainly gives a test suite as a very nice "byproduct", but first and foremost it forces you to think about what your code is supposed to do and write it down. Usually TDD code looks completely different from non-TDD code.
So you'll have to add some portion of time taken for upfront design to the time you take to write code in order to compare writing testdriven with non-testdriven code.
This doesn't mean, that the statement commented was wrong: I also believe that you have to manage some more or less steep learning curve in order to be able to really do testdriven development, just as you have to learn a lot about each process, language or framework you work with. I'd certainly be quicker writing a small java tool than writing equivalent five lines of PERL because I've never really (wanted to) manage the PERL learning curve.