Slashdot Mirror
Test Driven Development Examples?
esnyder queries: "I find the pragmatic/agile/XP hype about test driven development compelling, but find it hard to see how to test first (or even unit test at all) in some situations. I would like to explore some extended examples of it in a moderate to large scale real world codebase to improve my test design skills. Can anyone recommend some F/OSS software projects that consistently use test driven development processes that I could check out? Preferable over 50K lines of code, but I'd welcome pointers to anything that people think would be helpful."
Nothing reality tests the usability of a proposed API then writing unit tests against it.
For those who develop in Java, may I propose JUnit? If you want to test the GUI of a web server, then try HTTPUnit although the value of writing unit tests to this is less since GUI is usually subject to a lot of changes over time.
Take a look at Zope3 it has some 5,000 unit tests:
http://cvs.zope.org/Zope3/
Free Talk Live: Talk Radio where YOU ar
Where I work we're coding a dating site. Currently, it's up to about 180K lines of scripts written in PHP. However, not a lot of it was written with unit testing (and test-driven development) in mind. I'm a big believer in unit testing but it's been hard to get the other members on board as they complain how hard it is to write unit tests for web pages. Bah.
So, most of the library functions that I wrote (stuff like except an integer, return a text string from a list) have been unit tested by myself, and every time I change a function or a class, I try and write a unit test for it.
Seriously, you just need to dive right in and think about how you make your code easy to test. I use the SimpleTest testing framework (it's PHP), and I always feel good when my array of tests all run correctly when I make a major change to the code that impacts a huge portion of the site.
If test-driven development has done one thing, it's forced me to carefully examine my code to create a way to make sure it is actually working according to the business logic we've been asked to implement.
None of the projects I have looked at appear to have test first design. Some (especially CPAN items) have good testing of the functionality in place, but these are all modules inside a programming language. I have never seen a full on stand alone project with such testing built into the core.
I find that odd really, considering that the point behind "write tests first" is to create an executable specification for functionality. If a new piece is desired, surely it would be easier for someone else to pick up the module with the tests and write code to pass the tests than it is to try to pick up the flow of a project from out of date documents, etc, and just contribute. I look forward to those who know of such projects, because I plan to examine them for help on how to implement these techniques myself. (My current project uses regression tests driven by an external test application... I have had this same question for some time, especially in regards to web applications).
Sig under construction since 1998.
The Python programming language and all of its standard library modules make extensive use of unit tests and TDD.
Twisted is a networking framework in Python which has a lot of unit tests:
http://www.twistedmatrix.com
Free Talk Live: Talk Radio where YOU ar
It would have been helpful if you'd mentioned the language you're using, and the types of applications (since of course both of those make a huge difference).
If you're using Java for web development, I'd suggest reviewing the Struts Applications Project on SourceForge.
It's a collection of documentation and applications using Struts that are really "done right" -- with documentation, sensible and scalable design, fully implemented testing (unit level AND on the HTTP level). I'm currently roaming through the AppFuse source in there -- it's basically designed to give you a complete setup to start building your app on, with common functions already built-in.
From the site:
AppFuse
An application for starting your Struts-based applications. Checkout from CVS and execute "ant -Dapp.name=yourApp -Ddb.name=database" to instantly be up and running with a Tomcat/MySQL app. Uses Ant, XDoclet, Hibernate, JUnit, Cactus, StrutsTestCase, Canoo's WebTest, Struts Menu, Display Tag Library, JSTL and Struts (including Validator and Tiles). Features include CMA, Remember Me, Self Registration, Password Hint. The fuse to start your apps.
There are only 10 types of people: those who understand decimal, those who don't, and, uh, 8 other types I forget.
My experience has been that "unit testing" slows development to a painfully slow crawl. I find that the "gain" in quality I get is offset heavily by the reduction (or slow implementation) of features (usability, requirements, etc.). I did unit testing (once upon a time), and even developed my own test suite for C++, but I find that it catches VERY few bugs and I end up spending time writing unit tests AS WELL AS hunting down bugs the same old ways I always have. I just stopped bothering; I personally got little or nothing out of them.
They are, however, sort of fun to write. Like picking lint from your belly-button.
http://www.kernel.org/
Dialectician. Archology.
I'm far from convinced that TDD is actually a good approach. Although it's pretty obvious that without testing the code is often trivially buggy, and unit testing is the cheapest way to perform testing. For instance this kind of thing is all too easy to do with TDD.
For unit tests you want to write your code, and then look at the best set of unit tests to do complete code coverage. For an OSS e3xample of that you can look at Vstr string library and the code coverage for that project.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
My main OSS project is a window manager. How do you unit test that? It is nearly impossible without writing your own test X server and the like. Just not worth it.
We write or bring in small programs which test how we handle applications or events. But it is still really hard to test the whole thing.
I wrote a series of tests for code I have written for what I would term a trivial application. Took me about 3 months to develop. The series of tests for about 50% of the code and missed 50%. The vast majority of the bugs came from the second half of the code.
Second. The tests themselves act as documentation. I am anoyed that I pick up generic routines like a CSV file reader and there is subroutines but no way to figure out how to use them. A test suite acts as sample code as well.
This process is balance, you don't test every detail but you do codify some tests. When you find a bug you codify a test and then you NEVER reintroduce the same bug. I find that in general I reintroduce bugs about once for every month of coding. Since this application only gets worked on in random blocks of a week here and there I am not focussed on the whole application and I simple forget. Use the computer to validate that you have not made a mistake.
These sort of tests saves my reputation, does not frustrate my users.
I'm sure a lot of people have grounds to object, but I suggest, depending on the nature of the project, it's a good idea to be intelligently selective about what does and does not need to be unit tested. Consider:
::sigh::
Do write a test case if:
+ a failure could introduce data propagation issues
+ it performs some intricate mathematical or logical function whose result must be precise
+ you're writing test cases to hunt down bugs that you know are in you're code; keep those test cases
+ you're uncomfortable about the quality of your code
+ an error might kill someone, or otherwise be Really Bad
Don't bother writing a test case if:
- you can use a guard clause or assertion instead
- a failure in the code will otherwise be immediately obvious
- the code generates massive amounts of data which need not be mathematically precise (i.e. graphical output)
- you don't feel it
I should probably write more test cases, according to my own guidelines.
Upstairs Dog, Downstairs People.
The first wow, that's cool moment I had with test driven development was from an article on Object Mentor called The Bowling Game. It also highlights pair programming a bit.
The unique thing about the article was that it was presented as a discussion between two developers pair programming doing agile test driven development of the game. It was like watching over their shoulders.
If you want to get an idea of what extreme programming is like, I suggest reading this article AND writing the code and tests along with it, either in Java (and JUnit) or C#/VB.NET (and NUnit) or another language with a xUnit unit testing framework. Most object oriented languages have them now so you don't have to roll your own framework.
----- rL
Portland Pattern Repository.
Any ideas? I'd like to write tests for top.
SCons is a next-generation build tool, or make tool, written in Python with strong cross-platform support, integrated autoconf-like functionality, a lot of stable features, and a growing user community. We're currently at 14K+ lines of non-comment, non-blank source code, and 32K+ lines of non-comment, non-blank test code.
We use a combination of two different testing methodologies: 1) Individual modules all have PyUnit unit tests (similar to JUnit, but in Python, of course). 2) The SCons application itself is tested using a custom testing module that manages creation of temporary directories and files, execution of the application, and checking against expected results. This custom module is actually a wrapper around a generic "test any script/command" infrastructure module that could be easily used to test other scripts and/or commands. (The command under test could be implemented in any language, not just in Python.)
I use the Aegis change management system to manage the SCons development and testing cycle. Aegis' primary value add (for me) is its management of the test cases and the testing methodology it enforces. By default, all Aegis changes must have one or more modified or new tests. The new/modified tests must not only pass when run against the new code, but must (by default) fail when run against the old code. This helps guarantee that your tests are good, and that your code isn't passing because you made a mistake in your test and forgot to call the new feature.
By testing in this fashion from day one, we've built up a very strong regression test base--284 test scripts at last count, each script containing multiple individual tests. This test base has become crucial to our ability to refactor (and refactor and refactor...) the internals as we add more features. Sometimes it takes longer, of course, to make a rewrite satisfy all of the regression tests, but when you're done, you can be pretty sure you haven't broken anything. And if you did break something, then you have to add or modify a test when you fix it, and that becomes another part of the regression test base.
The key to getting going with this kind of test-driven development (in my opinion) is making writing and executing the tests as simple as possible (but no simpler!). If writing a test is too difficult, then a lot of developers will simply avoid it. But if you can get them over the initial hump by making it easy to write tests, it gets downright addictive because you get all of this positive feedback when your tests show you that your new code works.
We'd be glad to have you check out the testing infrastructure we've developed for SCons, either for code you can actually use, or simply as a source of ideas. Feel free to contact the SCons development team if you have any questions.
And check out the Boost unit test libraries. http://boost.org/libs/test/doc/index.html
Try the book called "Test Driven Development: By Example" by Kent Beck!
.. using the framework he is writing to write the framework! Here he shows you how to bootstrap a new testing framework, which you probably won't ever need to do, but it was cool. He starts with a single test that prints "1" or "0" on the screen for pass/fail, and builds on that.
It's a neat little book. The first third demonstrates writing a Money class in Java using test-driven development. It's kinda like you're sitting next to him and he is demonstrating. In that example he shows a lot of what comes up in an average TDD session. If you're new to it, you might not realize how fluid it is, and how tiny the steps can be. Even if you *think* you know what TDD is about from reading a description, this will really show you how it works.
Then the next chapter is really cool.. here he writes a unit test framework in Python
Final third of the book is patterns, example, theory, philosphy, somewhat interesting.
This is a book you'll only read once or twice, honestly, but you will benefit greatly from it. In my opinion.
Thanks for the input; I'll give it a look. But what I'm really after is larger scale real world systems. I've read quite a few books that walk you through the process, and I've written my share of tests on various projects, but there's nothing like full scale examples to point out the pitfalls.
Emile Snyder
www.talentcodeworks.com
Add Tcl and Tk to the list:-) They have a big test suite which pokes and prods at as many nooks and crannies of the language as possible.
http://www.welton.it/davidw/
Software Development Magazine has been running a series for about the past year called The Craftsman written by Robert C. Martin. It focuses on a young apprentice writing a java app with his mentors. Nothing is done that doesn't involve a test first.
While the series itself is kind of slow, it is a good introduction to TDD, and I really enjoy Robert's writing style. Might be able to lead you to some more examples.
Random Musings
I have been a proponent of unit testing, facing a lot of flak from the team members, project managers, etc for the added time in the development cycle. Over time, following are the observations that I've made about unit testing:
#1. Don't test the obvious - enthusiastic new developers (or naive old developers) tend to write miniscule tests (like testing the accessor methods of a class), which are equivalent to testing the compiler. Avoid them at all costs!
#2. Test for the obvious - if you wrote a method that expects a number and uses it to divide something else, don't write tests to catch divide by zero, not a number exception, etc. Test the boundary conditions of your unit and what it is expected to do!
#3. Remember you are testing the program unit - dont let loose your imagination and spend time thinking of new ways to break your code. Leave that for the testing stage.
#4. And finally update the tests along with the code - its tedious, we all hate it, but someone else wouldnt have to dig through your code to find out what was the change you did that broke the framework!
http://efil.blogspot.com/
Therefore most test first projects are probably closed source. I can't show you our whole code base but we are a TestFirst XP place with 1654 unit tests currently in place.
For large scale testing mock objects are invaluable. See www.mockobjects.com and www.mockmaker.org
Here is an example of what a test looks like using Junit and MockObject testing. It might give you a feel for how easy the tests are to write.It's a simple test - most tests have more mockobjects. Once you've written that test and it fails you modify the myServlet.processRequest(..) method to pass the test. Then you run AllTests get a green bar, sync with the repository system and go have a cup of tea and some marmite toast.
Matt.
It's not *exactly* test driven, but gcc has a very extensive test suite, and adds regression tests for every new bug that is uncovered. The project certainly passes your 50kloc specification!
http://gcc.gnu.org/install/test.html
I bought this book, and it was really useful in the sense of basic training, but it does not tackle any of the difficult questions, such as mock objects, user interfaces, etc.
I am left still wanting a nice non-trivial working example to look at.
Most writers regard truth as their most valuable possession, and therefore are most economical in its use - Mark Twain
I am very curious about TDD and would like to get in deeper, but all of the tutorials that I have seen are writing OO type applications. How would you make use of TDD if you were writing web applications using ASP/VBScript or PHP? Is it possible.
Thanks in advance for your replies.
1. Bad signature
2. ?????
3. Profit
Besides being an excellent development environment with support for XP (unit testing, refactoring, etc), it's developed with Unit testing galore. See http://www.eclipse.org and look at the downloads section. There are source downloads as well as complete tests. Excellent examples of Java development with patterns, testing, and refactoring.
I know this thread is a bit old, but I'll chime in anyway...
I generally do not use unit testing. However, I can think of a few occasions where I have used unit testing. I remember thinking, "I've written all this code, but how do I know it works?". Those situations demanded some test code.
The code to be tested was low-level stuff, but non-trivial. Several hundred to several thousand lines of code that was very tightly inter-dependent. One of the smaller ones was a collection class, one of the larger ones was a persistence layer. Individually, each of the bits of code looked fine on review, but whether or not the bits would all work together in concert was too complex to verify by looking. So rather than immediately writing other code that used this low-ish level stuff as a base and tracking down the bugs later, test code was written.
In these cases, it was obvious that unit testing was required. The test code itself was in all cases not more than a few dozen to a couple hundred lines, or somewhere around 2-10 % the size of the code to be tested. Not a lot of code, and well worth it in those instances.
For simpler code, I think the tests would be quite a lot of code relative to the amount of code to be tested, and so the (whatever the coder equivalent of ROI is) would be less, though maybe still worth it in some cases. For higher-level code, the behavior would be so complex that writing meaningful tests would require a hell of a lot of work, and again the (coder ROI) would have been less, though again maybe still worth it in some cases.
It seems to me that there is a "sweet spot" where unit testing is a must, and other cases where it is worth it but not really important, and still other cases where the code is so trivial or so complex that unit tests are more effort than they're worth.
And the TDD book by Kent Beck shows how the evolution of an application can be driven by unit tests.
Fwiw - I'd suggest that a large project is _not_ the best way to get to understand how unit testing works - there is likely to be way more detail than you need or want. Matt Raible shares some code (esp. AppFuse) which includes unit tests, and describes them in his wiki on Appfuse tutorials.
His "Struts Resume" demo application contains some pretty interesting unit tests of a real-world application.
Now, here's the skinny for me : using testing as an integral part of your development cycle helps you to separate the layers and components in your application. Every time you find yourself thinking "how am I going to test this - it's too difficult/intertwingled/dependent on other stuff", you have identified a design deficiency. Making the software easy to test will make it easy to understand, and easy to work with in future. By architecting the application to make it easy to test independently of the database, you make it resilient against changes in the underlying database technology, and - oh, yeah, you have some self-documenting code to show others how to call your api.
Good place to start reading is Robert Martin's "Agile Software Development".
It's all very well in practice, but it will never work in theory.