Slashdot Mirror
Why We Refactored JUnit
Bill Venners writes "In this article, three programmers tell the story of how their frustration with JUnit's API led to the creation of Artima SuiteRunner, a free, open source test toolkit and JUnit runner. These programmers simply wanted to create a small add-on tool to JUnit, but found JUnit's design non-intuitive and API documention poor. After spending time reading through JUnit's source code and attempting to guess at the API contracts, they gave up and rewrote it."
One of the biggest problems for a testing framework is fitting all the tests you need in it. Data types such as XML make it really, really difficult.
For example, you have a class that generates an XML document and you want to test it. Which level do you run the tests at? Best to test all levels, of course. How then do you check the accuracy of intermediate results? The answer often is that you can't; the intermediate results are meaningless until you have the complete (or at least almost complete) infrastructure to interpret the data. But once you've run all that, it is very difficult to pinpoint the source of the problem.
I'm glad that Artima SuiteRunner provides a way to handle all of that complexity and write unit tests that are complete but tractable. I just wish it worked with Perl, since I don't use Java anyway.
I find this quite amusing. I recently attended a talk in Montreal given by Kent Beck, one of the X-programming icons. One of the key elements of XP, as I understood his talk, is cutting down what he calls excess 'inventory' which include things such as excess documentation, architecture, etc. By keeping these iventory elements to a minimum, you get quicker feedback, because you have quicker results, which feeds back into the process and allows you to respond quickly to changing requirments.
Well, it also happens that JUnit was developed al la XP by Kent and one other guy (didn't pay attention to who, go look it up!) For a while I was thinking that this XP thing might actually be something, but after skimming through the first 5 chapeters of 'Planning XP' coupled with the statements concerning the JUnit API, I'm starting to think that XP really is just one big hot air balloon.
In his defense though, he did say that refactoring often was a GOOD idea. It's just that he didn't say that you should wait for someone else to do it for you.
My 2 cents.
All your base are belong to us!
I'm sure you've also heard of a guy named Fowler... Martin Fowler is well known in the community for exactly two reasons.
The primary one is his book Refactoring. It describes his experience as a consultant refactoring medium sized software projects, and gives a lot of advice on methods for refactoring.
Apparently Fowler decided that refactoring is a good thing. Not just that, he decided that since refactoring is a good thing, and so should be the basis for programming, since most of the results of programming are bad. By that I mean just that most programs suck, not that they are evil or anything.
At that point he joined forces with Beck and formed his second reason for being well known, XP. As far as I can tell, the philosophy of XP is, "Software usually ends up sucking and in need of refactoring after it has been extended too far. Why wait for it to be extended too far? Just make it suck in the first place and refactor all the time!"
It does have a sort of perverse logic to it, but when I say that XP is bad, I don't just mean that it sucks. As well as sucking, XP really is evil.
Well, reading the source is good. But in XP the real "API documentation" are the tests for the system. If you can't understand the tests, you can't understand the program.
Full disclosure: I haven't screwed with the JUnit internals. But I'm working on a system that has over a million lines of code and JUnit 3.7 works just fine for it. Thousands of tests, too.
i'd like to check out artima suiterunner, but i don't think one should have to register to get access to source. at the very least, we should be able to browse the source online.
don't get me wrong - i'm happy the code is open. we use junit for our open-source product and anything that could help us better utilize junit is much appreciated.
smd4985
I have recently begin to use JUnit and ran into problems right away with the classloading mechanism. If any of you have wierd problems with inheritance relationships breaking, try modifying the excluded.properties file inside the JUnit jar. This worked for me.That said, it would have been nice to be able to modify this without having to unjar anything.
However, I wonder why they rewrote it from scratch. Wouldn't it have been simpler to redesign the problematic parts. And I also find it interesting that they have written their project to be compatible with existing tests. Does that means that JUnit's interface is not problematic, only the implementation? Seems to me like more of a case for JUnit 4.0 then a complete re-write from scratch.
The follow up articles should be interesting
more about me
Overall though it seems like mostly a documentation issue, not a design issue. Good documentation for the internals of JUnit is pretty non-existent from what I could find. I discovered a lot mostly from examining other JUnit extensions. With some good documentation on JUnit internals and documentation on the internal flow of operations I think I could have hugely cut down on the time I needed to figure out where to plug in. I would be willing to take a crack at this documentation, but I am *definitely* not the best person to do it :) However if anyone else is interested I would be willing to give it a go?
And if the methods are there because they have to be to implement an interface, but they return some "Not Yet Implemented" error or exception? How do you handle that with your compiler? I agree with you that they could've accomplished their goal in a much simpler fashion (one test for each API that only fails if NYI is thrown, regardless of any other exceptions or error codes). However, that doesn't mean they shouldn't have re-written the software. They identified issues with extending the existing software, and if they had problems then surely plenty of other people are having problems as well. Assuming that they weren't on a schedule so tight as to prohibit it, they should then do the "right" thing and fix what's broken not only for their current use, but also for their future use and the use of others.
Our QA people have developed their own framework for running tests in C#. It automatically discovers test cases via Reflection API, allows to group them into suites, run suites, generate reports, debug. It took 2 people 1.5 months to write it (while also dogfooding it to themselves and writing actual tests). No big fanfare, no buzzwords.
"Refactoring" - holy fuck, where do you get such words?
*
Here is an honest question, I know very little about XP, but how does it keep the rate of change consistent with all this "refactoring" going on?
As your code base increases from cycle to cycle and more features are added to the system do you not end up spending most of your time "refactoring" your old code so that the new feature fits into the system?