Why We Refactored JUnit

Bill Venners writes "In this article, three programmers tell the story of how their frustration with JUnit's API led to the creation of Artima SuiteRunner, a free, open source test toolkit and JUnit runner. These programmers simply wanted to create a small add-on tool to JUnit, but found JUnit's design non-intuitive and API documention poor. After spending time reading through JUnit's source code and attempting to guess at the API contracts, they gave up and rewrote it."

Comments

I wish...

[Anonymous Coward]

I could refactor my unit.

Re:I wish...

[Forgotten]

You could, if you'd just read your spam before deleting it.

hrm....

[xao gypsie]

why does this sound like something of the beginning of a fantasy story about programming: "in the beginning was Iluvitar, the master of the JU API........then melkor, wishing to expand it...."

xao

genericity in testing

[Anonymous Coward]

One of the biggest problems for a testing framework is fitting all the tests you need in it. Data types such as XML make it really, really difficult.

For example, you have a class that generates an XML document and you want to test it. Which level do you run the tests at? Best to test all levels, of course. How then do you check the accuracy of intermediate results? The answer often is that you can't; the intermediate results are meaningless until you have the complete (or at least almost complete) infrastructure to interpret the data. But once you've run all that, it is very difficult to pinpoint the source of the problem.

I'm glad that Artima SuiteRunner provides a way to handle all of that complexity and write unit tests that are complete but tractable. I just wish it worked with Perl, since I don't use Java anyway.

Re:genericity in testing

[Anonymous Coward]

Data types such as XML make it really, really difficult.

My big challenges are (1) existing code that wasn't designed for unit testing, (2) testing with databases, (3) testing that involves user interaction and (4) testing that involves multi-tasking. I like unit testing, but it is hard to apply to every task.

Re:genericity in testing

[Anonymous Coward]

My biggest problem is writing code that works in the first place so that you can even test it ;)

Ah, you do not understand the zen of XP yet. First you write the test. Then you write the code. Then you correct the code so the test passes. The sun is warm; the grass is green.

Re:genericity in testing

[thempstead]

Then you correct the code so the test passes.

nonono ... you correct the test so the code passes ...

Re:genericity in testing

[Mike_R]

Unit testing databases can be simplified by using DBUnit.
With DBUnit, you can describe your database in xml, and each time the tests are run, dbunit creates a new test database starting from the xml files. So you always test on a clean database. Check it out!

The truth about XP

[tundog]

I find this quite amusing. I recently attended a talk in Montreal given by Kent Beck, one of the X-programming icons. One of the key elements of XP, as I understood his talk, is cutting down what he calls excess 'inventory' which include things such as excess documentation, architecture, etc. By keeping these iventory elements to a minimum, you get quicker feedback, because you have quicker results, which feeds back into the process and allows you to respond quickly to changing requirments.

Well, it also happens that JUnit was developed al la XP by Kent and one other guy (didn't pay attention to who, go look it up!) For a while I was thinking that this XP thing might actually be something, but after skimming through the first 5 chapeters of 'Planning XP' coupled with the statements concerning the JUnit API, I'm starting to think that XP really is just one big hot air balloon.

In his defense though, he did say that refactoring often was a GOOD idea. It's just that he didn't say that you should wait for someone else to do it for you.

My 2 cents.

Re:The truth about XP

[Anonymous Coward]

I'm surprised, it almost seems like you didn't read much of the information on the website... like you... maybe... just read the slashdot headline and then formed your opinion... because here's what they guy who re-wrote JUnit said:

"at one point I just threw up my hands and said to myself, "Why is this so hard? It would be easier to rewrite JUnit than figure it out."

Just some friendly advice: if you ever find yourself saying it would be easier to rewrite something rather than figure it out, slap yourself. Chances are you are wrong. I certainly was. Creating SuiteRunner was a huge amount of work. Despite my frustrations with JUnit, it would have been orders of magnitude easier to decipher JUnit's source code than to create a brand new testing toolkit."

Re:The truth about XP

[dubl-u]

but after skimming through the first 5 chapeters of 'Planning XP'

If you're a programmer, that's the wrong book to start with. Read Extreme Programming Installed, by Jeffries, et al.

I'm starting to think that XP really is just one big hot air balloon.

In my experience, it's not. Since adopting it a couple years back, my designs are better, my bug counts are much lower, and I am much happier. Feel free to drop me a line if you have questions.

Re:The truth about XP

[babbage]

On the other hand, there's Fred Brooks' advice from The Mythical Man-Month: "Plan to throw the first one away. You will anyhow."

Sometimes, it really is easier to treat the first one as a prototype of what you'd really like, and then write that second one correctly, from scratch. Witness Mac Classic -> Mac OSX, Win9x -> WinNT, Perl5 -> Perl6, etc.

A lot of these prominent "next-generation" systems may share ideas & even code from their predecessors, but the essential point with each is that they all represent deliberate jump-cuts from the past. Sometimes it really is easier and better to rewrite something, whether or not you fully grok the original.

The real trick is to design systems well enough that, when someone comes along with better ideas, the framework you provide is robust enough to adapt. Mac Classic, Win9x/DOS, and Perl5 are all too inflexible for future use, though at this point each of them is still useful to certain groups. Their successors, however, are all designed with future expandability in mind, so that the "second system curse" can hopefully be avoided. History will tell if it ended up working in any of these cases...

Re:The truth about XP

[tupps]

I think comparing operating systems as a way of throwing out your first born is not correct. Eg MacOS Classic was a direct descendent of the original MacOS that launched in 1984. I am sure that there is very little or no code from the original. Win9X was built on top of DOS which I believe has a lineage older than MacOS classic. Interesting note that both MacOSX and WinNT both came from an external base (NeXT and OS2 sort of). If you want a better example IE 3.0 --> IE 4.0, Microsoft is a classic for working out the problems in the first version or two and then releasing a quality product.

Re:The truth about XP

[WaKall]

If YOU write the first one, throw it away, and write the second one, then yes, you may be on to something. However, if someone else writes the first one, and you can't understand it/use it, it doesn't that the first solution wasn't good. Maybe the problem lies somewhere else?

You don't learn as much from reading someone else's code as they did from writing it. Don't try mis-quote Brooks to justify not understanding an existing product. Brooks was referring to large-scale team oriented development (like the System 360, or IBM 650 probjects). He wasn't claiming that you should throw out someone else's code because it was the first solution to a problem.

Intuitive

[nettarzan]

Is intuitive design necessarily good in programming.?
Recursion is intuitive but not necessarily efficient in terms of performance.
At what point do we decide which is better intuitive design or efficient design.

Re:Intuitive

[dangermouse]

Recursion is not a design principle, it's an implementation decision-- like choosing between a switch block and cascading if/else blocks.

The author of this article is talking about the JUnit API. An intuitive API is always a good thing.

Re: Intuitive

[Black Parrot]

> Recursion is intuitive but...

You've obviously never tried to explain recursion to a group of average co-workers.

Re:Intuitive

[dubl-u]

Optimization should never be done until after you've run real-world use cases on code with a profiler. Until then, one should write code that clearly communicates its intent.

"Premature optimization is the root of all evil."

says Knuth, and Martin Fowler says

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

Remember: Computers double in speed every 18 months, but programmers stay just the same. If we want our code to live a long and happy life, then clarity should almost always win over speed.

Origins of XP

[Anonymous Coward]

I'm sure you've also heard of a guy named Fowler... Martin Fowler is well known in the community for exactly two reasons.

The primary one is his book Refactoring. It describes his experience as a consultant refactoring medium sized software projects, and gives a lot of advice on methods for refactoring.

Apparently Fowler decided that refactoring is a good thing. Not just that, he decided that since refactoring is a good thing, and so should be the basis for programming, since most of the results of programming are bad. By that I mean just that most programs suck, not that they are evil or anything.

At that point he joined forces with Beck and formed his second reason for being well known, XP. As far as I can tell, the philosophy of XP is, "Software usually ends up sucking and in need of refactoring after it has been extended too far. Why wait for it to be extended too far? Just make it suck in the first place and refactor all the time!"

It does have a sort of perverse logic to it, but when I say that XP is bad, I don't just mean that it sucks. As well as sucking, XP really is evil.

Re:Origins of XP

[Anonymous Coward]

XP is about writing code that can easily be successfully refactored. Total-design-up-front bigots think the code "sucks" because the design wasn't much more general than actually needed at the time, not realizing that making a design at the start of the project that meets the requirements as of the end of the project is almost always impossible, and all the effort in doing so is going to be wasted.

Re:Origins of XP

[Anonymous Coward]

At that point [Fowler] joined forces with Beck and formed his second reason for being well known, XP.

Martin Fowler did not invent XP. It originated in the work of Ward Cunningham and Kent Beck, then was refined by Kent Beck with help from Ron Jeffries and other members of the original XP team. Martin Fowler is an active part of that community, so he co-authored one of the XP books.

Re:Origins of XP

[clyde7]

I agree. XP is like music in the ears of managers who handle projects and developers which already suck (BTW, Kent Beck is a consultant). XP is not meant to be applied by good developers.

In order to write good (stable, maintainable, extendible, generic, efficient, ...) software efficiently, you want to minimize the overhead (tests, refactoring, redesign, debugging). This can only be achieved by good upfront design skills and applying them as early as possible.

If the code is well written, there's few need
for writing tests (except for sophisticated algorithms), almost no need for refactoring (except for really simple ones like moving/renaming), redesign or debugging.

Unfortunately good OO design is a hard to learn skill and I haven't seen a good book on this yet.

Re:Origins of XP

[Ed Avis]

If the code is well written, there's few need
for writing tests (except for sophisticated
algorithms)

Well, duh. If it is well written it won't have bugs, by definition. But how do you find out whether the code is well written, except by thoroughly testing it?

it's all about compatibility

[ledbetter]

Since until now JUnit really was the only game in town for java test-writing, I'm so impressed to see these guys put out something that's still compatible with it!! Even if it was frustration with JUnit that gave them the inspiration. All of us have a bunch of JUnit tests for our code (ok, well SOME of us do), and it's nice to have the option to try out another framework without having to refactor our tests.

Really, in the world of open source, and free software way too little attention is paid to compatibility. Why not be compatible with your "competition"? It's not like you're competing for customers or market share, or any of that crap. We're all on the same team!

refactor == rewrite?

[_|()|\|]

I'm the first to admit that I'm not buzz-word compliant, but I was surprised to read the following:

[Artima SuiteRunner] is a design fork not a code fork, because we didn't reuse any JUnit code. We rethought and reworked JUnit's ideas, and wrote Artima SuiteRunner's code from scratch.

I thought refactoring was massaging, even rewriting parts of existing code. So, is Linux a "refactored" Unix?

Re:refactor == rewrite?

[bvenners]

I interviewed Martin Fowler, author of Refactoring, last year. In this interview I asked him to define refactoring. He said:

Refactoring is making changes to a body of code in order to improve its internal structure, without changing its external behavior.

We didn't refactor JUnit's code, because we didn't start with JUnit's code and make changes. But we did do what I consider a "refactor" of the design. We started in effect with JUnit's API, and made changes. We started with knowledge of what we liked and didn't like about JUnit's API, and refactored that.

Where you can see the refactoring is not in the code, but in the JavaDoc. Compare JUnit's JavaDoc with SuiteRunner's JavaDoc. I'm guessing it was version 3.7 of JUnit that I had in front of me when I decided I would start over. It may have been 3.6.

JUnit does have problems, but ....

[shamir_k]

I have recently begin to use JUnit and ran into problems right away with the classloading mechanism. If any of you have wierd problems with inheritance relationships breaking, try modifying the excluded.properties file inside the JUnit jar. This worked for me.That said, it would have been nice to be able to modify this without having to unjar anything.
However, I wonder why they rewrote it from scratch. Wouldn't it have been simpler to redesign the problematic parts. And I also find it interesting that they have written their project to be compatible with existing tests. Does that means that JUnit's interface is not problematic, only the implementation? Seems to me like more of a case for JUnit 4.0 then a complete re-write from scratch.
The follow up articles should be interesting

Re:JUnit does have problems, but ....

[bvenners]

Well, let me clarify. It would have been far easier to figure out JUnit's source code, guess at its API contracts, and integrate my signature test tool with JUnit -- without touching any JUnit code.

However, once I decided to "refactor" JUnit itself, I chose to not use any existing code. My main problem with JUnit was its API, not its implementation.

This is really a generic issue: when do you tweak existing code and when do you start over? By nature I am highly suspicious of urges to start over. In this case, I wanted to redesign the API first and foremost. JUnit's existing code didn't offer much assistance there.

On the other hand, JUnit's API design and very existence offered a huge amount of assistance. We could learn from what we deemed the mistakes in JUnit's design, and replicate what we felt were the good aspects of the design.

Paradigm shift?

[HisMother]

This is not a troll, but I'm struggling to try to understand why these guys had to work so hard to accomplish what they needed. By their own account, they were driven towards this because they wanted to write API conformance tests -- tests that pass if an API is implemented, regardless of whether it works or not. They wanted, I guess, to see a test pass for each API from the specification that was actually implemented, so they could compute a percentage.

While I can understand what they wanted to do, I guess I don't see why they didn't do this the much easier and (to me, anyway) more obvious way: you write code that uses the API, and if it compiles, then the API is implemented. You write one tiny test class for each API. You have each JUnit test run a compiler over one single class, this proofing each API. This isn't exactly rocket science: you have a directory full of probe-class source code, a single method that tries the compile, and then any number of TestCases that each try to compile one class and pass if they succeed. This is hard?

If you use the unpublished APIs to "javac," you could do this at lightning speed. Alternatively, you could just use a Java parser like the one that comes with ANTLR (since you don't actually need to generate any code) and do it even faster.

Re:Paradigm shift?

[Osty]

And if the methods are there because they have to be to implement an interface, but they return some "Not Yet Implemented" error or exception? How do you handle that with your compiler? I agree with you that they could've accomplished their goal in a much simpler fashion (one test for each API that only fails if NYI is thrown, regardless of any other exceptions or error codes). However, that doesn't mean they shouldn't have re-written the software. They identified issues with extending the existing software, and if they had problems then surely plenty of other people are having problems as well. Assuming that they weren't on a schedule so tight as to prohibit it, they should then do the "right" thing and fix what's broken not only for their current use, but also for their future use and the use of others.

Re:Paradigm shift?

[Mithrandir]

I am a spec writer, I don't play one on TV. I authored the EAI for VRML97 and the various programming APIs for X3D. These are ISO specs, so they come with a lot of weight behind them from the conformance perspective.

As someone who is involved in specification writing of APIs I can tell you that a "compile test" is wholly insufficient for checking class/method signatures. (for this reply, I'm using method as being interchangable for any message passing system - C functions, Java methods, RPC calls whatever)

The first and major problem is not that the methods exist - everyone can cut and paste the spec document - but ensuring that nothing else exists. The bane of every spec writer is the company/individual that decides that they don't like the spec and then proceeds to extend it with their own methods within the same class - often overloading existing method calls. It is these extensions that a spec writer dreads because the (often) clueless end user makes use of them and then wonders WTF their code won't run on another "conformant" implementations of that spec.

Checking signatures is there for one thing and one thing only - making sure the implementors don't embrace and extend the specification in a way that is not permitted by the spec. Creating derived interfaces with new method signatures is fine, adding new method signatures to the specification-defined collection is not. A good conformance test suite will look at everything about the signatures to make sure everything is there, and nothing that should not be is not there.

Re:Paradigm shift?

[The Musician]

To help answer the grandparent's question: compile != matches API.

Can you write a test-compile program that checks whether the proper public/private/proctected access modifier keywords are set up right on each method? That the method was specified for 'short foo' but the implementor used 'int foo'? Or like Mithrandir said, that no additional methods have been added (such as an overloaded version)?

Basically, compile-compatability is not the same as using the precise APIs in the spec.

This is what I like about Open Source Software

[Anonymous Coward]

This is one reason I like OSS--when people are not satisfied with currently available solutions, they can take the code and improve it. I have not looked at the new code (and for that matter, am unfamiliar with JUnit), but I think allowing people the freedom to take code and improve it one of the best features of OSS.

They rewrote the *runners*, not the *framework*

[lurp]

It would be more accurate to say that they rewrote the junit test runners and not the framework itself. JUnit's framework has an extremely simple and clean design, and really doesn't need any changing. The design of the runners, on the other hand really sucks, as Kent Beck has admitted on a couple of occasions.

That said, I don't know if artima has really contributed anything new here. Your IDE likely has a JUnit test runner built into it already (IntelliJ, JBuilder, and NetBeans all do). Ant also already has decent junit test and report targets, which basically include all of the capabilities artima has implemented. Another stand alone test runner is probably not all that useful for day to day development.

Re:They rewrote the *runners*, not the *framework*

[bvenners]

Actually, we rewrote JUnit's basic functionality. You can use SuiteRunner standalone to create and run unit and conformance tests. JUnit is not required.

We made SuiteRunner a JUnit runner as an afterthought, when it dawned on us that no JUnit user would use SuiteRunner unless it added value to their existing JUnit test suites.

One of the things we did originally was make "test" an abstract notion. Reporters can indicate a test is starting, succeeded, failed, or aborted. When it came time to turn SuiteRunner into a JUnit runner, we were able to just report JUnit test results as another kind of test. We felt the ease with which we could turn SuiteRunner into a JUnit runner was a validation of our abstract test notion.

Design first, or refactor? Re:Origins of XP

[2nesser]

making a design at the start of the project that meets the requirements as of the end of the project is almost always impossible

I believe that this quote is true for commercial software. The word processors, email clients and web browsers in the world should not be designed at the beginning of a project. The cost of doing that far outweighs the benifits. But...

When you are designing life critical applications (fly-by-wire, ABS, pacemaker, cancer radiation machine...etc) it is not acceptable to be redesigning while coding.

If you cannot model the way a system must react at all times, under any input, then what makes you think that your software should be responsible for someones life.

More and more software is being designed for life critical applications. If XP is being used to develop them, I'm worried. You cannot stumble onto 100% correct software. In a large system it takes a lot of money - an exponential curve, the last 3 bugs will likely take more money than the first 100 - time and a good design.

Re:Design first, or refactor? Re:Origins of XP

[chromatic]

In a large system it takes a lot of money - an exponential curve, the last 3 bugs will likely take more money than the first 100 - time and a good design.

That assumes that the rate of the cost of change rises over time. XP rejects that assumption, and the XP practices are designed to keep the rate of the cost of change consistent.

Re:I tried too..

[bvenners]

I also found that JUnit's documentation was poor, but it wasn't just that. It didn't seem to be designed as a user-friendly API, where the user is a programmer trying to use it as an API. As I wrote in the article, I really liked using JUnit as an application, once I got it going. It was when I went in to integrate my signature test tool with JUnit as an API that I really ran into trouble. The API documentation was very sparse and frustrating. I started looking at the code, which was confusing not just because of poor documentation, but of non-intuitive (to me, anyway) organization.

I was able to figure out what the code itself was doing, but then I was left guessing at what the contracts of the types and the methods were. When you decipher code, you understand one implementation of an interface, not the abstract contract of the interface. I got so mad at it that I opened IntelliJ and started writing my own testing toolkit. I thought it would be fairly easy, but I was wrong. I underestimated how much functionality JUnit actually has in it. And although I think we did come up with a much simpler public API, it took a lot of work to simplify it. So I ended up with more appreciation of all the work that went into JUnit.

I suspect that a lot of JUnit's bulkiness comes from the fact that JUnit started with Java 1.0 or 1.1 and evolved over time in wide-spread public use. Because it was so popular, any design blemishes couldn't be erased in later releases. Eventually enough such blemishes accumlate and it makes sense to start over.

What's the big deal?

[melted]

Our QA people have developed their own framework for running tests in C#. It automatically discovers test cases via Reflection API, allows to group them into suites, run suites, generate reports, debug. It took 2 people 1.5 months to write it (while also dogfooding it to themselves and writing actual tests). No big fanfare, no buzzwords.

"Refactoring" - holy fuck, where do you get such words?
*

Re:I tried too..

[Pastis]

First, to answer somebody's else comment, the other guy who coded JUnit that nobody can't remember was Erich Gamma. If that name doesn't tell you anything, go back and read some books. It's impressive how the most important people 10 years ago can be forgotten so quickly. Makes you understand why Software is often called Software Craft.

Open source software don't come up often with design documents. This one does. To those who say JUnit is hard to understand, go and read the Cook's tour. Then you'll understand everything. Things have changed a little bit but not that much. Of course if you are not familiar with Design Patterns, I recommend to read Design Patterns first. Yes that's by Erich Gamma and co also called the Gang of 4

I see allready people coming and say design pattern is crap, and that's a proof. But before coming in and complaining, remember the lessons learned by the people who have worked longer than us in this industry. KISS(Keep It Simple...) and don't reinvent the wheel: I am pretty sure that the elegant design used in JUnit could have been refactored quickly to do what those guys wanted without having to rewrite everything from scratch.

The author of this new test suite says it all:

Creating SuiteRunner was a huge amount of work. Despite my frustrations with JUnit, it would have been orders of magnitude easier to decipher JUnit's source code than to create a brand new testing toolkit.

Solid Foundation For Software Refactoring

[Bart Du Bois]

To get to know refactoring, I can refer to http://www.refactoring.com (by Martin Fowler).

For those interested in the value of refactoring (whether it's merely a buzzword), I can refer to a research project of ours, at http://win-www.uia.ac.be/u/lore/refactoringProject

By the way: since the article does not go into behaviour-preserving restructuring of JUnit, they shouldn't mention 'refactoring' in the title.

Re:if it ain't broke....

[bvenners]

As I mentioned elsewhere in this topic, I ask for registration so that I know who the users are. Even though SuiteRunner is free and open source, I consider it a product. If you check the Artima Newsletter checkbox, you'll be notified of updates to SuiteRunner as well as new articles on Artima.com.

Also, for several reasons we decided to create a web-based discussion forum for SuiteRunner user support rather than using a mailing list. The web forum (I fear this will add weight to your conspiracy theory, but the forum is based on Jive.) requires that you have an account, so by getting an account when you download SuiteRunner, you are ready to post to the SuiteRunner users forum.

On top of that, shouldn't I get points for requiring such a minute amount of information to register? Many times I have been confronted with a long scrolling page full of edit boxes I'm required to fill in. At Artima.com, all I ask for is a name, which you can fake, a nickname, which you can make up, and an email address that I confirm. The name and nickname shows up on forum posts. If you watch a forum topic you get notified by email when someone posts a reply, which is why I confirm the email address even if you opt out of the newsletter.

Interesting that you found the site "flashy." I was going for simple. If you look carefully, you'll see the site is primarily text with a bare minimum of graphics.

Nevertheless, although I incorporated Artima Software a couple years ago, it is still just one person (me). So what you really need to ask yourself is not whether you trust a faceless corporation with your email address, but whether you trust me with it. I assure you I am very concerned about privacy.

Yet another JUnit like tool.

[M.M.M.]

I have another one,
it is geared towards tests with multithreaded environment.

here is the project page
http://www.michaelmoser.org/jmtunit/

The library sets up a thread pool (n threads) and runs a number of identical test sequences; each test sequence is a virtual user (m users). All threads are waiting on one input queue for test requests. The threading modell is the same as in most of the popular application servers

The programming interface is very similar to JUnit - there is a JMTUnit.TestCase with some additions, there is JMTUnit.TestResult.
The tool does print out performance statistics