Why We Refactored JUnit

Bill Venners writes "In this article, three programmers tell the story of how their frustration with JUnit's API led to the creation of Artima SuiteRunner, a free, open source test toolkit and JUnit runner. These programmers simply wanted to create a small add-on tool to JUnit, but found JUnit's design non-intuitive and API documention poor. After spending time reading through JUnit's source code and attempting to guess at the API contracts, they gave up and rewrote it."

Comments

it's all about compatibility

[ledbetter]

Since until now JUnit really was the only game in town for java test-writing, I'm so impressed to see these guys put out something that's still compatible with it!! Even if it was frustration with JUnit that gave them the inspiration. All of us have a bunch of JUnit tests for our code (ok, well SOME of us do), and it's nice to have the option to try out another framework without having to refactor our tests.

Really, in the world of open source, and free software way too little attention is paid to compatibility. Why not be compatible with your "competition"? It's not like you're competing for customers or market share, or any of that crap. We're all on the same team!

They rewrote the *runners*, not the *framework*

[lurp]

It would be more accurate to say that they rewrote the junit test runners and not the framework itself. JUnit's framework has an extremely simple and clean design, and really doesn't need any changing. The design of the runners, on the other hand really sucks, as Kent Beck has admitted on a couple of occasions.

That said, I don't know if artima has really contributed anything new here. Your IDE likely has a JUnit test runner built into it already (IntelliJ, JBuilder, and NetBeans all do). Ant also already has decent junit test and report targets, which basically include all of the capabilities artima has implemented. Another stand alone test runner is probably not all that useful for day to day development.

Re:They rewrote the *runners*, not the *framework*

[bvenners]

Actually, we rewrote JUnit's basic functionality. You can use SuiteRunner standalone to create and run unit and conformance tests. JUnit is not required.

We made SuiteRunner a JUnit runner as an afterthought, when it dawned on us that no JUnit user would use SuiteRunner unless it added value to their existing JUnit test suites.

One of the things we did originally was make "test" an abstract notion. Reporters can indicate a test is starting, succeeded, failed, or aborted. When it came time to turn SuiteRunner into a JUnit runner, we were able to just report JUnit test results as another kind of test. We felt the ease with which we could turn SuiteRunner into a JUnit runner was a validation of our abstract test notion.

Re:Origins of XP

[Anonymous Coward]

At that point [Fowler] joined forces with Beck and formed his second reason for being well known, XP.

Martin Fowler did not invent XP. It originated in the work of Ward Cunningham and Kent Beck, then was refined by Kent Beck with help from Ron Jeffries and other members of the original XP team. Martin Fowler is an active part of that community, so he co-authored one of the XP books.

Re:Paradigm shift?

[Mithrandir]

I am a spec writer, I don't play one on TV. I authored the EAI for VRML97 and the various programming APIs for X3D. These are ISO specs, so they come with a lot of weight behind them from the conformance perspective.

As someone who is involved in specification writing of APIs I can tell you that a "compile test" is wholly insufficient for checking class/method signatures. (for this reply, I'm using method as being interchangable for any message passing system - C functions, Java methods, RPC calls whatever)

The first and major problem is not that the methods exist - everyone can cut and paste the spec document - but ensuring that nothing else exists. The bane of every spec writer is the company/individual that decides that they don't like the spec and then proceeds to extend it with their own methods within the same class - often overloading existing method calls. It is these extensions that a spec writer dreads because the (often) clueless end user makes use of them and then wonders WTF their code won't run on another "conformant" implementations of that spec.

Checking signatures is there for one thing and one thing only - making sure the implementors don't embrace and extend the specification in a way that is not permitted by the spec. Creating derived interfaces with new method signatures is fine, adding new method signatures to the specification-defined collection is not. A good conformance test suite will look at everything about the signatures to make sure everything is there, and nothing that should not be is not there.

Re:I tried too..

[bvenners]

I also found that JUnit's documentation was poor, but it wasn't just that. It didn't seem to be designed as a user-friendly API, where the user is a programmer trying to use it as an API. As I wrote in the article, I really liked using JUnit as an application, once I got it going. It was when I went in to integrate my signature test tool with JUnit as an API that I really ran into trouble. The API documentation was very sparse and frustrating. I started looking at the code, which was confusing not just because of poor documentation, but of non-intuitive (to me, anyway) organization.

I was able to figure out what the code itself was doing, but then I was left guessing at what the contracts of the types and the methods were. When you decipher code, you understand one implementation of an interface, not the abstract contract of the interface. I got so mad at it that I opened IntelliJ and started writing my own testing toolkit. I thought it would be fairly easy, but I was wrong. I underestimated how much functionality JUnit actually has in it. And although I think we did come up with a much simpler public API, it took a lot of work to simplify it. So I ended up with more appreciation of all the work that went into JUnit.

I suspect that a lot of JUnit's bulkiness comes from the fact that JUnit started with Java 1.0 or 1.1 and evolved over time in wide-spread public use. Because it was so popular, any design blemishes couldn't be erased in later releases. Eventually enough such blemishes accumlate and it makes sense to start over.

Re:Design first, or refactor? Re:Origins of XP

[chromatic]

In a large system it takes a lot of money - an exponential curve, the last 3 bugs will likely take more money than the first 100 - time and a good design.

That assumes that the rate of the cost of change rises over time. XP rejects that assumption, and the XP practices are designed to keep the rate of the cost of change consistent.