I'm planning on publishing a series of articles that delve into the differences between JUnit's and SuiteRunner's APIs. Since both are rather small APIs, I think most readers will be able to grasp the APIs enough to understand the design tradeoffs being discussed. One of the reasons I decided to go ahead and finish and release SuiteRunner was so I could use it to get discussion about design going in the Artima Forums.
In short, though, you are about right. The apples to apples comparison would be between junit.framework plus all the runner packages and org.suiterunner.
JUnit's TestCase, TestSuite, Test, and Assert ended up being collapsed into Suite. In the process we collapsed the dozens of assertThis assertThat methods into two verify methods.
JUnit's TestResult corresponds to SuiteRunner's Reporter, but whereas TestResult is a class, Reporter is an interface. Different implementations of Reporter can report results in different ways. One implementation of Reporter we have internally holds multiple other Reporters and forwards reports to all of them. That allows you to combine multiple reporters in a single test run. You can kind of get at the same thing in JUnit by registering TestListeners with a TestResult.
Anyway, I'll eventually go into details in a series of articles.
As I mentioned elsewhere in this topic, I ask for registration so that I know who the users are. Even though SuiteRunner is free and open source, I consider it a product. If you check the Artima Newsletter checkbox, you'll be notified of updates to SuiteRunner as well as new articles on Artima.com.
Also, for several reasons we decided to create a web-based discussion forum for SuiteRunner user support rather than using a mailing list. The web forum (I fear this will add weight to your conspiracy theory, but the forum is based on Jive.) requires that you have an account, so by getting an account when you download SuiteRunner, you are ready to post to the SuiteRunner users forum.
On top of that, shouldn't I get points for requiring such a minute amount of information to register? Many times I have been confronted with a long scrolling page full of edit boxes I'm required to fill in. At Artima.com, all I ask for is a name, which you can fake, a nickname, which you can make up, and an email address that I confirm. The name and nickname shows up on forum posts. If you watch a forum topic you get notified by email when someone posts a reply, which is why I confirm the email address even if you opt out of the newsletter.
Interesting that you found the site "flashy." I was going for simple. If you look carefully, you'll see the site is primarily text with a bare minimum of graphics.
Nevertheless, although I incorporated Artima Software a couple years ago, it is still just one person (me). So what you really need to ask yourself is not whether you trust a faceless corporation with your email address, but whether you trust me with it. I assure you I am very concerned about privacy.
A contract for a method in an interface doesn't say what the method does, it draws a fence around the full range of possible things that method could do. So for example, Collection.add method has the contract:
Ensures that this collection contains the specified element.
That leaves room for an implementation class, such as HashSet, to reject objects passed to add that already exist in the set. The Collection.add contract is abstract. It draws a fence around the full range of legal behavior for the add method. HashSet.add is a concrete implementation that is like a single point inside the Collection.add fence.
Imagine stumbling upon the code to HashSet.add without the benefit of the detailed contract-oriented API documentation of Collection.add. You want to implement an implementation of Collection.add in a different way, but all you know is the single point represented by the HashSet implementation. You have to guess where the fence is.
That was the real trouble I had trying to integrate with JUnit as an API. I spent time attempting to decipher the JUnit source code and understood fairly well what it was doing. I made educated guesses at the contracts, but given the non-intuitive (to me) nature of the JUnit API, I wasn't confident in my guesses. I could have integrated my tool and tested it with JUnit 3.6 or 3.7, but had little faith that JUnit 3.8 would not prove my guesses wrong and break my tool.
I interviewed Martin Fowler, author of Refactoring, last year. In this interview I asked him to define refactoring. He said:
Refactoring is making changes to a body of code in order to improve its internal structure, without changing its external behavior.
We didn't refactor JUnit's code, because we didn't start with JUnit's code and make changes. But we did do what I consider a "refactor" of the design. We started in effect with JUnit's API, and made changes. We started with knowledge of what we liked and didn't like about JUnit's API, and refactored that.
Where you can see the refactoring is not in the code, but in the JavaDoc. Compare JUnit's JavaDoc with SuiteRunner's JavaDoc. I'm guessing it was version 3.7 of JUnit that I had in front of me when I decided I would start over. It may have been 3.6.
It's Larry Rosen's Open Software License (OSL), not ours. I think it is a great license. I liked that they mention trademarks, for example. The main thing I didn't really like about it was the requirement that derivative works also be released under the OSL. I would have been OK with derivative works being proprietary. But I liked the OSL so much, compared to the many other existing open source licenses, that I went with the OSL.
The reason I have people register is to capture email addresses of users. Even though SuiteRunner is free and open source, I consider it a product, and I want to be able to contact users. If you don't want to register, just go to sourceforge and look up the suiterunner project. You can download it there anonymously. Or if you register but don't want to be emailed ever, just uncheck the two newsletter checkboxes.
Well, let me clarify. It would have been far easier to figure out JUnit's source code, guess at its API contracts, and integrate my signature test tool with JUnit -- without touching any JUnit code.
However, once I decided to "refactor" JUnit itself, I chose to not use any existing code. My main problem with JUnit was its API, not its implementation.
This is really a generic issue: when do you tweak existing code and when do you start over? By nature I am highly suspicious of urges to start over. In this case, I wanted to redesign the API first and foremost. JUnit's existing code didn't offer much assistance there.
On the other hand, JUnit's API design and very existence offered a huge amount of assistance. We could learn from what we deemed the mistakes in JUnit's design, and replicate what we felt were the good aspects of the design.
I also found that JUnit's documentation was poor, but it wasn't just that. It didn't seem to be designed as a user-friendly API, where the user is a programmer trying to use it as an API. As I wrote in the article, I really liked using JUnit as an application, once I got it going. It was when I went in to integrate my signature test tool with JUnit as an API that I really ran into trouble. The API documentation was very sparse and frustrating. I started looking at the code, which was confusing not just because of poor documentation, but of non-intuitive (to me, anyway) organization.
I was able to figure out what the code itself was doing, but then I was left guessing at what the contracts of the types and the methods were. When you decipher code, you understand one implementation of an interface, not the abstract contract of the interface. I got so mad at it that I opened IntelliJ and started writing my own testing toolkit. I thought it would be fairly easy, but I was wrong. I underestimated how much functionality JUnit actually has in it. And although I think we did come up with a much simpler public API, it took a lot of work to simplify it. So I ended up with more appreciation of all the work that went into JUnit.
I suspect that a lot of JUnit's bulkiness comes from the fact that JUnit started with Java 1.0 or 1.1 and evolved over time in wide-spread public use. Because it was so popular, any design blemishes couldn't be erased in later releases. Eventually enough such blemishes accumlate and it makes sense to start over.
Re:They rewrote the *runners*, not the *framework*
on
Why We Refactored JUnit
·
· Score: 5, Informative
Actually, we rewrote JUnit's basic functionality. You can use SuiteRunner standalone to create and run unit and conformance tests. JUnit is not required.
We made SuiteRunner a JUnit runner as an afterthought, when it dawned on us that no JUnit user would use SuiteRunner unless it added value to their existing JUnit test suites.
One of the things we did originally was make "test" an abstract notion. Reporters can indicate a test is starting, succeeded, failed, or aborted. When it came time to turn SuiteRunner into a JUnit runner, we were able to just report JUnit test results as another kind of test. We felt the ease with which we could turn SuiteRunner into a JUnit runner was a validation of our abstract test notion.
I think I was lucky that I didn't come across his book for a few years, particularly because of his skepticism of MI. I might have followed his advice and my career would have been the worse for it.
Funny, you had the opposite experience that I had. I discovered Scott's book quite by accident when I first started programming C++. It guided me away from multiple inheritance, which I ended up never using until I turned to Java five years later.
To avoid self-aggrandizement, I didn't independently invent the notion of pure abstractions. I had fiddled with Objective-C and it had a construct (whose name I forget) that allowed you to define a pure interface.
Your comment about Objective-C reminded me of something James Gosling once said in one of my interviews. I went to Artima.com and searched for it, but couldn't find it. To my surprise, the comment wasn't in the article anywhere. I went back to the text file that originally came back from the transcriber in 1999, and there was his comment. Somehow it never got published. So I just published it four years after he said it. Sorry.
Like you, Gosling found inspiration for Java's interface from the corresponding construct in Objective-C. Here's what he said in 1999:
Slashdot Mirror
JUnit's TestCase, TestSuite, Test, and Assert ended up being collapsed into Suite. In the process we collapsed the dozens of assertThis assertThat methods into two verify methods.
JUnit's TestResult corresponds to SuiteRunner's Reporter, but whereas TestResult is a class, Reporter is an interface. Different implementations of Reporter can report results in different ways. One implementation of Reporter we have internally holds multiple other Reporters and forwards reports to all of them. That allows you to combine multiple reporters in a single test run. You can kind of get at the same thing in JUnit by registering TestListeners with a TestResult.
Anyway, I'll eventually go into details in a series of articles.
As I mentioned elsewhere in this topic, I ask for registration so that I know who the users are. Even though SuiteRunner is free and open source, I consider it a product. If you check the Artima Newsletter checkbox, you'll be notified of updates to SuiteRunner as well as new articles on Artima.com.
Also, for several reasons we decided to create a web-based discussion forum for SuiteRunner user support rather than using a mailing list. The web forum (I fear this will add weight to your conspiracy theory, but the forum is based on Jive.) requires that you have an account, so by getting an account when you download SuiteRunner, you are ready to post to the SuiteRunner users forum.
On top of that, shouldn't I get points for requiring such a minute amount of information to register? Many times I have been confronted with a long scrolling page full of edit boxes I'm required to fill in. At Artima.com, all I ask for is a name, which you can fake, a nickname, which you can make up, and an email address that I confirm. The name and nickname shows up on forum posts. If you watch a forum topic you get notified by email when someone posts a reply, which is why I confirm the email address even if you opt out of the newsletter.
Interesting that you found the site "flashy." I was going for simple. If you look carefully, you'll see the site is primarily text with a bare minimum of graphics.
Nevertheless, although I incorporated Artima Software a couple years ago, it is still just one person (me). So what you really need to ask yourself is not whether you trust a faceless corporation with your email address, but whether you trust me with it. I assure you I am very concerned about privacy.
A contract for a method in an interface doesn't say what the method does, it draws a fence around the full range of possible things that method could do. So for example, Collection.add method has the contract:
That leaves room for an implementation class, such as HashSet, to reject objects passed to add that already exist in the set. The Collection.add contract is abstract. It draws a fence around the full range of legal behavior for the add method. HashSet.add is a concrete implementation that is like a single point inside the Collection.add fence.
Imagine stumbling upon the code to HashSet.add without the benefit of the detailed contract-oriented API documentation of Collection.add. You want to implement an implementation of Collection.add in a different way, but all you know is the single point represented by the HashSet implementation. You have to guess where the fence is.
That was the real trouble I had trying to integrate with JUnit as an API. I spent time attempting to decipher the JUnit source code and understood fairly well what it was doing. I made educated guesses at the contracts, but given the non-intuitive (to me) nature of the JUnit API, I wasn't confident in my guesses. I could have integrated my tool and tested it with JUnit 3.6 or 3.7, but had little faith that JUnit 3.8 would not prove my guesses wrong and break my tool.
I interviewed Martin Fowler, author of Refactoring, last year. In this interview I asked him to define refactoring. He said:
We didn't refactor JUnit's code, because we didn't start with JUnit's code and make changes. But we did do what I consider a "refactor" of the design. We started in effect with JUnit's API, and made changes. We started with knowledge of what we liked and didn't like about JUnit's API, and refactored that.
Where you can see the refactoring is not in the code, but in the JavaDoc. Compare JUnit's JavaDoc with SuiteRunner's JavaDoc. I'm guessing it was version 3.7 of JUnit that I had in front of me when I decided I would start over. It may have been 3.6.
It's Larry Rosen's Open Software License (OSL), not ours. I think it is a great license. I liked that they mention trademarks, for example. The main thing I didn't really like about it was the requirement that derivative works also be released under the OSL. I would have been OK with derivative works being proprietary. But I liked the OSL so much, compared to the many other existing open source licenses, that I went with the OSL.
The reason I have people register is to capture email addresses of users. Even though SuiteRunner is free and open source, I consider it a product, and I want to be able to contact users. If you don't want to register, just go to sourceforge and look up the suiterunner project. You can download it there anonymously. Or if you register but don't want to be emailed ever, just uncheck the two newsletter checkboxes.
However, once I decided to "refactor" JUnit itself, I chose to not use any existing code. My main problem with JUnit was its API, not its implementation.
This is really a generic issue: when do you tweak existing code and when do you start over? By nature I am highly suspicious of urges to start over. In this case, I wanted to redesign the API first and foremost. JUnit's existing code didn't offer much assistance there.
On the other hand, JUnit's API design and very existence offered a huge amount of assistance. We could learn from what we deemed the mistakes in JUnit's design, and replicate what we felt were the good aspects of the design.
I also found that JUnit's documentation was poor, but it wasn't just that. It didn't seem to be designed as a user-friendly API, where the user is a programmer trying to use it as an API. As I wrote in the article, I really liked using JUnit as an application, once I got it going. It was when I went in to integrate my signature test tool with JUnit as an API that I really ran into trouble. The API documentation was very sparse and frustrating. I started looking at the code, which was confusing not just because of poor documentation, but of non-intuitive (to me, anyway) organization.
I was able to figure out what the code itself was doing, but then I was left guessing at what the contracts of the types and the methods were. When you decipher code, you understand one implementation of an interface, not the abstract contract of the interface. I got so mad at it that I opened IntelliJ and started writing my own testing toolkit. I thought it would be fairly easy, but I was wrong. I underestimated how much functionality JUnit actually has in it. And although I think we did come up with a much simpler public API, it took a lot of work to simplify it. So I ended up with more appreciation of all the work that went into JUnit.
I suspect that a lot of JUnit's bulkiness comes from the fact that JUnit started with Java 1.0 or 1.1 and evolved over time in wide-spread public use. Because it was so popular, any design blemishes couldn't be erased in later releases. Eventually enough such blemishes accumlate and it makes sense to start over.
We made SuiteRunner a JUnit runner as an afterthought, when it dawned on us that no JUnit user would use SuiteRunner unless it added value to their existing JUnit test suites.
One of the things we did originally was make "test" an abstract notion. Reporters can indicate a test is starting, succeeded, failed, or aborted. When it came time to turn SuiteRunner into a JUnit runner, we were able to just report JUnit test results as another kind of test. We felt the ease with which we could turn SuiteRunner into a JUnit runner was a validation of our abstract test notion.
Funny, you had the opposite experience that I had. I discovered Scott's book quite by accident when I first started programming C++. It guided me away from multiple inheritance, which I ended up never using until I turned to Java five years later.
Your comment about Objective-C reminded me of something James Gosling once said in one of my interviews. I went to Artima.com and searched for it, but couldn't find it. To my surprise, the comment wasn't in the article anywhere. I went back to the text file that originally came back from the transcriber in 1999, and there was his comment. Somehow it never got published. So I just published it four years after he said it. Sorry.
Like you, Gosling found inspiration for Java's interface from the corresponding construct in Objective-C. Here's what he said in 1999:
http://www.artima.com/intv/gosling13.html