Slashdot Mirror
Why We Refactored JUnit
Bill Venners writes "In this article, three programmers tell the story of how their frustration with JUnit's API led to the creation of Artima SuiteRunner, a free, open source test toolkit and JUnit runner. These programmers simply wanted to create a small add-on tool to JUnit, but found JUnit's design non-intuitive and API documention poor. After spending time reading through JUnit's source code and attempting to guess at the API contracts, they gave up and rewrote it."
I could refactor my unit.
why does this sound like something of the beginning of a fantasy story about programming: "in the beginning was Iluvitar, the master of the JU API........then melkor, wishing to expand it...."
xao
xao
http://TheHillforum.hopto.org
"After spending time reading through JUnit's source code and attempting to guess at the API contracts, they gave up and rewrote it."
Now if someone would just do that for Windows we'd be in business...
One of the biggest problems for a testing framework is fitting all the tests you need in it. Data types such as XML make it really, really difficult.
For example, you have a class that generates an XML document and you want to test it. Which level do you run the tests at? Best to test all levels, of course. How then do you check the accuracy of intermediate results? The answer often is that you can't; the intermediate results are meaningless until you have the complete (or at least almost complete) infrastructure to interpret the data. But once you've run all that, it is very difficult to pinpoint the source of the problem.
I'm glad that Artima SuiteRunner provides a way to handle all of that complexity and write unit tests that are complete but tractable. I just wish it worked with Perl, since I don't use Java anyway.
I find this quite amusing. I recently attended a talk in Montreal given by Kent Beck, one of the X-programming icons. One of the key elements of XP, as I understood his talk, is cutting down what he calls excess 'inventory' which include things such as excess documentation, architecture, etc. By keeping these iventory elements to a minimum, you get quicker feedback, because you have quicker results, which feeds back into the process and allows you to respond quickly to changing requirments.
Well, it also happens that JUnit was developed al la XP by Kent and one other guy (didn't pay attention to who, go look it up!) For a while I was thinking that this XP thing might actually be something, but after skimming through the first 5 chapeters of 'Planning XP' coupled with the statements concerning the JUnit API, I'm starting to think that XP really is just one big hot air balloon.
In his defense though, he did say that refactoring often was a GOOD idea. It's just that he didn't say that you should wait for someone else to do it for you.
My 2 cents.
All your base are belong to us!
They are ... see ReactOS
and it works pretty durn well too nowadays...
Is intuitive design necessarily good in programming.?
Recursion is intuitive but not necessarily efficient in terms of performance.
At what point do we decide which is better intuitive design or efficient design.
I'm sure you've also heard of a guy named Fowler... Martin Fowler is well known in the community for exactly two reasons.
The primary one is his book Refactoring. It describes his experience as a consultant refactoring medium sized software projects, and gives a lot of advice on methods for refactoring.
Apparently Fowler decided that refactoring is a good thing. Not just that, he decided that since refactoring is a good thing, and so should be the basis for programming, since most of the results of programming are bad. By that I mean just that most programs suck, not that they are evil or anything.
At that point he joined forces with Beck and formed his second reason for being well known, XP. As far as I can tell, the philosophy of XP is, "Software usually ends up sucking and in need of refactoring after it has been extended too far. Why wait for it to be extended too far? Just make it suck in the first place and refactor all the time!"
It does have a sort of perverse logic to it, but when I say that XP is bad, I don't just mean that it sucks. As well as sucking, XP really is evil.
Since until now JUnit really was the only game in town for java test-writing, I'm so impressed to see these guys put out something that's still compatible with it!! Even if it was frustration with JUnit that gave them the inspiration. All of us have a bunch of JUnit tests for our code (ok, well SOME of us do), and it's nice to have the option to try out another framework without having to refactor our tests.
Really, in the world of open source, and free software way too little attention is paid to compatibility. Why not be compatible with your "competition"? It's not like you're competing for customers or market share, or any of that crap. We're all on the same team!
Well, reading the source is good. But in XP the real "API documentation" are the tests for the system. If you can't understand the tests, you can't understand the program.
Full disclosure: I haven't screwed with the JUnit internals. But I'm working on a system that has over a million lines of code and JUnit 3.7 works just fine for it. Thousands of tests, too.
Explaining one's actions in a detailed and public manner is self-incriminating.
I wonder what really went on behind the scenes here.
Reply or e-mail; don't vaguely moderate. Ex-O'Reilly/MIT employee, now a full-time Google employee.
Huh?
*scratches head*
JUnit is unit testing for Java, right? How does the JVM dumping core become the application's fault? How does network IO have ANYTHING to do with unit testing (unless you're testing a network app that is)?
my parents say I'm not old enough to look at core dumps rated NP-17.
Get with the XP lingo!
i'd like to check out artima suiterunner, but i don't think one should have to register to get access to source. at the very least, we should be able to browse the source online.
don't get me wrong - i'm happy the code is open. we use junit for our open-source product and anything that could help us better utilize junit is much appreciated.
smd4985
I have recently begin to use JUnit and ran into problems right away with the classloading mechanism. If any of you have wierd problems with inheritance relationships breaking, try modifying the excluded.properties file inside the JUnit jar. This worked for me.That said, it would have been nice to be able to modify this without having to unjar anything.
However, I wonder why they rewrote it from scratch. Wouldn't it have been simpler to redesign the problematic parts. And I also find it interesting that they have written their project to be compatible with existing tests. Does that means that JUnit's interface is not problematic, only the implementation? Seems to me like more of a case for JUnit 4.0 then a complete re-write from scratch.
The follow up articles should be interesting
more about me
While I can understand what they wanted to do, I guess I don't see why they didn't do this the much easier and (to me, anyway) more obvious way: you write code that uses the API, and if it compiles, then the API is implemented. You write one tiny test class for each API. You have each JUnit test run a compiler over one single class, this proofing each API. This isn't exactly rocket science: you have a directory full of probe-class source code, a single method that tries the compile, and then any number of TestCases that each try to compile one class and pass if they succeed. This is hard?
If you use the unpublished APIs to "javac," you could do this at lightning speed. Alternatively, you could just use a Java parser like the one that comes with ANTLR (since you don't actually need to generate any code) and do it even faster.
Cantankerous old coot since 1957.
This is one reason I like OSS--when people are not satisfied with currently available solutions, they can take the code and improve it. I have not looked at the new code (and for that matter, am unfamiliar with JUnit), but I think allowing people the freedom to take code and improve it one of the best features of OSS.
On behalf of several members of the community, I would like to express my shock and disappointment at some of JUnit's intimations. Let's get down to business: It's easy enough to hate JUnit any day of the week on general principles. But now I'll tell you about some very specific things that JUnit is up to, things that ought to make a real JUnit-hater out of you. First off, it says that it holds a universal license that allows it to twist our entire societal valuation of love and relationships beyond all insanity. Yet it also wants to inspire a recrudescence of obnoxious fatuity. Am I the only one who sees the irony there? I ask, because I am aware that many people may object to the severity of my language. But is there no cause for severity? Naturally, I believe that there is, because there is a cost, a cost too high to calculate, for messing with the lives and livelihoods of thousands of people. I mean, think about it. Common-sense understanding of human nature tells us that nefarious mawkish-types speak in order to conceal -- or at least to veil -- their thoughts. Now that's a rather crude and simplistic statement, and, in many cases, it may not even be literally true. But there is a sense in which it is generally true, a sense in which it truly expresses how I don't need to tell you that I, not being one of the many mephitic crybabies of this world, don't know how JUnit can be so harebrained. That should be self-evident. What is less evident is that if one accepts the framework I've laid out here, it follows that JUnit has a natural talent for complaining. It can find any aspect of life and whine about it for hours upon hours.
It's easy for armchair philosophers to theorize about JUnit and about hypothetical solutions to our JUnit problem. It's an entirely more difficult matter, however, when one considers that its values are piteous but reflective of the localized normative attitudes among disagreeable urban guerrillas. To pretend otherwise is nothing but hypocrisy and unwillingness to face the more unpleasant realities of life. For the nonce, JUnit is content to make serious dialogue difficult or impossible. But sometime soon, it will set the wolf to mind the sheep. Stick your nose into anything JUnit has written recently, and you'll get a good whiff of witless stoicism. JUnit says that it is a bearer and agent of the Creator's purpose. The inference is that the best way to make a point is with foaming-at-the-mouth rhetoric and letters filled primarily with exclamation points. I'm happy to report that I can't follow that logic. Anyway, that's it for this letter. Let JUnit read it and weep.
--
kvetch, kvetch, kvetch
Overall though it seems like mostly a documentation issue, not a design issue. Good documentation for the internals of JUnit is pretty non-existent from what I could find. I discovered a lot mostly from examining other JUnit extensions. With some good documentation on JUnit internals and documentation on the internal flow of operations I think I could have hugely cut down on the time I needed to figure out where to plug in. I would be willing to take a crack at this documentation, but I am *definitely* not the best person to do it :) However if anyone else is interested I would be willing to give it a go?
That said, I don't know if artima has really contributed anything new here. Your IDE likely has a JUnit test runner built into it already (IntelliJ, JBuilder, and NetBeans all do). Ant also already has decent junit test and report targets, which basically include all of the capabilities artima has implemented. Another stand alone test runner is probably not all that useful for day to day development.
Funny how the insightful comment was moderated down.
and also, I don't have to put my email in the database of a company that may or may not honour their privacy policy when times get tough, when I download junit. Maybe I'm a little jaded about open source sites that look flashy and require emails to download...burnt by the whole 'jive' from coolservlets.com fiasco (making lot's of cash are you now boys...remember, you can't buy back your dignity). The simple concepts are often the best, junit works great. I see no real advantage in this one.
I'm not certain why the parent was moderated as "informative"..it's clearly just a bunch of superficially technical sentences strung together to make it look interesting, but it has nothing to do with JUnit. And what's a signal NP-17? (Null Pointer SIGUSR2? grin).
I believe that this quote is true for commercial software. The word processors, email clients and web browsers in the world should not be designed at the beginning of a project. The cost of doing that far outweighs the benifits. But...
When you are designing life critical applications (fly-by-wire, ABS, pacemaker, cancer radiation machine...etc) it is not acceptable to be redesigning while coding.
If you cannot model the way a system must react at all times, under any input, then what makes you think that your software should be responsible for someones life.
More and more software is being designed for life critical applications. If XP is being used to develop them, I'm worried. You cannot stumble onto 100% correct software. In a large system it takes a lot of money - an exponential curve, the last 3 bugs will likely take more money than the first 100 - time and a good design.
Our QA people have developed their own framework for running tests in C#. It automatically discovers test cases via Reflection API, allows to group them into suites, run suites, generate reports, debug. It took 2 people 1.5 months to write it (while also dogfooding it to themselves and writing actual tests). No big fanfare, no buzzwords.
"Refactoring" - holy fuck, where do you get such words?
*
What the hell is a maschoist? Some kind of hideous deformity?
public class TheUltimateUnitTestingFramework {
public static void assert(boolean b) throws Exception {
if(!b)
throw new Exception("assertion failed");
}
}
I give it to you for free.
Documentation isn't done yet, but I'm planning to spend the next year developing it.
For those interested in the value of refactoring (whether it's merely a buzzword), I can refer to a research project of ours, at http://win-www.uia.ac.be/u/lore/refactoringProject
By the way: since the article does not go into behaviour-preserving restructuring of JUnit, they shouldn't mention 'refactoring' in the title.
The junit docs document not only the framework, but the samples, and several implementations of TestListeners - an apples and apples comparison would be with the junit.framework package.
Focussing in on this, it looks like you've dropped the Assert class (presumably this is only for jdk1.4?). TestCase and TestSuite have been merged (seems sensible enough), and you've also provided a concrete Runner (no equivalent in the junit framework). This leaves the refactoring of TestResult etc into Report (roughly speaking).
Can you explain the advantages of what you've done in that last bit? I don't see that refactoring as resolving the classloader issues you mention in the article - they'd be handled in Runner/TestListener implementation.
Thanks,
Baz
I have another one,
it is geared towards tests with multithreaded environment.
here is the project page
http://www.michaelmoser.org/jmtunit/
The library sets up a thread pool (n threads) and runs a number of identical test sequences; each test sequence is a virtual user (m users). All threads are waiting on one input queue for test requests. The threading modell is the same as in most of the popular application servers
The programming interface is very similar to JUnit - there is a JMTUnit.TestCase with some additions, there is JMTUnit.TestResult.
The tool does print out performance statistics
Sadist moderators.
the API contracts, they gave up and rewrote it.
If they rewrote the API then this is not refactoring. Refactoring is when you rewrite code without changing the external interface.
Kent Beck - co-author of JUnit - recently published "Test-Driven Development", a book about how you can change your coding habits by creating the tests up front.
One of the most useful sections is a complete walk-through of the creation of unit-testing tool for Python - it's probably the best way of understanding the internals of the JUnit framework without trying to reconstruct every single line of code...
It's all very well in practice, but it will never work in theory.
'kin right.
Superfluous layers of abstraction are the real enemy today. Too much flexibility, too many layers in the class library, not enough code that actually *does* stuff.
Permature optimization is not the enemy any more. When Knuth complained about it most people wrote in assembly code and it was a concern. He was talking about things squeezing out an instruction by reusing the value in a register in a counter-intuitive way. Since OO became the dominant paradigm we have gone too far in other direction. The great thing about optimization is it causes people to focus back on what they are actually trying to achieve instead of concentrating on their over elaborate badly design class hierearchies.
http://rareformnewmedia.com/
I found the same problem when working with Ant and XML in JUnit, because the Ant system does its own class loader for the underlying XML parser, and my stuff got a LinkageError thrown when I tried to instantiate a parser, forcing me in a utility class to catch that error and call the good old fashioned "new" on the parser class within Xerces2.
"But remember, most lynch mobs aren't this nice." (H.Simpson)
-- Joe
Man, does this make my blood boil every time I read it. At best, it's outdated, at worst irresponsible. 'Back in the day' when implementations were typically close enough to the hardware that the 'fundamental cost of operations' was relatively well understood, this type of thinking wasn't unreasonable *at the code level*.
However, when selecting base architectures, you have to think about performance *up front*, since you're not going to change the fundamental architecture at the end of the project with a few lines of tweaked code.
Also, now that we have Java and OO and abstraction up the ying-yang, optimizing code is orders of magnitude more difficult in a real world project than it was when a project was a few 10k's of C code. Ever try 'refactoring' a significant API at the end of a project when you're already late?? Doesn't happen, and you ship something that runs like a dog.
Truth is, you have to worry about performance right from the start. Performance and clarity are by no means incompatible when done properly.
Why is this modded troll, seriously? The fact is .NET is genuinely a quality piece of software. Perhaps his use of "MS" in place of the more proper "M$" is what flagged it troll material.
I hate to be a troll, but maybe you guys should put up a warning on the front page that people that use Microsoft products or think MS is anything short of the antichrist are not welcome - it would save those of us suckered in by that "News for Nerds. Stuff that matters." tagline some trouble.
existing code
In general that should not be an issue, as the code really expresses the function of the system and testing that function is somewhat orthogonal to the actual implementation. Obviously it makes life easier if you are able to create certain probing point inside the system, but even if that is not possible to do, you still can dissect it into manageable "units" that you will be able to test - the regular drill apply some input, check whether the output matches. Most of the time for the "foreign" code it's just an issue of finding proper point where to feed test data, and where to observe output. They are there, just have to be found
databases
More challenging, though still very possible. The easiest approach is to create a special schema just for testing. The main goal here is to have known data, i.e. to be sure that no other application changes them while test is running. It also helps that no one becomes upset about your test changing other peoples' view of the system state. One more trick is to write tests so that they restore database to the state it was before the test is run. And the last one - it is helpful to code an alternative data access methods, f.e. if youe object saves itself in the database, you check the result but reading it though "raw" JDBC.
user interaction
In general you are absolutly right - it's almost (maybe even totally) impossible task. Yet, again the right way is approach it is to split the "horizon", i.e. apply testing input to methods that normally accept UI inputs. In other words you'll have to create a "fake" implementation of your UI api, and make you code use it during testing, and that "fake" UI will feed the system with data. An example (not really GUI, but illustrates the concept): I had to test an object that parses HTTP requests and provides more convenient than HttpServletRequest and system specific services to the rest of the system. All I had to do is to create an implementation of HttpServletRequest that I populated with test data during test set up, and then feed to the object being tested my "fake" implementation. The same, maybe with a little more effort, can be done to emulate mouse clicks and movements, key presses, etc.
multi-tasking
No general answer unfortunately, at least not the one that can be applied everywhere. I used to "split the horizon" here too and to test extensively methods that run without cross-thread interaction, and minimize and encapsulate at the same time the code that serves as a cross-thread "bridge" and ... "test" it in my head many many times. Key part is making the code that let threads communicate really-really small, so that you can "run" it in your head.
Hope that'll help, even if indirectly.
... but it is not the whole story. The rule against premature optimisation exists for a good reason: most programmers have terrible intuition about performance. The one thing you should do up-front is get the architecture - the distribution of code over machines, processes and loosely-coupled modules - right, so that it does not contain performance bottlenecks that will be impossible to get rid of later. Prototypes work well for this, as do small applications that are structurally similar to the one you want.
After that, all performance optimisation should be left until you can profile the application. Bits of the result will undoubtedly suck and need to be rewritten, but I'll put money on most of them not being the ones you expected.
This is where abstraction comes is. Abstractions are a very important aid to clarity, because they allow parts of the system to be considered in isolation from one another. To give a slightly practical example, a product I recently worked on used a variant of the banker's algorithm to avoid deadlocks. An odd choice, I know, but it was absolutely imperative that the system shouldn't dealock, and killing or restarting threads was unacceptable. The algorithm is complicated, and the original implementation took the simplest possible approach. When we profiled it, this turned out to be too slow, so I rewrote it, but because it was reasonably isolated from the rest of the system, I could do this without touching any other code.
Good abstraction is not the enemy of performance. It is its friend. In good abstractions, the data and code that need to work together, live together, and are shaped to one another. Oddly enough, this is what you need for performant software, too. On top of that, good abstractions make code easier to change.
What is the enemy of performance is bad abstraction. Which means too little or the right amount in the wrong place, just as much as too much. Aspects of Mozilla's design show signs of bad abstraction. In particular, it is hard to develop good abstractions for cross-platform UIs, and Mozilla's attempt is no exception. Using the HTML/XML renderer and JavaScript to implement the UI is a clear example of Bad Architecture, see paragraph 1, even if it is convenient. Similarly, the need for the elaborate modular design is debatable.
The design of the Linux kernel, is in many ways an example of Good Abstraction. Although it isn't written in a language that particularly facilitates abstraction (it is an OS kernel, after all), the different components are reasonable separated out. There don't appear to be direct dependencies between the IDE driver and the memory allocator, for example.
From the article...
Then why don't you spend some time integrating your code into the JUnit code base? The Open Source cause (or the XP cause or whatever) is not helped by having two implementations of the same concept. This is a self-serving article reads like a marketing piece designed to show how clever the authors are. I'm not falling for it.
Four fifths of all our troubles in this life would disappear if we would just sit down and keep still. -C. Coolidge
You didn't read the article! How did you get a score of 3? Picture this: I don't even read the topic of the post, so I can add a comment like "I wonder what this is about?"
Come one Shamir.
Bill
Upon seeing the box was too small, Schrodinger's Elephant breathed a sigh of relief.
Damn, this guy Kent Beck surely (and Martin Fowler, and Jeffries, etc.) surely can write a ton of books on the same topic.
It is all really the same philosophy - XP and refactoring and TDD (Test Driven Development), and so on. There is a lot of overlap there, and all is (obviously arguably) just good advice for approach to software development.
How many times can they repeat the same thing?
It is all just refactoring of the same material, same philosophies.
What's the big deal?
At over $30 USD per book, plus all the invitations to give speaches, presenations, and so on, it's quite obvious to me why they are doing it.
Isn't that a bit lame, though, at least for them?
Doesn't it get boring?
Simpy
Suite! oops I mean Sweet.
the moderator could see this - its that simple.
If you don't believe me check out some of tshak's other posts.... careful though he's sly, reasonable in some forums to gain the good-K but once the topic of the beast is raised... well its fake and green as far as the eyes can see
Hi,
r ation/index.htm, but these only generate the skeleton of the methods
:)
I'd like to know the experiences of people with regard to automatic(?) code generation of JUnit test cases. I'm refering to projects which do not work the XP way, but still would like to have high % of unit testing (meaning, unit test code is done at the end), and to projects that need to deal with legacy code.
Specifically,
(a) What has been your experience with tools which auto-generate test cases that give a good % of code coverage when tested with code coverage tools, such as Clover?
I've worked with
- open source options mentioned in http://www.junit.org/news/extension/testcase_gene
- JTest, which generates pre-determined values for primitives, and does nothing about Java value object classes, and expects the programmer to add in the "other" test cases - which in effect turns out to be a lot.
I guess what I'm looking for is a tool that "understands" the code and generates unit test cases for all the branches and conditions in the code, in order to achieve maximum code coverage. Is there any such beast out there?
(a1) Has anyone worked with TogetherJ/XPTest (http://www.extreme-java.de) for generating JUnit test cases? I didnt see too much of high-lighting on the TogetherSoft site on this one... What have been your experiences?
(a2) On a related note, does anyone know of XMI-compatible JUnit generators, which would remove the dependency on the UML tool?
(b) What has been your experience with mock objects? Have they been too cumbersome a process for the average project size (say, 100 classes, with 10 value object classes)?
(c) Building functional test cases: I've heard of JFunc; what are the other options, and how do they compare?
Thank you very much,
situ