Retrofitting XP-style Testing onto a Large Project?

Mr Pleonastic submits this query for your consideration: "I work for a small startup (ok, me and another guy comprise the entire development team) that has somehow managed to survive the bust, attract a number of customers, and build up about 300K lines of functionality. Up to now we've made it by being smart and conscientious hackers, but I'm increasingly embarrassed by our shortcomings in testing. I like the XP approach to making enduring, automated test suites, but most of what I read about XP focuses on obvious stuff and changing your programmer culture at the outset. Does anyone have experience with, or advice for, retrofitting it onto a fairly mature project? What do your test suites look like, anyway? The bugs I fear most are of the 'If the user does X and then Y, the result blows away our assumptions' variety, not the 'Oops! My function returned the wrong value' variety (which happens of course). How do you write good test code for the former, without spending even longer debugging the test code? Is XP just for small, new projects?"

Comments

Use the FailFast principle

[Dr. Bent]

Finding conditions that are outside your assumptions is not something you can do with a unit test. I have found that trying to simulate user creativity (stupidity?) with unit tests is an exersize in futility. Use your unit tests to make sure your methods do what they're supposed to do.

To find all those tricky combinations of use cases that blow away all your assumptions, just stick to the Fail-Fast principle. If you find anything that goes even slightly wrong, complain. Loudly! Throw an exception, pop up a dialog, whatever you need to make sure that everyone knows an error just occured. This will do two things:

1) You'll find a lot more errors in your code. You'll also be motivated to fix them quicker because the app will be unusable until you do.

2) You'll reduce the likelihood of generating bad data. The only thing worse than your program doing something wrong and crashing is doing something wrong and NOT crashing. Users will usually forgive you if your software crashes. If you start giving them bad data, they'll lose confidence in your app and never trust it again.

Good luck, it's tough!

[Anonymous Coward]

I have a similar situation, I have a bunch of code that "mostly works" and I'd love to have unit and acceptance tests.

But it's really hard to add it later. I mean REALLY HARD. The tests are tedious and boring and after 2-3 I get tired and the tests have errors.

If you follow the XP test-first technique, you code comes out MUCH different. You have low coupling, you have "testable" code where the pieces are interchangeable (so you can easily use mock printer objects or non-RDBMS backends, etc), and generally it's really elegant code with little extra work.

And you don't get bored writing test-first because every time you write a test, you then write the code that passes the test and it's really a feeling of accomplishment! And you don't get "lost in the big picture" because you are focusing only on passing that one little test.

The same is true for acceptance tests. I use HttpUnit to automate web apps, and although I'm not quite as religious about testing the interface, it's great for "add record, query record, delete record" stuff, to make sure it doesn't blow up when the end-user does something basic. For instance I had some code once that worked wonderfully, except login was broken. Since I was testing while logged in and never thought to log out and log back in, I never caught it in my manual tests. Automated tests can catch the stuff you forget.

So I'd recommend requiring tests on all NEW code (you'll see a big difference between the old and new code I bet, in terms of simplicity and low coupling).

And whenever you refactor the old code, start by writing tests that the old code passes.

But it will really be tough to retrofit ALL your old code with tests. I'd even say it's not worth it because your tests will not be good.

And remember: EVERY LINE OF CODE MUST EXIST TO PASS A TEST. That should be your goal on new code.

Refactor to testable code

[Chris Hanson]

If there's something that's particularly hairy in your existing codebase, the next time you need to modify it, spend just a little time refactoring it first so you can make your modification more easily. You don't need to do a six-month rewrite, just a little bit of refactoring to remove local duplication and confusion. And remember this about refactoring: It's not just about improving the design of existing code by making careful transformations of it, it's about rigorous removal of duplication.

Write unit tests for the code you're refactoring immediately before you do so, in order to verify that your refactoring isn't actually changing the functionality. In other words, if you're extracting a method Foo::ExtractedMethod from the method Foo::BigMethod, start by writing a couple unit tests for Foo::BigMethod and verify that they pass. Then write a unit test for Foo::ExtractedMethod and watch it fail (because you haven't done the extraction yet). Then actually perform the extraction.

Once you've added a new feature, don't consider it completed until you've done another small round of refactoring to remove any duplication that adding the feature introduced to your codebase. Do the same thing as above, writing unit tests for any code that you're refactoring that doesn't have any yet, and ensuring that your new feature's tests all continue to pass. After a few feature additions your codebase should actually start getting cleaner and your test coverage should go way up.

Another tip: Try not to have interface-layer (View in MVC termonology) code do too much work. Try to keep the application's business logic in the Model layer and its interaction logic as much as possible in the Controller layer. This will make it much easier to write unit tests; your View layer will be very thin and mostly serve as a wiring-up of your Controller layer to various human interface widgets. The end result is that you can then write a test to validate that entering one value here and another value there doesn't invalidate your assumptions simply by tickling the appropriate Controller-layer code, rather than by trying to "fill in" text fields and "push buttons" via test code.

If you currently have a lot of logic built into your human interface widgets, those are prime candidates for refactoring the next time you need to touch them (or notice they contain some duplication relative to the code that you're currently touching).

And as others have said, check out the Extreme Programming mailing list at Yahoo! Groups. It's a great list with a lot of great people who are very willing to answer questions and help any way they can.

Makes assumptions explicit

[nimblebrain]

One thing unit testing is often spectacularly good at is pointing out where assumptions have been made and not spelled out. This often takes the form of "negative testing". (What happens if you add a NULL to that list? What happens if you try to access the -1th element in an array? What happens if you neglect to set an IP address?)

What you'll likely find is that, in a number of spots, the refusal or assertion is not spelled out. Occasionally, it can take some mulling to figure out how to deal with those edge cases (could NULL be valid in some circumstances? Should we make a different derivative or member variable to determine that behavior?)

There's much positive testing to do, too.

Don't bother with testing the extremely simple stuff. If it's to the point where you might as well be questioning whether the compiler can do its job, you won't gain value from it, and it will bore you to tears. (Mind you, if you have one of "those" compilers whose very foundations you question... :)

If you have classes that produce output lists/objects, one nice technique to use is, instead of checking the output manually, is to create an equals/== method for your output, then create the expected output in your test and compare it (via your equals) with the output from the class.

Some other miscellaneous observations I've made:

• Unit testing gets a lot more interesting and a lot easier the first time it flags something real. Fortunately, it often happens just as unit testing was becoming boring again.
• Always throw in a negative check (something designed to produce the wrong output) - it's easy to produce checks that always return true even if the output is wrong (e.g. I had a list comparison function to determine equality which would kick out if one of the lists was shorter, but the elements were equal up to that point - doh!)
• Isolate the class as much as you possibly can. There's a whole technique to creating "mock objects" to help. For example, instead of using a real database object, make a fake one that returns specific rows, or instead of creating something that listens for a hardware signal, create one that waits for a command from the test.
• If you think of some condition your class might violate, avoid the temptation to go off and fix the class first. (At the very least for motivation's sake so you can see a failed test - seriously!) This is also a good thing to do if you're having a debugging session and one of those so-called "impossible!" conditions is happening (this could never be negative! there should always be something on the command line!).
• If there's an illegal condition you want to test for, add an assertion/exception into your class instead of merely checking the result. This helps 'fail' other tests and code that aren't setting up objects properly but report that they 'pass'.

One of the hardest things about writing unit tests is trying to interface to the outside world. Whenever you can, avoid it. You can fake things to a point (using 127.0.0.1 as an IP address in some tests, for example), but you'll have to fall back on functional testing at some point. That's another good reason for keeping as much logic out of the view as you can.

One note of hope: the most difficult part of unit testing is getting started. Honestly. Once you "get it", you will always "get it", so hang in there :)