Slashdot Mirror
Retrofitting XP-style Testing onto a Large Project?
Mr Pleonastic submits this query for your consideration: "I work for a small startup (ok, me and another guy comprise the entire development team) that has somehow managed to survive the bust, attract a number of customers, and build up about 300K lines of functionality. Up to now we've made it by being smart and conscientious hackers, but I'm increasingly embarrassed by our shortcomings in testing. I like the XP approach to making enduring, automated test suites, but most of what I read about XP focuses on obvious stuff and changing your programmer culture at the outset. Does anyone have experience with, or advice for, retrofitting it onto a fairly mature project? What do your test suites look like, anyway? The bugs I fear most are of the 'If the user does X and then Y, the result blows away our assumptions' variety, not the 'Oops! My function returned the wrong value' variety (which happens of course). How do you write good test code for the former, without spending even longer debugging the test code? Is XP just for small, new projects?"
This is hard to answer in a short comment. I'll try, though you're welcome to contact me for more details through the information on my website.
Retrofitting tests onto an existing project is hard. Not only is it tedious, time-consuming work, but you're always haunted by specters that ask "How do you know the test isn't broken?" It's nice to have the tests, but you'll spend a lot of time and energy creating them that could be better spent adding new features and improving existing features. Besides that, it'll likely sap any motivation you might have had for testing.
It's much easier to draw a line in the sand and say "all new features and bugfixes will have tests, starting now". Before you fix a bug, write a test that explores the bug. It must fail. Fix the bug. The test must now pass. Before you add a new feature, write a customer test that can demonstrate the correct implementation of the feature. It must fail. Add the feature. The test must now pass. From the programmer level, you can write programmer tests through the standard test-driven development style.
It still can be tricky to get started, especially with customer tests, but they don't have to be beautiful, clever, or comprehensive. They just have to test the one feature you're working on sufficiently to give you confidence that you can detect whether or not it works. You'll likely have better ideas as you gain tests and experience and it's okay to revisit the test suite later on to make it easier to use and to understand.
The nice part about this system is that it adds tests where you need them where the code is changing, whether it's a part full of bugs or a part under continual development.
Keep in mind that to do testing this way, you need to be able to work in short, clearly-defined, and frequently-integrated steps (story and task cards, in XP terms). You also need the freedom to change necessary sections of the code (collective code ownership). It helps to have a good set of testing tools, so, depending on your language, there's probably an xUnit framework with your name on it. Also, it can be counterproductive to express your development and testing time estimates separately. At first, testing well will slow you down. It's tempting to throw it out altogether as a time sink. As you learn and your test suite grows, however, the investment will pay off immensely.
Your goal is difficult but doable. It's well worth your time.
how to invest, a novice's guide
One word: JUnit
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
This is something I've wrestled with too. Start where you'll get the most bang for your buck. Start with regression tests. I assume you're doing *some* testing (or at least your users are ;). When a problem shows up, make an automated regression test that surfaces that bug. Run it often and make sure the bug stays fixed.
With a 300KLOC codebase I have to ask is it boken down into components that can be tested in isolation. If it is, congradulations you've done some good software architecture. You can start by testing the interfaces to the components. Make a test that triggers each error condition from each interface function/object. The tests will seem braindead simple (like passing in a NULL when a valid pointer is expected), but these sort of tests are suprisingly useful. Infrequently exercised error checking is one of the easiest things to let slip through the cracks when modifing an implementation. That will be enough to get your test framwork set up, and shake out all the forgotten dependencies between your components. Then it will be straightforward to add more testing.
It won't be easy. You should expect you'll have to modify your code to make it testable. But if you expect to keep this code around for a while, it will pay off in the long run.
yes, these automated test tools are called "smart monkeys".
"James Tierney, former Director of Testing at Microsoft has reported in internal presentations that some Microsoft applications groups have found 10-20% of the bugs in their projects using monkey test tools."
cpeterso
Get what the QA people call a "fuzzer".
There's two general types (often bundled together with a few other things as a test suite).
The first generates random keystrokes and the second generates random data either completely randomly or following some set of guidelines (field length etc.)
It still won't do everything that exposure to the real world will, but it'll get you a lot closer!
You're more likely to get people who know XP and TDD on http://groups.yahoo.com/group/extremeprogramming than you are on Slashdot, simply because the yahoo group is focused on that very topic.
No Zen is good zen
What several posters have already suggested.
Don't write tests for existing code just because it's there. Write tests for any code you have to change, and then do the change test-first.
Initially, it's going to be hard because legacy code is usually highly coupled. If you pay attention to reducing coupling, over time your code base will start to improve.
And get one of the xUnit clones for unit tests, and FIT (fit.c2.com) for acceptance tests.
John Roth
> 300,000 lines of functionality
If you have this much code, I bet there's some duplicated code in there. Ferret it out with CPD and you'll have that much less code to write tests for.
It probably wouldn't hurt to search for unused code while you're at it - again, you'll reduce the amount of code you need to write tests for.
The Army reading list
Instead of having to write tests for HttpUnit, you can simply record tests with JMeter. You also have the added benefit of JMeter's load testing capabilities. The only downside is that JMeter's UI isn't very mature (or intuitive for that mature).