Slashdot Mirror
How Do You Prove Software Testing Saves Money?
cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"
Basically you have to develop a mathematical model of the costs of the current situation, and compare it with a mathematical model of the costs of using tests. As part of this you will have to produce a plan for introducing tests, with the costs for each step itemised. Use the best numbers available, but don't worry if some of those numbers are "best guess". Just don't try to hide the fact. Put both models in a spreadsheet and come up with a number for how long it will take to recoup the initial investment (break even). Don't forget to discount future cash flows. In MBA-speak this is known as a "business case".
You are lost in a twisty maze of little standards, all different.
Excellent advice. Don't try to start a BIG, EXPENSIVE testing program. Have a look at the book: Working Effectively with Legacy Code by Michael C. Feathers
TCAP-Abort
#include "myclass.h"
#include "assert.h"
void test_bug123()
{
myclass a(10);
assert(a.value() == 10);
}
int main()
{
test_bug123();
}
---
How hard was it to create that test framework? You don't need something that is overly complex, just something that will pass if the test succeeds or fails if it doesn't. You can then improve the test cases/framework as you go along -- look at the Wine project for example. They didn't create http://test.winehq.org/data/ over night, it was built up gradually, starting with getting the tests running as part of the build and slowly defining a wine test API as needed.
Collecting metrics and reporting them should be done automatically, but not initially.
Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.
It's not easy but I have had good luck with pretty simple load generators and having the system put in random (from very long to zero) delays in the processes. Find lots of race conditions (short delays) and poor or missing interlocking (long delays).
The tricky part is making it replayable by recording the process step, random pieces of data used, and delays to a log which can be re-executed to prove the bug exists and prove the correction.
For really important systems I've run the random load generator for a month before sending the product out.
Rod Taylor
The Economic Impacts of Inadequate Infrastructure for Software Testing finds an average ROI for software testing somewhere between about 100% and 1000%.