Slashdot Mirror
How Do You Prove Software Testing Saves Money?
cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"
Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?
Are you sure that it will be?
but I'm sure that in the long run it would save time and money
How much time and how much money?
What you need here is hard numbers to back up your opinion. Specifically, and I hate to say the word, but you need metrics.
- How much do these bugs cost (these are a mixruee of hard numbers (dev time, etc) and "fuzzy" numbers (customer relations and so forth)?
- How much is setting up this test suite going to cost? How much is it going to cost to use it?
- How effective do you think it will be (look at existing bugs.. would they have been caught)?
That last one is especially important. Testing is good, but it doesn't solve all problems and there are other options. Really take a good look at whether previous bugs would have been found in your proposed system.
Once you have this information, the math then becomes pretty simple.
If your program is modular-ish .. it might be possible to do a trial. This is probably a good idea anyway. There are different ways to do testing, some of which are incompatible with certain development mindsets. Best to find that out early on a small piece vice the entire code base.
Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.
http://stephan.sugarmotor.org
We've had to justify creating auto test suites where I work.
Over the last decade our product has grown from one code-base into three strands, each with separate customer foci, and we've had a healthy amount of staff turnover so that there are still brilliant, creative and skilled people working on it but some of the original knowledge has left us.
We found* numbers to justify that automated testing of existing features, applied later will protect against regressive changes. Even where there are complicated features which were not modular in design, or which lack good interfaces, the tests have saved us massive amounts of time testing by hand. The real win is hidden under something we didn't realise until later: creating the tests have forced us to really document what the features are and how they work**, sometimes from a unit-test level, sometimes at the interface level and sometimes in a top-to-bottom vertical slice. Once you have a record of what your software does, in a computer which is skilled at remembering exactly and repeating exactly what some former staff member told it a couple of years ago, you have a decent reason to be confident that your bug fixes won't cause more harm than good.
*: ballpark figures / educated guesses / made up.
**: We favour working code over comprehensive documentation, until our agile team is reassigned to other projects or leaves the company.
WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?
Rest assured that Chinese companies test their products precisely *because* it saves money. If it didn't, they wouldn't; they don't have a commitment to good quality for its own sake.
An untested product leads to high rates of return (lost money), customer dissatisfaction and brand name damage (HUGE loss of money in the medium to long term). Nobody puts up with bad products anymore. Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.
Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.
1%? I think even if the code is buggy, it's quite likely that the code now depends on those bugs in various ways.
In any case, the right way to refactor is to make sure you've got proper tests for the code you're going to refactor. That way, after refactoring, you can check whether the functionality is still unchanged.
So yeah, the submitter has a problem. What he really needs from a code-quality point of few is pretty big and expensive. If there's no money for it, but there is money for continuing as before, then I guess that's the only option here.
If you truly are working for a small software development company, cost may not be the issue. Frequently (I may even say universally) the issue facing the owners of small companies is not cost, or even profitability, but cash flow -- making sure that there is enough money coming in to make the next big expense coming up (frequently payroll). In small companies there typically aren't the cash reserves available to spend on unanticipated expenses or program delays. The boss may even agree with you that the overall cost would be reduced if software testing were introduced, but may not "want to invest the time and money" because the company does not have the cash available to make the investment.
For example, the software you're working on needs to ship by the end of January to receive payment from the customer by the end of February, so that payroll can be made in March. If bug-fixing stops in January for development of the bug-fixing program, the customer doesn't get his software until the end of February, so he doesn't pay until the end of March and so payroll is missed that month. Having fewer bugs in the long term has to be balanced by the need to stay in business until the "long term" is reached.
It's like anything else in finance: You have better options and get a better return on your investment if you invest $100,000 than if you invest $1,000, but if you don't have $100,000, it doesn't much matter -- you do what you can with the resources you have available. Similarly, you get a better return if you're able to invest your money for a year than if you must have it back in a week. (The guy who said "time is money" actually knew something.) This trade between cash flow and long-term efficiency (leading, one hopes, to profitability), i.e., knowing when and how much to invest, is the real art of business management.
The solution to your problem? IMHO, incremental investment. I agree with those above who suggest implementing tests for new bugs found. This should enable you to begin to work more efficiently over time, without substantially delaying current work, while collecting metrics (I know, I know...) that can help quantify the cost of the bug re-work. Once a substantial body of tested bugs has been collected, one can institute a formal testing program (preferably by including it in the next project, so that it is a planned expense) as a cost reduction, since by then it should speed up development work over the ad hoc method then in place.
If you are catching shit when a bug happens and they refuse to set up formal testing get out of the job now.
If you aren't catching shit of any kind, then look at the bugs as helping you keep employed. If it annoys you, go find another job.
I've worked for similar cheapskates. They aren't likely to change.