How Do You Prove Software Testing Saves Money?

cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"

Comments

Design from the beginning is important too.

[ls671]

I would say the most important thing to save money is a good design and programming guidelines for the application.

Test suites are obviously a good idea but it may end up as a tedious task if the application as poor or no real starting design and if everybody just did things their ways.

I have seen many applications that had grown larger than expected that typically cost 10 times more than it should to maintain, bug fix and implement change requests.

It is sometimes possible to fit existing code in a new clean design and that saves a lot of money if the application is going to be used for a long time.

Let's hope your application has proper design ! ;-)

Re:Design from the beginning is important too.

[Chatsubo]

I agree with the above, when you set out to write code knowing you're going to have to test it, it will influence how you do the design and insulate functionality from the rest of the system, in order to be easily testable.

The problem, however, is you've just made this guys problem even worse. He can't even get allocation for testing. Now he'll need allocation for refactoring AND testing.

My opinion is to break it down into small incremental changes that may take years to filter through the whole system. He can get into the habit RIGHT NOW of writing a small test case for every problem he finds, this will solve his immediate problem, well... immediately. It will focus testing only where it's most badly needed.

Then design wise, the thing to do would be for every problem found, see what can be re-factored and improved right there and then, and do it immediately (with test cases to prevent regression).

The trick then is to keep metrics of how many issues boomerang now, how much time is spent fixing them, and then also keep track of how many "test-cased-and-refactored" issues boomerang. Yes, this will require rigorous timekeeping.

Re:Design from the beginning is important too.

[ls671]

> My opinion is to break it down into small incremental changes....

I agree, I just posted a reply that is along the same line:

http://slashdot.org/comments.pl?sid=1932550&cid=34742086

Re:Design from the beginning is important too.

[Kjella]

Then design wise, the thing to do would be for every problem found, see what can be re-factored and improved right there and then, and do it immediately (with test cases to prevent regression).

Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

That is why you have to find some kind of border where you have pretty solid control on all the ways this code is called, it can be a module or a group of classes. Then you preserve the exterior while you build a new and clean interior, where you can safely remove the quirks because you're sure nothing depends on them. The kind of "micro-refactoring" that you seem to suggest he could do along with bug-squashing just doesn't have much effect in my experience. You need to have a certain scale to it before you get any real clean-up effect in the middle.

Re:Design from the beginning is important too.

[mcvos]

Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

1%? I think even if the code is buggy, it's quite likely that the code now depends on those bugs in various ways.

In any case, the right way to refactor is to make sure you've got proper tests for the code you're going to refactor. That way, after refactoring, you can check whether the functionality is still unchanged.

So yeah, the submitter has a problem. What he really needs from a code-quality point of few is pretty big and expensive. If there's no money for it, but there is money for continuing as before, then I guess that's the only option here.

Re:Design from the beginning is important too.

[Anonymous Coward]

Do you have a formal process for versioning and versioning software? When you stated that you have seen bugs that you fixed crop again it brings to mind people not using the production version piece of code, modifiying it then putting it prod which can reintroduce previously fixed bugs. Good test plans/tools that get good coverage are important but you also need to have some sort of process in place when changing your codebase as well.

Re:Design from the beginning is important too.

[Anonymous Coward]

Spending money today to save in the long run is always a dicey proposition. First how long is the long run 1 month, 1 year, 10 years. Second people are usually overconfident in the accuracy of future predictions. You need to show real savings in a reasonable time frame. Managers are not impressed by your natural instinct to improve the code.

Re:Design from the beginning is important too.

[Anonymous Coward]

I know this sounds stupid, but what's a good way for someone to learn how to do that?

I mean, I've been writing code nearly my whole life, and yet I have no idea how to test properly.

Re:Design from the beginning is important too.

[ebuck]

Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

1%? I think even if the code is buggy, it's quite likely that the code now depends on those bugs in various ways.

In any case, the right way to refactor is to make sure you've got proper tests for the code you're going to refactor. That way, after refactoring, you can check whether the functionality is still unchanged.

So yeah, the submitter has a problem. What he really needs from a code-quality point of few is pretty big and expensive. If there's no money for it, but there is money for continuing as before, then I guess that's the only option here.

When that happens, you refactor out the bugs by formalizing the dependencies between the bugs and the code that depends on them. Then you migrate the two sets of odd behaving code together, and you let the oddities disappear as they annihilate themselves in a sort of positive cancels negative, yin / yang solution.

Time consuming, yes. But typically the results well worth it.

Re:Design from the beginning is important too.

[whizbang77045]

I agree with the above. One can't really test something without knowing what it was intended to do. One needs a software requirements document (or specification), perhaps an implementation document describing the software approach that will be used, and finally, a test procedure that verifies that the software really does the things in the requirements, and is not buggy. It's really the final step in a design/engineering process.

You can't test quality into software. It has to be designed in from the start. The test just verifies that it's tere.

Re:Design from the beginning is important too.

[Anonymous Coward]

Have rewrites refer to the initial design documentation when there are grey areas. Problem solved.

Re:Design from the beginning is important too.

[infosinger]

I fully agree that you need to design in quality. I go by the adage "You can't test in quality". Regardless, you WILL test. The question is whether it is more economical to test before you release the product or once its in the customer hands. Some failures can have significant financial impacts(for example, how about a defect in a car engine control module that causes the engine to shut down -- VERY expensive). In the less impactful consumer market you will have degradation in sells if the quality falls below a certain point. It comes down to a tradeoff between risk to business vs. cost to test pre-release. One lower way to mitigate the risk is to turn up/down the depth of testing depending on the change. For example, if you are making some minor UI changes, you might want to do a very light regression test but if you are doing some fundamental refactoring of a core part of the product then you might want to have a more involved set of unit/subsystem/integration tests that cover as much of the product as possible.

One method we use is to err on the conservative side and to over-test initially and if the defect density is low enough then back off on later releases as the quality level is much better known and the resulting post-release risk is lower.

Bottom line, there is no hard rule, you have to know your costs and your risks and make daily tradeoffs between the two.

Re:Design from the beginning is important too.

[zigamorph]

What is "initial design documentation"? I have heard of it but have never seen it in the wild.

Re:Design from the beginning is important too.

[rtb61]

In this case, the real answer is how do you prove testing saves money, truth is, exactly the same way, customer service and support saves money, you can't.

It's the old bean counter, no real world experience con. It is dead easy to prove not spending money on testing or it's partner customer service and support, saves money and if you still need to, offshore it to third world contractors for cents on the dollar. Of course experience teaches exactly why those easy to prove bean counter exercises blow up in everyone's face, with lost customers.

So once you have a boss or manager who lacks the experience and wants you to mathematically prove how spending money to prevent losing money is worth it, don't, give up walk away and let the noob learn their lesson the hardway (avoid the "I told you so" because they will want to blame you for their mistake).

Don't forget to ignore those people who say, just do a perfect job and you don't need testing or customer service and support, to defend their bean counter no customer relations experience mentality. Investing in precautions is an impossible sell, until the have had to pay the price of failing to do so.

Re:Design from the beginning is important too.

[mcvos]

Testing is a big subject. The basic ideas behind automated tested aren't that hard to grasp, but actually doing it requires some foundation. To be honest, it's only during the last year that I wrote unit tests for a lot of my code. And I didn't even do it all the time, which is what I should have been doing. It really requires a different mindset, and once you're there, I've been told it's really easy and actually saves you a lot of time, but it's not so easy to get there.

Here's my little testing primer for the complete newbie:

There are three kinds of automated tests:

Unit tests: they're very fast, and test the basic behavior of individual objects. You'll have to mock the relationships between that object and other objects, and for that, a mocking framework would help a lot, and that's something you'll need to learn before you can do unit tests well. Mocking is really the big skill you need to learn in order to write good unit tests.

Integration tests: They actually start up your application before running the tests, and they can test database stuff and complex relationships. They're a lot slower than unit tests, however. I don't have much experience with them yet.

Functionality tests: These don't test your code, but what actually appears on the screen. You can ask functionality test frameworks (like Ruby's Watir) things like: click a button with the text "next". Check if we're sent to page X. Now click on link "foo". Etc. Really cool, but obviously slow.

To get started on testing, you might try fixing some bugs in Ruby's Rails web framework. They will only accept your fix if it includes tests that fail on the old code, and succeed after your fix. It can be annoying, but it's a good way to get your head in a proper test-driven mindset.

Re:Design from the beginning is important too.

[Smallpond]

Do you have a formal process for versioning and versioning software? When you stated that you have seen bugs that you fixed crop again it brings to mind people not using the production version piece of code, modifiying it then putting it prod which can reintroduce previously fixed bugs. Good test plans/tools that get good coverage are important but you also need to have some sort of process in place when changing your codebase as well.

More likely what it means is that when they fix a bug once, they aren't incorporating a test for that bug into their test process or updating their documentation. Doing that prevents bugs from reincarnating. I've seen the same bug reappear in code 3 times because successive programmers thought there was a simpler way to do something and didn't realize that the current code dealt with special cases they hadn't considered.

Obviously, a good spec and a good test up front would have covered the special cases, prevented the bug in the first place and would have been cheaper than fixing it 3 times.

First things first

[Anrego]

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

Are you sure that it will be?

but I'm sure that in the long run it would save time and money

How much time and how much money?

What you need here is hard numbers to back up your opinion. Specifically, and I hate to say the word, but you need metrics.

- How much do these bugs cost (these are a mixruee of hard numbers (dev time, etc) and "fuzzy" numbers (customer relations and so forth)?
- How much is setting up this test suite going to cost? How much is it going to cost to use it?
- How effective do you think it will be (look at existing bugs.. would they have been caught)?

That last one is especially important. Testing is good, but it doesn't solve all problems and there are other options. Really take a good look at whether previous bugs would have been found in your proposed system.

Once you have this information, the math then becomes pretty simple.

If your program is modular-ish .. it might be possible to do a trial. This is probably a good idea anyway. There are different ways to do testing, some of which are incompatible with certain development mindsets. Best to find that out early on a small piece vice the entire code base.

Re:First things first

[mcvos]

but I'm sure that in the long run it would save time and money

How much time and how much money?

And how long is that run exactly? I'm all for proper test frameworks, but if it takes 5 years before it starts paying off, and nobody knows where the app will be in 5 years, then it's really not such a great investment at all. So, like parent said: do the math. Figure out how much the test framework will cost, how much the recurring bugs cost, and then you've got some numbers to convince your boss. Or yourself.

Re:First things first

[Splab]

Also, putting testing infrastructure into an existing system is a huge undertaking. And it requires a very strict handling for all future commits - if you just slip a little on the test suite you are basically invalidating the whole thing.

Many a times I've tried to implement test suites, both as request from employers and also for own sanity sake - and every time it has failed because the workload was simply too overwhelming to support both testing and development.

In the end, the basic economics of it has dictated that it is cheaper to hot fix than to make it right the first time around.

Re:First things first

[tibit]

Frankly said it's not much of a business if they can't at least wish for where they'll be in 5 years.

Re:First things first

[arth1]

Also, testing introduces bugs. Both because there will be bugs in the test code itself, and because of bias - even when not meaning to, new code will be skewed towards passing the test case, and the tests skewed towards validating the code instead of finding bugs. I've seen examples of bugs that wouldn't have existed if it wasn't for the testing.

Then there's time wasted on testing things that simply don't go wrong, or where the impact would be so huge that it doesn't require a test case - if it prevents the software from running in the first case. Or where the impact of the bug is so small that it doesn't really matter, and rewriting the code is too expensive compared to the theoretical risk of triggering the bug (like a system not allowing someone to be have a name with all digits, or a rounding error only showing up at speeds over 1000 mph).

Yes, testing is overall good for quality, but can increase costs quite substantially.

If the same bugs keep reappearing, perhaps the bug fixer could be better at commenting when fixing bugs -- not only in comment blocks above routines (because they can be missed), but inline comments about what was fixed, and more importantly why?

Re:First things first

[balloonhead]

Why not just edit the splash screen with a big Google-esque 'Beta!" stamped over the application title? Problem solved.

Re:First things first

[Glonoinha]

In the end, the basic economics of it has dictated that it is cheaper to hot fix than to make it right the first time around.

What the OP failed to provide is the nature of the bugs his users are experiencing, the impact to the users and to the business, and the costs associated with the manifestation of those bugs.

If we're talking about user interface glitches where a button is not aligned with the others, the color of the button when pressed is wrong - then it is probably way cheaper to hot fix than to insure 100% quality the first pass.
If we're talking about workflow and data errors in a financial application with regulatory implications, or bugs that bring the system down for an hour and the company loses five figures in revenue for each hour of downtime, that's a different story.

Testing doesn't necessarily save money. Saves pride and credibility more often than not, but as others are saying it isn't necessarily the most fiscally prudent choice. Neither is a wholesale re-write of 10 year old code (even though that's our first inclination when we inherit code.)

Re:First things first

[msclrhd]

Tests can fails either due to the code under test, the test code or the environment.

Tests form a contract between teams (e.g. defining the way a protocol works, invalid and valid data with the expected results). They also serve as a form of documentation -- this is how you use this class; we need to support this corner case; this test checks that we don't regress this security bug.

Tests also document what your assumptions are -- e.g. does this method allow NULL pointer strings or not? One of my primary testing axioms is: assume nothing, assert everything. In the previous example, this means that there is an assumption that passing a NULL pointer to a function taking a string argument is valid; this axiom states that there needs to be an assertion/test case to check that the assumption holds.

When creating test cases, you need to define how the program will behave under certain circumstances. With sufficient coverage it then gives you the confidence that you haven't regressed anything.

But as always, start small and build up from there.

The article describes how a certain bug is recurring. Explain to your managers/team that this bug keeps cropping up (using evidence from the bug tracker). Write a test case for just that and include it as part of the build process. The issue should then be picked up on the developers machine, or on the daily/incremental build and not during the manual test cycle for a release. This can then be used as evidence to support adding tests for any bugs that are found. You can build it up from there.

Re:First things first

[Lumpy]

Exactly, the cost of not testing may be a IT helpdesk position that can be eliminated by doing better software testing. It can also be loss of productivity, loss of work quality, etc....

Honestly, just because it's software does not make this hard, replace the word software with "business process" or "machine" and you have the exact same problems. Would a company buy a $250,000 machine without research or testing to see if it was the best fit for their solution? No.. yet many buy the crap Ticket system that is BMC Remedy for close to the same price just because everyone else does.

testing is not a IT process it's a business process. Lack of software testing is simply a indicator of a failure of the management in the company.

Re:First things first

[Splab]

Absolutely agree.

Right now I work for a telecom, small one - but we still have some pretty strict demands on availability. Testing in our system is an absolute nightmare; what we can do, is prove that our program works for well defined input, but once you add customers to the equation, all bets are off. We have at several points in our history looked into test driven development etc, but fact is, the type of errors we see aren't easily detected by a fixed test. Most problems we experience in production is race conditions, testing for them and spotting them on a system with hundreds of simultaneousness calls is downright hard.

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

Re:First things first

[rtaylor]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

It's not easy but I have had good luck with pretty simple load generators and having the system put in random (from very long to zero) delays in the processes. Find lots of race conditions (short delays) and poor or missing interlocking (long delays).

The tricky part is making it replayable by recording the process step, random pieces of data used, and delays to a log which can be re-executed to prove the bug exists and prove the correction.

For really important systems I've run the random load generator for a month before sending the product out.

Re:First things first

[bmajik]

You don't say this explicitly, but most people in the thread seem to be assuming it:

why is there a focus on automated tests here?

I would contend that step 1 is writing a document. It describes the test scenarios you think are important, why you think they are important, and their relative importance w.r.t to other scenarios.

Shop the document around -- include it and update it when the cost/benefit excercize is done.

Eventually, you will arrive at a point in time where people begrudgingly agree that "we should check those things before [blah]". Once there is buyoff from management that it is indeed important to check some/all of the things in your document(s), automating as many of them as possible as cheaply as possible follows quickly behind. Nobody wants to do repetitive testing by hand.

It's still, however, a good idea to have a breadth-first search of the product's functionality that is done by a real human being.. a vague script fit for human consumption.

I say this because automated tests can only notice what they've been explicitly coded to look for. You can certainly make very advanced automated tests -- including fuzzy bitmap comparison to try ainf find "out of place" things, but the human mind seems to work in just the opposite way: it notices what it wasn't told to look for. That's a fantastic and amazing quality for something tasked with the job of looking for defects in complicated software.

Re:First things first

[Uzik2]

Kudos! Well thought out and written response.

Re:First things first

[fuliginous]

Such metrics exist. I've seen the evidence in the past with regards to software for automotive control (running the engine). Basically if you have to recall 300000 cars to be fixed at your cost (even just an ECU software upgrade) you have at the very least say 300000 hours of technician time and all the rest. Plus possibly if it was bad a few lawsuits to burn cash defending or paying out on.

I think full life cycle metrics in automotive put bug fixing once a vehicle was out the door at something like 50 - 200 times the cost of finding it early on during design reviews, code reviews and ultimately testing of unit, module and system as different stages.

But in that instance you are talking scale of units in the millions ultimately with many years of life so many millions of operational years in which bugs can bite! Plus the goods are high value. If you are talking low value goods and a few thousand copies well loose it under GPL with suitable disclaimers of suitability of anything is denied:-)

You get the picture, you might not be able to prove what you want because it is a time money equation with varied parameters.

In short you need the metrics for what you are focussed on and gathering those metrics may cost more than it's worth. The collection cost itself is a similar time money equation to consider in advance. Reality for contractor style companies is they don't want to bother testing because they get to fix the things they break in adding features for you or fixing other bugs. Testing is a path to being redundant.

Re:First things first

[doublegeek]

I have to disagree. For a reasonably small app that doesn't require certification or formal acceptance testing, it doesn't have to be all-or-nothing. You can start with just a few test scripts, and build up a test suite a little at a time. There's nothing that says the test suite has to be all-encompassing, especially if you've never had one and are doing testing ad hoc. Developing a complete test infrastructure from scratch can be a bit intimidating, but it's a lot easier if you start small and build it up. Eventually you'll have to refactor the test suite (most likely to improve automation and make it easier to write & integrate new tests), but by then you'll have a good idea exactly what you need to do. Also, depending on the app, there are a number of 3rd party test tools that make it easier to set up a test suite. These may or may not be applicable, but it might be worth looking into.

To convince the owner (the one who's paying the bills), you need to make an economic argument. That means you need to estimate both the cost to develop the test suite and the savings you'd get from it. This in turn means two things. First, the initial estimate - you'll need to guess how much it time it will take to write the test suite (that's one reason to start a little at a time - it keeps costs down), and how much time you currently spend fixing bugs you've seen before or that would have been caught by a test suite. You'll also need to take a guess at how many of those would be fixed with a test suite, both in the short term (when the test suite is immature) and in the long term (when it has matured). Second, it means you'll have to start collecting data to back up your estimates. How much time do you spend fixing bugs in the delivered product? What sort of bugs (i.e., would they have been caught earlier)? How much time do you spend on new development? During development, how much time is spent fixing bugs in new code, and at what stage do you find the bugs (unit testing level, integration, system?). With data like that, you can make a pretty strong argument.

One more thing: you may not be in a position to do this, but any data on customer satisfaction would also be helpful. Lacking data, you could make an argument based on anecdotal evidence, but this gets risky. In any case, all this would go into the "savings" part of the argument (which is really the ROI).

One final comment - reading between the lines, I'm wondering if it's worth paying a bit more attention to configuration management. Keeping track of exactly who has touched the code, when, and for what reason, plus the whole revision history, can really help with making sure bugs don't reappear and to quickly fix it when they do. I don't know if CM is an issue in this case, but it's critical to have solid CM - not just for development, but also for a test suite.

Re:First things first

[Meddik]

You are absolutely right there. Automated testing does have it's place... But that place is doublechecking a repetitive test case in solid, stable functionality that would be boring to run over and over again. Automated testing rarely finds new defects, but is great at making sure old defects don't pop pack up. Human eyes are best at exploratory testing, and finding new problems. As a QA Analyst, I'd say that about 75% of the defects I discover are things I was NOT specifically looking for... They are things that I come across incidentally, and say "Hmm... That's strange" and go off script to investigate.

Re:First things first

[swillden]

Write a test case for just that and include it as part of the build process.

This.

Setting up a comprehensive test suite for the whole application would be a huge, and hugely expensive, undertaking. The odds are basically zero that it will pay for itself in much less than a decade. However, writing a test case for each bug you fix will only increase the cost of the fixes somewhat, and will ensure that regressions are noticed quickly.

To make this work you will have to invest some time in creating the test infrastructure and wiring it into the build process, but there are plenty of tools that provide the foundation for whatever language you're using, so this shouldn't take more than a day or two. You may have to go to your management and get permission to take the time to do that, or else you can do it bit by bit over time in spare hours, or even just stay late a few nights and do it on your own time.

But before you do any of that, the first thing is to get buy-in from the other developers. Well, you may want to put the test framework together and write a test or two so you can show them what you're talking about. Management buy-in is good, but buy-in from your peers is essential unless you can get VERY strong management support. If other developers won't write test cases or, even worse, remove or disable your test cases, then this test suite idea is going nowhere.

Re:First things first

[julesh]

Automated testing rarely finds new defects, but is great at making sure old defects don't pop pack up.

Note that this is something the OP explicitly states is a problem here; in light of this I'd say automated testing is the best solution for him.

Re:First things first

[xmundt]

greetings and salutations.
          that issue of testing code has been a point of discussion for decades...It has been my experience
that generally code is written to pass the tests, and, in most applications these days, the data
flow is so complicated it is almost impossible to write a test suite that will adequately exercise
all the possible paths.
            It is my feeling that the better course of action is to create a good data flow plan, then, write
the code to make it happen. If the design is good, then, the probability of disaster drops
by quite a bit.
          For an existing system, like this one, it is likely that the best option is to start from scratch,
create the data flow diagrams, and, remake the code to match. At this point in time, the
code base is likely a scary mish-mash of programming styles and methods, so, a re-write
with some consistency is better than just testing the existing code.
            Having said this, though, I will say that there are some basic rules for that code that
need to be hard walls that have to be done. Simple things, like sanity checks on the input
data, checks for buffer/array over-runs and that sort of thing are very cost effective and go
a long way towards keeping the bugs out.
          So...on the down side, I cannot hop on the "test suite" ideal without a lot more info
on how effective it might be. On the up side, re-writing the code "from scratch" will
ensure job security for quite a while, and, will create a more consistent and usable
code base.

Re:First things first

[bmajik]

I read it as being a specifically frustrating example that was indicative of the complete lack of quality focus and sound software engineering governing the project.

And apparently the relevant people in his organization either don't know or don't care that this is the case.

So assaulting them with a bunch of technology, process, or methodolgy (nUnit!, Scrum!, Test First!) is a non-starter.

Testing is an unbounded problem. You can always do more of it, and the things that can be easily unit tested are usually the least interesting and valuable things to test. Professional testers tend to like to plan out the overview of the testing problem space and then cut it up into chunks according to some value-per-time function that the people writing their paychecks agree with.

Re:First things first

[poetmatt]

I see the issue here in the same light as you, more of a business analysis issue than a direct programming issue, cleaning up some programming being the result if appropriate.

good post.

Re:First things first

[theshowmecanuck]

It is my experience that test driven development is good in that it makes sure the code is unit tested. But unit tests do not take the place of user testing. What many people fail to take into consideration is that just because the application works doesn't mean it is doing what it is supposed to. For example, just because it displays the results of a financial calculation, doesn't mean the calculation is correct. Many times it requires someone with in depth knowledge of the business in order to make that determination. As much as developers like to think they are experts in a business, they rarely have the knowledge of someone with many years experience in it. Think of it like a poorly written SQL statement. It will likely return a result. It passes the unit test. But as we know about SQL... result sets from complex queries can be wrong or incomplete unless you validate the data. i.e. just because the code works, doesn't make it right... debugging logic is way harder than debugging syntax.

Re:First things first

[JustinOpinion]

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

Are you sure that it will be?

This is the key, I think.

There is a generally-held belief among coders that "doing it right, the first time" and "rewriting this mess" will save money in the long-term, and that managers are idiots for not seeing that. This can, of course, be true. But it isn't always true, and coders are sometimes projecting their OCD-desire to have nice code (and sometimes suffering from "I didn't write it so it must be crap" syndrome) and assuming that this will translate into the dollars and cents that the company cares about.

Sometimes it's worth it; sometimes it's really not.

The thing about money is that it is both non-linear (double the money doesn't necessarily have double the value; sometimes it has more than double because you can overcome barriers; sometimes it has less because of diminishing returns, etc.) and temporally varying (inflation, time-value-of-money, etc.). Because of this, it can actually make economic sense to do something in a half-baked way, and "pay the price" later on (in terms of higher support costs, or even having to totally re-do a task/project). For example, in cases where you "absolutely need it now" (the value of having it finished soon becomes larger than down-the-road problems) or because you can't spare the cash right now (the value of using that money to do something else right now is larger than the down-the-road problems). (If you want a physics analogy, notice that money is not a conservative force-field: it is a path-dependent process...)

I'm not saying that it always makes sense to do slipshod work now and suffer the consequences later. There are plenty of dumb managers who over-value short-term gains compared to long-term. But that doesn't mean that the optimal solution is to spend massive effort up-front; there is such a thing as being too much of a perfectionist. And, importantly, the right answer will vary wildly depending on circumstances and the current state of the business. A startup may need a product to show (anything!) in order to secure more money. Doing it "right" will mean bankruptcy, which is far worse than having to keep fixing and maintaining a piece of shoddy code for years to come. A very well-established company, on the other hand, may do serious damage to their reputation if they release something buggy; and can probably afford to delay a release.

Actually figuring out the cost/benefit is not simple. In principle this is what good managers and good accountants are there to do: to figure out how best to allocate the finite resources. If you think you've found a way to reduce costs by implementing testing, then by all means show them the data that supports your case. However don't assume that just because testing will make the product better, that it actually makes sense from a business perspective.

Re:First things first

[e70838]

The subject is about bugs fixed that come back, formal test suite and cost. Automated tests are the only cost effective manner to avoid regression.
Human testing using documentation is very valuable for unformal test and finding new bugs, but are another subject.

I think that automated tests shall be implemented with almost $0 cost. Each time you find a new bug, write the simplest test that fails because of the bug, then fix the bug and see the test succeeding after your correction. Add the test to the makefile test target (ant target). Writing the test is a way to document the problem you have fixed and it is a way to check that your correction is working fine. After a short starting period, writing the test will not be slower than the usual tests you were doing before. In the end, you have an automated test suite. When the benefit of the method become visible, ask for other developers to follow your example and discuss with architects in order to have easy to test programs.

In general, lack of time and lack of resource is a good stimulus to do things in a creative cost effective manner, it is not a justification for bad work. The single project I have seen having time and money was a (almost) complete failure.

Re:First things first

[Splab]

Think you are mixing people up.

We have very good knowledge of why most commits happened, strict enforcing of commit policy has helped that quite a bit.

I don't know what world you come from, but the one I program in, any application needing automated tests are never small. Our need for formal verification started at around 70.000 lines of SQL and somewhere north of 400.000 lines of unproven code. At this point it became impossible to verify functionality across all systems talking to each other; we looked into starting unit testing and gave up, problem was simply too big for a small company to take on. Instead we have solved it by not modifying existing code where multiple service can access (e.g. the database or external APIs), this way any modifications are limited to whatever sub project is being worked on making subjective tests feasible, but never perfect => if you need a function to do something slightly different, we implement a new function next to the old one. Even functions known to be broken are kept, because as it turns out, some programs expect wrong data to be returned.

Now this sounds like an obvious submission to the daily wtf, but when working with limited time and limited funds you just have to make due with what you got.

Re:First things first

[jimrthy]

Rewriting from scratch is almost always the wrong thing to do.

Things You Should Never Do, Part I

Yeah, it's an "appeal to authority" argument. But it's been my experience that he's absolutely correct.

The only ones who win in this scenario are the consultants who are getting paid by the hour. Well, and your competitors.

I totally understand the urge when you inherit code from someone else. Heck, a lot of times, I feel the same urge when I go back to look at code I wrote 6 months ago. There's a reason that snarled-up nightmare was written the way it was. Make sure you understand that reason before you just scrap it and decide to start over.

Otherwise, you're just going to wind up with a newer snarled-up nightmare. It'll just have a bunch of bugs that were fixed (and left uncommented) in the existing version.

Re:First things first

[dgatwood]

Automated testing rarely finds new defects, but is great at making sure old defects don't pop pack up.

Depends on what sort of testing you are doing and on how complex the inputs/outputs of your functional units are. I recently put together a test suite for HeaderDoc (a documentation generation tool). The test suite treats the entire parser core as a single functional unit for testing purposes. At the moment, there are just shy of 300 parser and/or C preprocessor tests, all of which consist of a blob of input and the expected output from the parser core. Whenever anything changes in the parser, if it changes the output for any of the tests, I find out about it. This has only found an old defect popping back up once (and that was while I was deliberately reworking a poorly thought out fix to do it the right way). It finds new defects with regularity. Sometimes the changes are expected, so I approve them. Most of the time, they're errors, though, so I go back and figure out why the behavior changed unexpectedly.

Now obviously this design does not have test coverage for every line of code---there are probably edge cases somewhere that aren't hit by any of the parser tests despite my best effort---but it has been immensely successful at preventing the introduction of new bugs into the code. I've caught literally dozens of bugs before committing that would otherwise have made it into the code. Given that many of the programming languages supported are rarely exercised, those bugs would probably not have been caught otherwise.

More importantly, such a test suite can only determine whether something changed. That means that it cannot discover existing bugs, only new bugs. Similarly, it cannot say definitively whether the change is correct, incorrect, or unimportant. Those are things that only human-driven QA testing can achieve.

Similarly, when it comes to UI testing, automated tests are unlikely to show new bugs unless they involve pixel-by-pixel comparison of entire screens (in which case the false positive rate could be astoundingly high, depending on the app). So for UI testing, QA testers are absolutely essential.

Re:First things first

[dave562]

Another thing to consider is, what will you do with the time saved when you are not squashing bugs. It could be that the employer has factored the OPs salary into the cost of maintaining the application and it is less expensive to have the OP squashing bugs than it is to invest in a testing process.

Re:First things first

[Anonymous Coward]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

In my personal experience you're better off planning carefully to avoid them than testing to prove they're not there (which is akin to solving the halting problem, ime).

If you're hacking together, rather than sitting down and making an overarching design for something, Simple approaches I've found to work are:
- one big lock, so that you have no contention (performance often loses out here but your code is more likely to work)
- modular locking in small components, such as FIFOs with a blocking output and a notify on adding to the FIFO - the lock is either held briefly or a function is documented to block,and only one lock is held at a time if you're careful
- labelling every function that runs with a lock held before it's called; asserting that that lock is held when it runs (if you know which locks are held you run less risk of deadlocking)

I'm coming from a C background here, I imagine that the rules are different if you're using synchronised methods in Java.

Note that I'm not saying 'if you do this your program will never deadlock'. I'm saying 'if you do this you reduce the chances of a locking problem'. It's akin to setting yourself quality standards on your progamming (e.g. limits on McCabe complexity, avoiding certain legal-but-confusing language constructs) to avoid bugs.

This is a list that needs to live. If you come up with other tactics, add them on. Make sure all of your team agrees the rules and obeys them (and be aware that if they don't agree that a rule is sensible they'll almost inevitably break it).

Finally, find automatic rules for checking the rules you choose (e.g. you can prove that code never calls pthread_mutex_* directly, if that's what it takes).

Re:First things first

[mkawick]

Other fuzzy costs:

How many customers tried to use your software and found it unusable (lost sale there each time)?
How many customers have stopped using your software due to frustration?
How quickly can you get working fixes out the door?
Does adding new features cause a lot of breakage? This is where you lose customers.

Often the best way to measure the value of test is to look at return customers. Customers who are satisfied buy new licenses, recommend your software, or mandate it internally. This is where your business grows. If you don't have that, then you need to fix your development and the first place to start is test.

Re:First things first

[Anonymous Coward]

Having good, tested software might even cost the company money. Some software houses who charge big bucks for support contracts would lose money if their customers no long had to buy support contracts. And they would be less likely to buy the next version, if the current one works well. I can think of a large software company that is famous for this, with each new version purportedly being, "more secure."

Next, what's your salary for fixing the bugs? If it's not high, fixing bugs might actually be cheaper for the boss. (I'm purposely not mentioning customer satisfaction, as that can vary with wording of sales pitches and knowledge of customers.)

Re:First things first

[doublegeek]

My world is different -- mostly mid-sized (up to 500K or so SLOC) applications generally written in C/C++, with little to no dependence on RDBMS's. When we've used databases, we've generally kept the SQL stuff contained and well-separated from the main application.

Testing is always important, but when you're thrown into supporting an application that's been there for years (which was generally the case for me), you don't have the luxury of developing test code that's fully integrated with the application. So you write test cases -- vignettes -- that run the entire application on a set of input data, and look at the results to make sure they're what they're supposed to be. And you start building in automated unit tests into new code. It's infeasible to refactor the entire application to integrate automated unit testing, but we've had pretty good success building up large sets of automated vignettes that end up testing most of the applications' functionality. It's not perfect, but it does catch a lot of errors. And we instituted a policy that any new functionality or bug fix had to have a test case written for it before it could be incorporated into any production code. That also helped a lot.

How you do it depends on the application. Your case seems like one of the more difficult ones, since it seems you have lots of places where unpredictable inputs are injected, and you have all of the associated issues with security, robustness, etc. But for a well-designed application where external inputs are controllable, it's a lot easier.

Re:First things first

[Lashat]

This is the thread with the best and most useful information for you.

That said, I want to add that a good, experienced, professional quality assurance engineer would be a huge benefit for you. This person can go back in to the history of the recurring bugs and help determine why they are occuring. They can also assist you in determining if you have a need for further QA or test suites worked into your process. You can contract a QA engineer for 3 months and that would help you determine the metrics you need to measure to justify further expense.

Re:First things first

[MrMarket]

Cost savings may not be the correct metric. There is a larger question: "What is the acceptable quality level for this product (and does our development model allow us to achieve this level of quality)?" Improving the quality of your product may very well cost you more money. If your current mode of doing things prevents you from meeting your organization's quality standard, it's time to have conversations about investing rather than saving.

Re:First things first

[DamnStupidElf]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

The foolproof way is to write your software as a state machine that has well defined atomic transitions between states. If that's not an option, put mutexes on all shared objects and use a deadlock detection algorithm like the Linux kernel has for proving the (in)correctness of the locking operations.

Re:First things first

[stewbacca]

The fact that "test/don't test" is even up for debate explains a lot of bad software out there.

Re:First things first

[Mafia$oft]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

Simply using Valgrind's Helgrind is one way to do that - available only on the best operating system that money can _not_ buy, of course ;)

And the expert method here would be to Helgrind-run the unit tests themselves ;) (plus standard Valgrind memcheck runs)

But generally spoken, if people happen to have severe trouble getting their code free from race conditions, then that easily suggests that they have a plenty weird relationship with threading and should rethink their model (think isolated worker threads with cleanly passed _work units_ data and resulting "completed" notifications).

Having some 30 different threads in an app, all with liberal - and then not even properly locked - access to global data is __NOT__ the way to do things.

Re:First things first

[Anonymous Coward]

I no longer wonder why Ericson decided to create a whole new programming language for the job after reading your post.

Re:First things first

[Anonymous Coward]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

It's not easy but I have had good luck with pretty simple load generators and having the system put in random (from very long to zero) delays in the processes. Find lots of race conditions (short delays) and poor or missing interlocking (long delays).

I've had good results using a similar scheme, but with slightly more intelligent delays. The idea is described here, but the basic approach is to first set breakpoints on some subset of your memory accessing instructions, and then when those breakpoints fire you calculate the address which they're accessing and set a watchpoint on that address, and then stall the thread which hit the breakpoint for a few milliseconds. If the watchpoint fires then you have a race, and otherwise you clear all the watchpoints and run a bit further. Every so often, select a different set of instructions to install breakpoints on. That seems to be much more effective than just blindly inserting delays wherever, and it also gives you a pretty strong hint as to which precise instructions are timing-dependent.

Unfortunately, MSR, who originally came up with the technique, didn't release any code, but it's something which any half-way competent C hacker should be able to knock together in a few days, and I've put my implementation up at https://github.com/sos22/ndc.git; it's hardly robust, but it should get you started.

Re:First things first

[MillerHighLife21]

At a local university near here, they're IT department has setup an internal cloud using Eucalyptus specifically for that purpose. The generate Selenium tests and then fire up a ton of VM's to simultaneously run them to generate said race conditions. It's pretty slick.

Re:First things first

[HeadlessNotAHorseman]

This is exactly why there is a market for people like me, the business analysts. Slashdot's user base seems to be very coder-biased, I don't recall ever hearing of anyone identifying themselves as a business analyst here. But ideally it is the BA's job to liaise with the customer and find out what their requirements are (and what I mean by that is, what they actually need, not what they think they need), and then translate that into a requirements document that the developers can use to build the system. The requirements document then feeds into the functional testing, and user acceptance testing. The devs should also create a techncial spec, which describes how the system meets the requirements, and feeds into the unit and integration testing. In my experience BAs often have some level of involvement in the testing too, though ideally you would hire professional testers for that.

Re:First things first

[theshowmecanuck]

I suspect there are a lot of people here who are BAs but have gone the route via programming/development. I moved to this role several years ago from development. But I still make sure I at least play with the latest tech at home every few weeks to keep current on what is going on. I believe that makes me a better Business Systems Analyst. It allows me to short circuit silly requests before they actually go to far. We all know the scenario: the business says, 'well all it is, is one more field field to update (in xyz context), can't you just add it in to what you are already producing?"... Meanwhile in order to do so would take an extra month of design/coding/testing. (I know... an extreme example, but...). It is easier to spot the requests that smell bad, and take those to a development lead right away to get a better handle on things.

Re:First things first

[Splab]

Yes, but if we had time and money to do it the right way, we woulnd't be in this trouble.

And your hints are unfortunately useless - if you know how all objects interact you don't have the problem. Since we have the problem, we obviously don't know how they interact in all cases, thus the need for some outside program to help grind it.

Deadlock detection won't help, since race conditions are (usually) due to lack of locks.

Re:First things first

[theshowmecanuck]

Incidentally, if anyone out there has suggestions on how to reliably test for race conditions, please speak up.

Sure. Just write it correctly the first time. [BIG grin].

I don't know...

[Anonymous Coward]

But when you find out, let Steve Jobs know.

Charge him

[MrQuacker]

Charge him by the bug. You'll see how quick he'll make sure hes not paying to fix the same one twice.

Re:Charge him

[Billly Gates]

Then he could save money by firing him instead.

Why will it save money?

[SuperKendall]

You need to think very hard about exactly how any why this will save the business money. What do YOU think will be gained, for your software, from a technical standpoint?

Many people just think testing will make things better. But I have seen testing make things better, and I have also seen testing make software far harder to maintain and later to deliver.

What kinds of problems are you seeing that you think testing will solve? Do you see a lot of maintenance causing other errors? Do you want to do this in preparation for overhauling some part of the system?

You see bugs come back every 2-3 months, but is preventing that really worse than the overhead of maintaining and adhering to a test suite that all of the developers would have to do? That alone may not be enough of a reason to impose the overhead that testing can cause.

Since you are fixing bugs you have access to the codebase. Start a test framework yourself, of bugs you cover, and run it against the code day by day. See how often it catches issues, vs. how often you have to alter the tests because the code has changed in ways that are valid. Then you will have more real world metrics about problems solved vs. costs...

Re:Why will it save money?

[ls671]

> Start a test framework yourself, of bugs you cover,
> and run it against the code day by day.

I second that. It is amazing sometimes how you have to take the decision and just bill it on normal bug fix time or whatever task is assigned to you. Often, managers won't care about it as long as the average time to fix a bug or the time spent on your regular tasks remain the same.

It is harder to do it this way because you then have do it gradually. We even have redesigned application cores without letting the end customer know ! If we had told him, he would have been disturbed and worrying about it. For him, all we did was change requests and bugfixes. In this case, in was more complex this way because the old design had to be compatible with the new design for quite a while.

In the end, we managed to implement changes more easily, we could afford to charge the customer less while making better margins. ;-)

Re:Why will it save money?

[Fulcrum of Evil]

You see bugs come back every 2-3 months, but is preventing that really worse than the overhead of maintaining and adhering to a test suite that all of the developers would have to do? That alone may not be enough of a reason to impose the overhead that testing can cause.

What overhead? Fix a bug, write a test that exercises the bug first, then set up a continuous build server that shouts when a test fails. All the developer has to do is keep the pass rate at 100%, which is only overhead when they reintroduce a bug or rework the design (which is unlikely here). Way cheaper than waiting for someone to find the bug later on.

Re:Why will it save money?

[Anonymous Coward]

I've recently come to the conclusion that this sort of exercise isn't worth even starting if the company isn't likely to resolve the problem. If you're being told "we don't need to do testing" (or whatever), then that's not an engineering choice - it's a business choice. Unless you stand a chance of reversing a business choice, then you might as well forget about all of this, and either live with it, or leave.

I'm starting to think that either companies "get" engineering, or they don't. It's not hard to be a programmer, but it is hard to be a software engineer. Being an engineer involves making non-obvious choices between too much testing and too little, or what sort of tests for a given amount of investment, or whatever. On the other hand, being a developer involves doing what your boss tells you.

I you're being hired as a developer, then it's hard to become an engineer if your boss isn't up for it. If (s)he is, you've hit the jackpot, and doing some sort of study like this and justifying the next step of work is absolutely the thing you should be doing.

Re:Why will it save money?

[SuperKendall]

Well honestly I expected nothing to change there but it would be educational for the developer, so I figured it was worthwhile from that standpoint.

And there's always the chance something more could come from it which he would have started. Basically you can't lose with something like that - if you start small and don't fight the bureaucracy.

start small

[sugarmotor]

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

Re:start small

[ratbag]

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

This needs modding up since it is the "right" answer to the question - I've got no points at the moment.

Re:start small

[VirexEye]

+1 to moding parent up.

You won't get far convincing a product owner that you should spend months writing tests for the entire system.

Convincing someone that you should write unit tests for all new functionality to help guarantee the bug fix/new feature will continue to always work into the future is a much easier sell.

Re:start small

[Anonymous Coward]

If the owner does not get convinced even on a incremental start small plan, do it anyway under his nose. It's for your own good.

Re:start small

[Anonymous Coward]

Adding unit tests to code that was not designed for testing often requires refactoring -- a considerable effort and a source of bugs itself.

Re:start small

[Usagi_yo]

Best Answer by far for small non mission critical apps. If you can't justify the cost of testing vs the cost of a bug, then you'll never get funding for a distinct project for testing. When you do fix a bug, you invariably code review and write a test app against it. When it comes to internal apps, testing is typically overlooked and bugs are fixed ad hoc or work around become tribal knowledge. But even the tiniest mission critical application needs to have a test suite developed side by side with the app in question. Think of the team that let slip the bug that crashed the mars probe. Not only will that slow your career down, it will scar you knowing that your bug cost billions of dollars and years of work hours.

Re:start small

[Anonymous Coward]

+1 ... if the unit under test is reasonably testable, such that you can achive some degree of isolation from other interfaces/components/systems

Otherwise, exploratory testing might be more reasonable at this stage in the game. Legacy, mutated mish mash might have more side affects "by design" than a framework based scripted testing would reasonably allow or handle.

Re:start small

[Anonymous Coward]

Whereas integration tests don't have that problem. Don't limit yourself to one type of test.

Re:start small

[Anonymous Coward]

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

Exactly! One has to duplicate bug before fixing it. Do that with an automated test. Use the test to verify that you fixed the bug. In that way you have slightly increased test coverage and charged it to fixing the bug. After a while, you'll start getting less bugs.

Re:start small

[arth1]

Convincing someone that you should write unit tests for all new functionality to help guarantee the bug fix/new feature will continue to always work into the future is a much easier sell.

Not necessarily. You need to factor in the time for creating a framework for actually running the unit tests, and vetting the results (is a fail due to flaws in the code, flaws in the test (false positive), or irrelevant?).
And quite often you have to refactor at least parts of the code to accomodate introduction of unit tests.

Then there's the meta-cost of collecting and presenting metrics, and meetings with your boss(es) and developers.

Yes, tests are good. But not necessarily cheap to introduce, even in a limited scope.

Re:start small

[Anonymous Coward]

The problem is when you have lots of short term workers on the job they seem to re-invent the wheel each time. I've seen countless examples where some small function or subroutine has its own parsing and formatting code in it when there are provided already. You end up fixing one problem just to find the same one somewhere else.

Refactoring is something that should be considered definitely but likely in smaller chunks. Try putting together a coherent set of calls that you find define core functions with in the system. Once these are built and have been sufficiently debugged you start to implement the code and test that as a whole. Little by little or more as you see fit the whole project will start to fall into place. Make sure anyone working on the system will be able to locate where (what file) each function call can be found in and out line its importance in the header, a bit more work but it's like threading if one thread is unaware of a related value then it will not be able to anticipate it. You end up having a parsing error or something similar and having to look in too many places. Added benefits you can convey to your employer are it will be much easier to scale and add new features, compare your app to ones that are larger or more robust, don't promise that you will deliver something identical but suggest some features you feel are feasible, show how these features could help in terms of customer/client satisfaction, streamlining the company or expanding to a new client base.

Having stable and reliable code is like having a sturdy ladder it makes your work much more predictable ;)

Re:start small

[johnjaydk]

Excellent advice. Don't try to start a BIG, EXPENSIVE testing program. Have a look at the book: Working Effectively with Legacy Code by Michael C. Feathers

Re:start small

[msclrhd]

#include "myclass.h"
#include "assert.h"

void test_bug123()
{
        myclass a(10);
        assert(a.value() == 10);
}

int main()
{
        test_bug123();
}

---

How hard was it to create that test framework? You don't need something that is overly complex, just something that will pass if the test succeeds or fails if it doesn't. You can then improve the test cases/framework as you go along -- look at the Wine project for example. They didn't create http://test.winehq.org/data/ over night, it was built up gradually, starting with getting the tests running as part of the build and slowly defining a wine test API as needed.

Collecting metrics and reporting them should be done automatically, but not initially.

Re:start small

[Anonymous Coward]

Agreed!

Add a test project to the solution; write the test case to match the bug YOU are fixing.. write the fix (using your test case to verify the fix)... next time, let the developers fix the code when the test case fails.

Re:start small

[hopeless case]

I had the same reaction. If the OP is a developer working on this code, and has to fix some of these bugs when they come up, then he should start writing scripts that detect existence of the bug in the app, then fix the bug like he normally would. The next time one of his colleagues reintroduces the bug, his test script will catch it and he can point it out.

Bugs that go away by themselves, come back by themselves. Automated unit tests let you know exactly when they come back, and why (as long as you run them often, then not much will have changed in the code since the last time the test passed, letting you know that something in the last small change provoked the bug).

You don't need to write a massive suite of tests to demonstrate the usefulness of automated testing. Automated testing pays off in small increments as well. If the OP starts writing automated tests for the work he is doing, then the success of those tests will speak for themselves.

I think the main reason so few developers write automated tests is that it is a non-trivial skill that takes practice and time to learn. So the first task is to get good at doing it yourself. Then you can lead by example.

Re:start small

[Froggie]

Still, this is the approach I'd start with. It costs little and doesn't look like code change for code change's sake (which, from this sort of manager's perspective, is 'changing the code without improving the product from the user's perspective').

Also, give it a while and you should be able to demonstrate that the 'recurring bug' rate comes down, which is evidence to back up your claim that testing is a good thing to be doing and needs applying more broadly.

Re:start small

[cyberfunkr]

Take some time before starting this and think about how you are going to test everything. Set up a workable framework first. Otherwise you'll end up with some tests saved as text files, some as SQL queries, a few spread sheets, instructions on how to properly format exported data so they can be diff'ed against previous versions, etc.

You'll end up having a testing system as convoluted as the code you're trying to test.

Re:start small

[kwoff]

That's fine at first, but then your test suite gets bigger, while at the same time that more code is added and old code is left to rot in place. Other developers aren't so enthusiastic about keeping the tests up to date -- they have business needs to satisfy, after all -- so some tests start failing, but nobody fixes them. Your data sets get out of date when the design changes a bit and certain columns are hacked into or out of tables, which breaks tests that nobody wants to fix. All this time completely wasted writing tests that will rot, for code that will just get thrown away.

Re:start small

[SoftwareArtist]

Exactly. Any tests are better than no tests, and this takes essentially no extra time on your part. Every time you fix a bug, you need to verify that it's really fixed in any case. So verify it by writing a test case that can then become part of your small but steadily growing test suite.

WHY do you have to prove software testing saves mo

[dbIII]

WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?

Maybe it won't save money

[initialE]

Perhaps the app is just more important to you than it is to the company? First take your ego out of the equation, i guess, then make your case before the boss. Just sayin'. Also if you have a time sheet of how much time you've spent on this app so far it would be helpful in drumming up some numbers, plus a few scenarios where the failure of the app causes lost time for the company as a whole.

Re:Maybe it won't save money

[Anonymous Coward]

Mod up. As a current business owner and former developer I can look back at what was seemingly important to me at the time from the top down. The company didn't care and rightfully so. Developers get emotionally invested in what they are constantly sweating over.

The simple fact is that most of the time management knows what's good for the company and developers only think they do. Without metrics and provable numbers the argument will never be won. Unless you're also an executive, as a developer you must assume you are not privy to the financial reasoning behind why management/ownership chooses the directions they do. You may very well be right but you absolutely must assume they know more than you do about what's good for the company as a whole.

Chances are management has good reasons to make the decisions they do. Keeping the company progressing positively is their entire purpose and your specific project may simply not be worth their current focus. All you can do is bring them the data (in a robust and clearly defined manner), submit your opinions and then let them steer the ship. That's their job.

Then again, if your confidence in management's ability to make good company decisions is waning due to perceived incompetence, you might find more satisfaction elsewhere.

Re:Maybe it won't save money

[stewbacca]

I hate this attitude. Why should something be less important to me at work than something I would be willing to spend my own money on?

I think it is really lame that my home computers are far better than my work computers (I work for a software company). It would be beneficial to my company to invest in decent equipment for me, in that my product would be better. But no, instead, I get a business grade Dell shitbox to try and edit hours of video in Premiere and After Effects and the whole project ends up taking 3x as long (due to instability, lost work, rework and slowness) as if they just invest in a machine that I personally own at home.

But noooooo, they still get the $1.5 million contract and I get to churn along with my $500 Dell shitbox. For chrissake, I don't even get a widescreen monitor (let alone monitors).

Cost-Benefit Time!

[wild_berry]

We've had to justify creating auto test suites where I work.

Over the last decade our product has grown from one code-base into three strands, each with separate customer foci, and we've had a healthy amount of staff turnover so that there are still brilliant, creative and skilled people working on it but some of the original knowledge has left us.

We found* numbers to justify that automated testing of existing features, applied later will protect against regressive changes. Even where there are complicated features which were not modular in design, or which lack good interfaces, the tests have saved us massive amounts of time testing by hand. The real win is hidden under something we didn't realise until later: creating the tests have forced us to really document what the features are and how they work**, sometimes from a unit-test level, sometimes at the interface level and sometimes in a top-to-bottom vertical slice. Once you have a record of what your software does, in a computer which is skilled at remembering exactly and repeating exactly what some former staff member told it a couple of years ago, you have a decent reason to be confident that your bug fixes won't cause more harm than good.

*: ballpark figures / educated guesses / made up.
**: We favour working code over comprehensive documentation, until our agile team is reassigned to other projects or leaves the company.

Tough luck

[hardtofindanick]

In small companies most of the time it is not really worth the money to invest time and money in test suites and engineers. It does not make sense to operate in such a manner and is very frustrating but the goal of the owner is just to make money. You can start charging him by the bug, but then the way you are describing this is a trivial software so he will find someone else who will work the old way.

All it takes is one CVS idiot...

[Anonymous Coward]

With 25 developers, all it takes is one who doesn't merge correctly to cause old bugs to resurrect themselves.

Re:All it takes is one CVS idiot...

[tibit]

With 25 developers, you should have customary code reviews before things get merged into production.

Eventually you need a solution

[nanospook]

The metrics is a good idea. This will allow the owner to see what it costs him to not have good QA. However, eventually, the conversation will float to how expensive will it be to have QA? Like any security situation, you will want to find the happy medium where the trade off in time/money balances the risk. Given you only have a few hundred users you might wish to experiment by adding QA incrementally. Also, you might find why those bugs are coming back? Poor documentation? Or are developers not maintaining up to date code bases? Process is just as important as QA when it comes to fixing bugs.

Re:Eventually you need a solution

[Billly Gates]

Well his salary is a cost. If he spends 35% of his time fixing previous bugs then y x .35 is the cost that could be saved right there or better put to use developing software instead of fixing it.

Re:Eventually you need a solution

[Anrego]

Implementing a test suite isn't free.. nor is working testing into your process.

If he implements the wrong test methodology (I feel dirty using that work) he may end up spending 40% of his coding time satisfying test requirements, which ultimately only catch 1/4 of the bugs.

I would generally say some testing is always a good idea.. but too much testing can actually be a bad thing. Especially if it is very process and time intensive.

In other words, more analysis than "less bugs good" is required for something like this. Salary and dev time is definitely a factor, but there is a lot of other stuff to consider.

Re:Eventually you need a solution

[Thing 1]

However, eventually, the conversation will float to how expensive will it be to have QA?

We had a similar discussion a few weeks ago, where the gist was: if you're paying your programmers to write documentation and perform tests, then you're paying way too much for doc and test. This sounds even worse; this sounds like they're "paying their users" to perform the testing.

Well...

[Anonymous Coward]

How can you be so sure that it'll save money? If you need help to prove it, then it's likely that you don't know enough to make a significant change.

it won't save him money, thats your problem

[Anonymous Coward]

fixing bugs won't save him money unless your willing to recommend firing a few developers because it'll reduce workload. thank your lucky stars he's smart/kind enough to know this and say no rather then let you go to all this trouble then fire you because your not needed anymore.

Re:it won't save him money, thats your problem

[SanityInAnarchy]

I don't know about you, but I've never worked anywhere where the problem was developers with too little to do. If they aren't spending all their time fixing bugs, maybe they can devote a bit more time to actual enhancements?

Re:it won't save him money, thats your problem

[Anonymous Coward]

but how is THAT going to save money? the only way eliminating bugs saves money is to fire developers. i'm betting these guys are salaried like the rest of us so their time is already paid for. if on the other hand this guy has some other grand plan that involves turning that time currently spent fixing bugs into delivering a new feature that sells more product, i'm all ears.

he first needs a solid delieverable he can hold out like a carrot. slashdotters really do need to have their hands held on business thinking don't that.....

Re:it won't save him money, thats your problem

[scdeimos]

but how is THAT going to save money? the only way eliminating bugs saves money is to fire developers. i'm betting these guys are salaried like the rest of us so their time is already paid for.

Yes, the developers' salaries are already paid for. But their time is also already allocated to developing new features.

The more time "wasted" having to stop to fix bugs and then get back into the zone of creating, the more project schedules will slip. Delaying feature launches can have a negative impact on a company, particularly if those features are being developed under SLA/contract with a major customer - sometimes millions need to be refunded. There's your cost.

One thing I've not yet seen mentioned here - quite often fixing a bug also requires fixing data that's been mangled by it. Sure, you could fix a couple of lines of C#/SQL/favourite language and the bug is fixed... but you've still got hours/days of time getting the mangled data right again. Having a proper test suite helps you to catch bugs before they get out the door and cause data carnage. Even if you only write regression tests, at least you won't be repeating data fixes. Doing the same thing in front of customers time and again wears thin after a while.

Re:it won't save him money, thats your problem

[Anonymous Coward]

If the company is growing, the savings may not be from letting anybody go, but from hiring less in the future.

If management is more concerned with maintaining or growing employee base at the expense of productivity or efficiency, they had better know why they're doing it and have a plan... it's just the kind of thing a slinky competitor might exploit to get a foothold.

If you want a make-work project, join the public sector.

Re:it won't save him money, thats your problem

[Anonymous Coward]

Yes, the developers' salaries are already paid for. But their time is also already allocated to developing new features.

The more time "wasted" having to stop to fix bugs and then get back into the zone of creating, the more project schedules will slip. Delaying feature launches can have a negative impact on a company, particularly if those features are being developed under SLA/contract with a major customer - sometimes millions need to be refunded. There's your cost.

Agreed. And if management has to make a choice between losing millions or shelling for an extra body or two for a year or two, they'll hire if authorized to do so.

Between salary and benefits, or contracting fees, think along the lines of $50K - $500K, depending on the domain of expertise, skills required and local job market.

Doing the same thing in front of customers time and again wears thin after a while.

Agreed. With a patient and polite user, it wears thin. With abusive users, the company risks greater employee turnover and having to pay more to keep experienced employees.

Replacing employees can be difficult and expensive. Replacing your best employees, more so.

Re:it won't save him money, thats your problem

[JonySuede]

I've never worked anywhere where the problem was developers with too little to do

Where I work it sometimes happens and it is the worst thing in the world, being paid to no nothing is awful. And this happen because our management and business analysts suck but we the programmers kick ass.

Re:it won't save him money, thats your problem

[lennier1]

Happens to a former colleague of mine since he has started working now for one of the largest companies in the country.
Often the developers have already done the preparations and are ready to start but can't do so until the suits are finished with their flash games and finally give the green light (budget lottery, last-minute rounds of buzzword bingo, ...).

Re:it won't save him money, thats your problem

[Fulcrum of Evil]

So go read slashdot. When you get bored, install something new (like ruby) and play with it.

You are almost to the point of needing QA

[linzeal]

QA can be contracted out for 20 dollars an hour in house, I would suggest you look into that. It is a full time job for anything but the smallest of code bases.

Re:You are almost to the point of needing QA

[Jane Q. Public]

QA is not the same as testing, in this context. The kind of tests he is referring to tell you whether the code is broken before it ever gets to the QA stage.

Baby steps

[sjames]

Start off easy, you must have test cases for the bugs you fix (so you'll know they're fixed and some sort of script or procedure you used for that testing. Just hack together a quick script on top to run them in succession as part of your regular duties fixing bugs.

Now, put them somewhere where other developers can access them. Mention it around the coffee pot.

That's not exactly comprehensive testing, but it's a start and should only take a few minutes. You could argue that you had to do it anyway as part of your regular duties, so it cost nothing. With any luck it'll catch on and prove itself useful.

the 200 Million answer

[Anonymous Coward]

It is not just that testing is needed, when bugs are found, they need to be fixed (security bugs with priority).
I have personally seen bugs that were not deemed 'need to fix' that went out with the following fallout (this is NOT a joke or embellishment):
1. possible loss of life - as in death of a real person
2. loss of revenue that was supposedly in the millions
3. death of company
4. loss of accounting revenue in the multiple thousands

I have been in the industry long enough to see these things. Per NDA, I cannot give details. But this is real and depending on the company and its politics, could happen in front of your eyes.

- SPT

There's your problem...

[SanityInAnarchy]

...using CVS.

Re:There's your problem...

[Anonymous Coward]

Yeah I know. Real developers copy .zip snapshots over using Explorer!

Re:There's your problem...

[SanityInAnarchy]

No, real developers use real version control software. Even SVN would be an upgrade, let alone Git, Mercurial, bzr, etc.

Re:There's your problem...

[Anonymous Coward]

GP here. I was using Darcs long before Git and Bazaar even existed. But thank you for stating what we all knew in 2002 -- about SVN's superiority to CVS.

Re:There's your problem...

[Vegard]

I have seen the same happen with SVN. Trust me, people who don't understand version control systems will *always* find a way to fuck up! :)

Re:There's your problem...

[pinkushun]

real men just upload their important stuff on ftp, and let the rest of the world mirror it

FTFY

Re:There's your problem...

[Anonymous Coward]

Even people who have been using version control for years still screw up. Any VCS that doesn't show you a three-way diff on a conflict (what was in version A, what was in my file, what version A was changed to in version B) is simply asking for a developer to not figure out what was changed that caused the conflict. I can't count the number of times I've had to entirely throw away a merged file and redo my changes from the start thanks to loops being merged in the wrong places (oh look a brace this must be where the code goes), conflicts over hundreds of lines (when I first started fresh out of college, I had my own indentation style -- entire files conflicted whenever there was a change), and so on.

A test suite doesn't have to be expensive.

[julesh]

If all you do to make the test suite is add one test each time a new bug is discovered, you'll probably find it doesn't take much longer than fixing the bug and retesting manually anyway, and over time you'll build up a reasonable suite. Yes, the benefits won't be instant, but you can do it without much outlay of time (or money) and therefore it's easier to justify to the product owner.

In case you haven't, you should read Robert C. Martin's book Working Effectively With Legacy Code, which is full of advice on just this kind of situation.

Re:A test suite doesn't have to be expensive.

[gbjbaanb]

unfortunately, you'll find that your test suite becomes bigger and more costly to maintain that the actual software its supposed to test.

I think there are better ways to achieve quality software, and while unit tests do have their place, an over-reliance on them is just too much overhead to justify.

Re:A test suite doesn't have to be expensive.

[OFnow]

If a test suite is costly to maintain it is defeating the purpose (because test failures don't indicate
code bugs), and something is wrong. You have to find a way at reasonable cost to eliminate
the bogus failures. Testijng a GUI and testing the output of multithreaded code is problematic,
good solutions are hard to find.

Take the other side of it: if you code for easy testability your code does look different and
the tests you write will work perfectly forever (and only fail if the software being tested gets
broken). But you won't get there if you don't start by writing the test driver and then the code,
as unless you start right you won't write easily-testable code. BTDT (both ways).

       

It saves money if the data is worth something

[finwind]

If the data is worth anything, you can always argument that software that has not been tested thoroughly can mess up the data and then the worth/value of the whole system collapses. Yes, there are backups, but will the backup schmea work if the software gradually corrupts the data? And will the customers suffer damage during the downtime that you try to debug/correct the bug (that will eventually occur) ?

Just do it!

[Anonymous Coward]

Why are you asking for permission? Most programmers today know that there is no point in asking for permission when it is much easier to ask for forgiveness after the fact ... if they ever find out and care.

Just start in one corner, create test cases for exactly those bugs that you don't want to see again. Then add incrementally whenever you have a couple of minutes to spare.

Just do it!

Re:WHY do you have to prove software testing saves

[Rosco P. Coltrane]

WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?

Rest assured that Chinese companies test their products precisely *because* it saves money. If it didn't, they wouldn't; they don't have a commitment to good quality for its own sake.

An untested product leads to high rates of return (lost money), customer dissatisfaction and brand name damage (HUGE loss of money in the medium to long term). Nobody puts up with bad products anymore. Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.

Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

Kids these days

[oldhack]

I don't have a solution to your problem, but I can tell you the shit only gets worse.

You're welcome. :)

Wrong Question tag appropriate

[wrook]

The better question to ask yourself is, "Why am I still working here?"

If I were working on that team, I would walk up to the owner and tell him straight out, "You hired me for my ability and experience in software development. It's my job to give you the most customer satisfying functionality for the money. You've seen my credentials. You've seen me work. Stop second guessing me and let me get on with my job." But my employers have historically hired me because of my experience in these kinds of areas.

Asking, "How do I prove to management that X practice is a good idea?" is precisely the wrong thing to be doing. Do you *know* the practice is good? Do you have experience implementing the practice? Do you know when it is appropriate and when it isn't? Do you know how to implement the practice in the particular circumstance you are in? If not, who on your team does? If nobody, who will pay for your learning experience in the probable case that it takes time to get things right?

I'm not telling you that you shouldn't be doing these things. But if the guy paying you is unsure of your ability to pull it off and you have no prior experience, I highly recommend relocating to a job where you can get the experience.

Re:Wrong Question tag appropriate

[Rosco P. Coltrane]

Yeah, walk up to your boss and tell him you know more than he does. Great career move...

Have you heard of diplomacy? You may know more than he does, but he doesn't want to hear it. Instead, you should convince him that what *you* want to do is *his* idea.

Re:Wrong Question tag appropriate

[Anrego]

That's a little.. extreme. That kind of trust isn't assumed with credentials (at least not in my book) but builds over time.

Not only that but the boss may (and likely does) have a broader knowledge of the company and it's objectives, and any time you're talking about spending more money I would expect to have to at least show the boss a few power point slides or something. At the very least, having to convince management that an idea is sound is a good exercise, and may cause you to look at things you would have conveniently ignored.

Re:Wrong Question tag appropriate

[Anonymous Coward]

Instead, you should convince him that what *you* want to do is *his* idea.

Achievement earned: verbal cocksucker.

Re:Wrong Question tag appropriate

[mcvos]

If you need to use that kind of deception to do your job, maybe you're better off looking for a better job.

Re:Wrong Question tag appropriate

[Anonymous Coward]

The question is what the distribution of reasonable managers vs. ego-maniac managers is. Posting anonymously expresses my opinion in the matter (and also my opinion about my current managers, who have just fired someone because they didn't like what he had to say).

If the probability of getting into the same kind of environment is 80% or above, what's the point in leaving your job?

I have the same opinion about unpaid overtime (everybody expects you to do it), no testing (most companies don't invest in quality because the managers are ex-sales that are used to sell crap and not take responsibility), over-committing, ridiculous dead-lines, etc. etc. This is the face of the industry, there's not point in looking for another job in the same industry, because it's exactly the same or worse in other companies.

The only way to deal with it is to put a mental barrier between yourself and your company - the job allows you to have a life, but it does not have to reflect your own values and opinions.

Re:Wrong Question tag appropriate

[tibit]

I have subordinates and I actually like learning new stuff from them. So please don't make all bosses stupid, self-centered jerks.

Re:Wrong Question tag appropriate

[gbjbaanb]

it came across like the guy himself was a stupid, self-centred nerd-jerk.

I find managers are often just ignorant. (which isn't a criticism as they tend to know more about other stuff, but still have some idea that they should also know everything about the code/systems their staff work with and have too much input into the nuts and bolts when guidance and assurance are what's really needed from the boss).

Re:Wrong Question tag appropriate

[sosume]

obligatory big bang theory quote incoming ...

Sheldon's Mom: Now, you listen here. I have being telling you since you were four years old, it's okay to be smarter than everybody but you can't go around pointing it out.

Sheldon: Why not?

Sheldon's Mom: Because people don't like it. You don't remember all the ass kickings you got from the neighbor kids. Now lets get cracking. Shower, shirt, shoes and let's shove off.

Sheldon: There wouldn't have been any ass kickings if that stupid death ray had worked.

Re:Wrong Question tag appropriate

[fishbowl]

Assertiveness can be risky but it can also deliver huge rewards. Advice: don't frame anything in terms of problems.

Re:Wrong Question tag appropriate

[JonySuede]

unpaid overtime (everybody expects you to do it)

My first boss, an old wise man, told me: If you respect yourself never work unpaid. I never did unpaid overtime and I never will. I value myself so I value my time. You have no self respect, you should grow a spine.

Re:Wrong Question tag appropriate

[JonySuede]

Let me add that a " should go after the : and another one after unpaid.

Re:Wrong Question tag appropriate

[mcvos]

There are reasonable managers out there. There are companies that respect their employees. When I look for a job, I don't take just any job, I take a job where I think I'll enjoy working.

Of course many good, self-respecting programmers are looking for those kind of jobs, so maybe you wouldn't have much chance at landing a job like that. On the other hand, hopefully it means that the egomaniac managers will end up with crappy spineless programmers and their companies will go bust. Tough for you, but hey, apparently you're equipped to deal with it. Right?

It's all about cost

[morbingoodkid]

The problem is you need a software (bug control system) already setup to prove that it is important to have a testing framework.

But the calculation I would use goes something like this.

Needed
1. Number of bugs fixed.
2. Number of bugs returned.
3. Time added to original bug to fix additional requirements
4. Time to test properly.

Cost factor for bugs not tested properly (time to test)/(time added to original bugfix)

This would be your cost of not having a testing framework. Work this out over the period you would like to aggregate the testing framework cost.

Say there was 30 bugs that returned
If took you about 60 hours to fix those 30 bugs.
It would have taken you 20 hours to test the bug fixes properly.
You will get
(20/60)=0.3 (30 %) reduction in cost.

To calculate if it is worth the companies money to implement the testing framework.
0.3*(cost to fix returned bugs over 2 year period) if this value is less than the cost of the framework it is not necessary to setup the testing framework.

Honestly even on the simplest system it always works out cheaper to have a proper testing framework. But I understand your frustration to bring this concept under the eyes of the managers are quite difficult.

The easiest way is to try and calculate a factor of cost savings and bring it to their attention.

Re:It's all about cost

[fishbowl]

+1 Jira

I can't believe the incompetence

[Billly Gates]

No QA?? Sadly, I see many slashdotters in similar situations. How unprofessional and arrogant have the bean counters become? I have been out of I.T. full time for many years but times were different earlier last decade. Any manager who refused to check the quality of the work used to be fired. Are you really managing if you wont even do basic QA?

Even fast food restaurant managers do QA and have district managers nail them on satisfaction and statistics of food ingredient use and statistical sampling.

Time to update your resume. From what you described it sounds like a cheap work environment if they have college interns writing critical software. Quality is free and any good manager knows this which saves money. Six sigma started out in software design before being a big hit on the factory floor.

Re:I can't believe the incompetence

[Opportunist]

College interns writing mission critical software isn't as far fetched as it seems. Especially if it wasn't meant to be but just "developed", as software often does. Allow me to give you a piece of anecdote "evidence".

The year is 1993 and little Opportunist is working as an intern during the college holidays. Instead of stuffing envelopes as usual, the company found out that this guy can do "stuff with computers" and we have those boxes that record where our traveling salesmen go, maybe he can get the data out of them and into a more meaningful format than the software we got. Opportunist reverse engineers that format, writes a cute little piece of code (IIRC it was even originally in Access Basic) and finds out after two days that he's done. Back to stuffing envelopes? Not if I can fight it!

To make a long story short, I kept suggesting features and adding them to it to stay out of the assembly line hell for as long as I could and eventually we ended up with a tool that could do the travel plan for a salesman. Yes, they didn't have that. Their people had to do it by hand and it took them always almost the whole Friday to get a travel plan down for next week. The system was able to learn and adapt and it was pretty good (for my knowledge then, today I would toss such a piece of junk before letting the guy writing it finish it).

Now human nature invariably struck. My boss boasted how he can now send his salesmen around a day longer since they're not tied down with planning, the bigwigs came flying in and a month later I received a call that I have to make a few changes that $headoffice wants, because now the whole company should be able to use it.

This software (or rather the successor I wrote in C++ Builder a little later) is still being used by a rather big company to send around its traveling salespeople. And we're talking about a make-or-break piece of software, if these people do not travel and visit the right people at the right time, the company will lose serious amounts of money. All that hanging on an application a college intern wrote almost two decades ago. Today I would write a LOT of it differently, knowing a lot more about algorithms and efficiency (Fortunately they didn't care that the program took almost 30 minutes to complete on the ancient 486 machines they used, they were used to IBM and overnight calculations), and I am very, very sure it has some very, very nasty bugs that rear their ugly heads at the worst possible times because back then I had not the slightest idea about sensible testing.

But, and that's the point, it still beats the alternative: Not having anything at all, doing it by hand and hence relying on human ability. Even given the bugs, I am sure it makes fewer errors than human does. And when I compare my hour rates then and now, I can see why they wouldn't hire the back-then equivalent of me (i.e. someone who "really learned" how to do it). The intern does it well enough and he costs about a tenth.

Whether maintenance is manageable or tenfold (because, well, easy maintenance was not really a focus for me when I wrote the code, I never had to maintain code before so I didn't know just how nasty it could be and how important it is), is a different matter.

But it was cheap! And that's what counts! Ask your manager if you don't believe me!

Re:I can't believe the incompetence

[Anonymous Coward]

College interns writing mission critical software isn't as far fetched as it seems. ...

Further on using college interns. Our engineering company maintains a moderately complex data analysis tool for a customer--one analysis run shows the user a couple of thousand plots to guide the user's modeling decisions. We never had test code, just had our sharp engineering analyst run some ad hoc tests when he added new features (he's a good journeyman programmer, but the first to admit that he's not a computer science whiz).

After an embarrassing bug crept in, we had a summer intern write a couple of test programs to look at some simple aspects of the application--just to make sure that basic functionality wasn't broken. It's been very helpful for very little cost.

I'd suggest trying this low cost approach first to see what the payback is--before investing heavily in some comprehensive test code.

It's ready now!

[hcs_$reboot]

how to convince the owner?

Replace the owner's default homepage with this Slashdot article.

Hmm ...

[lennier1]

Send the suits one of their cherished Excel sheets?

    ((downtime duration) * (customer's lost income per hour) * (customers))
+ ((bugfix duration) + (rollout duration)) * (hourly developer salary)
+ (overhead costs to appease angry customers)

Paying technical debt.

[SanityInAnarchy]

As others are saying, if you can get hard numbers, get them. Nothing like actual evidence to back up your claim.

However, there's a more important reason for doing this which is harder to find solid evidence for -- the concept of "paying down your technical debt." Often, we hack software together relatively quickly, without fully testing, etc -- every time we do that, we're effectively borrowing against the future stability and maintainability of the product, or realistically, against our own debugging and refactoring time in the future.

Like all debt, it has interest -- the longer you let it fester, the worse of a mess it's going to be. So like all debt, it makes sense to pay it down when you can -- a little refactoring now can save a lot of headaches later.

So, it may be difficult to get hard numbers -- though again, those are better. But if you can't do that, sell it as an investment. Invest a little time right now at least putting the automated tests in place, and maybe start with just some regression testing. Twenty minutes writing a regression test will almost certainly save you an hour the first time that bug appears -- and those are pessimistic numbers (though also made up on the spot -- find your own actual facts!)

Re:Paying technical debt.

[Mr Z]

Well, one could also start with a more narrow notion of "regression tests." Specifically, whenever somebody fixes a bug, they should also write a set of short tests that should try to expose the bug they just fixed and add them to the suite. After all, they need to ensure they fixed the bug, right? Don't worry about retrospectively writing tests just yet for everything that already works (or mostly works). If nothing else, such an approach would protect any regressions among the existing set of bug fixes. (ie. "I fixed that, why did that bug come back?")

Aside from that, for any new code that goes in, presumably the person writing the code has at least some quick and dirty tests to make sure the new functionality works at least a little, right? What cost is there to checking those in? It may not be the most rigorous test suite, but it's something.

Re:Paying technical debt.

[qwijibo]

I'm pretty sure that business schools teach people that you can borrow against the future forever. After all, if go out of business, the interest you owe to the future becomes irrelevant. If you've moved onto another job by the time you have to pay the piper, you can claim success while your successors are the failures for not moving faster.

From a business standpoint, russian roulette is the best possible business model. What other gamble offers you an 80% chance of winning? Business schools weed out the analytical types who ask about the other 20%.

Unfortunately, testing is one of those things that you're never going to convince anyone will save any money. If you've always been doing it, you can't prove otherwise. If you never did it, how could you have made it so far if testing was so important? I'm all in favor of good development practices, but there's no way to make a business case for trying to achieve the stability and durability of a pyramid when you have the budget and manpower for a shanty town.

Test your own code?

[Old Wolf]

I guess I'm missing something, but don't you test your bug fix when you write it?

You said that your bug fixes "come back", it sounds like you are cutting some code and sending it off without testing that you actually fixed the problem?

Re:Test your own code?

[pinkushun]

Some times a scenario slips through, like testing on a machine without the required dependencies, or on a low-privileged user account, against a different versioned data source, or on a box that was used in last week's ritualistic goat sacrifice.

The "but it works on my machine" syndrome is a common occurrence.

Re:Test your own code?

[Anonymous Coward]

I assume you are using source control? Either you didn't fix the problem in the first place, or you only fixed a single shipment (branch) and
the fix didn't make it into the main code base (trunk). Do you have a single code base that is sent to all the users, or does each user get
a custom version of the code?

I would follow the advice of others and start your own test suite. Start small and keep adding cases. No need to ask permission.

The other possibility is that you are asking for more. Are you asking the boss to implement a policy that all other coder have to follow?
If so, how do they feel about it? Maybe you need to get their buy-in first.

Or, are you really asking for money to buy an expensive automated code testing tool? If so, start with your own home-grown test suite.
As it grows (and you can demonstrate value added), then go ask for money.

Re:Test your own code?

[PhilHibbs]

Never get a programmer to test their own code! Only someone else can test code. Programmers, just like humans, are blind to their own mistakes.

In the simplest of cases, the programmer might have fixed three out of four instances of a problem, and only test those three instances because those are the only ones s/he thought of. At least with a fresh pair of eyes you get a chance of catching the other case.

Answer: Write a test and see!

[VortexCortex]

#!/usr/bin/perl -w
#####################
# File: money-test.pl
# Desc: Tests if Money is Saved.
#####################

use strict;
use FileHandle;

# This Should Save Money!
sub saveMoney {
    my $tout = new FileHandle( "> test-output.txt" ) or return 0;
    $tout->print( "Money\n" );
    $tout->close();
    $tout->open( "< test-output.txt" ) or return 0;
    die "Lost Money!" if ( <$tout> !~ /^Money/ );
    return 1;
}

saveMoney() or die "Money not saved: $!\n";
print "Money Saved.\n";

Once every two-three months?

[Anonymous Coward]

How do you prove (automated) software testing saves money? Many factors are involved. A test suite and unit tests can cost a lot of time to set up. You may very well find your system wasn't designed to allow automated testing and would require refactoring to allow for automated testing. How much time would you spend on setting up a test suite? How much time would it save? (Keep in mind right now you may not be doing manual tests- there is added value to introducing such tests as well). One factor is how long does it takes to re-fix a bug if one comes back every two-three months- some bugs take a bit more, some a bit less, but I find handling two (unknown) bugs a day is typical. Another, more important factor to take in account is how much damage can potentially be done by bugs. Depending on your system this may range anywhere from lost entertainment value in a video game to a plane crashes in air control systems. That is to say, in some systems, automated testing simply matters more than in others.

Just do it...

[rastoboy29]

Just do it and show him the tests running.  It always impresses people to see a bunch of automated tests happening--it makes it more intuitive what the value is.

Anybody can understand how that can save valuable human tester time (particularly if that human is expensive and talented).

Re:Just do it...

[benjamindees]

Anybody can understand how that can save valuable human tester time (particularly if that human is expensive and talented).

Well, wherever you live that might be true. But in the US, home of gangster-capitalist-accounting-fraud-as-a-defense-against-the-social-democratic-kleptocracy, showing your boss something you've automated would just make him think "Why did you waste time doing this when I could have outsourced it to a shell-company in which I have a 49% stake and hired some Indians to write it and taken that new tax write-off for creating jobs in a gay woman minority owned software outsourcing firm. Now I'm going to have to track it as a capital expense, ugh."

Re:Just do it...

[gabebear]

You really sound like you need to quit your job. It's not like that everywhere.

Re:Just do it...

[Lennie]

It isn't very impressive if it hasn't found any bugs. Then it's just plain boring.

You see...

[benjamindees]

There's this thing, called 'specialization'. You work for a small business somewhere in the Western world, so you probably have never heard of it.

Basically, the way it works is this. One person does one job. Another person does another. Everyone sort of agrees ahead of time what the jobs will be, and how they can interact in the most efficient and least obtrusive way. That way, you don't end up with five people on a 'team' spending all their time either in meetings or overwriting each other's work.

For instance, one person might fix bugs. Another person might test the software. Each person can 'specialize' in one relatively small area of 'special' knowledge and skills and not have to worry about everything his co-workers are doing. The person fixing bugs can know which bugs have been fixed, and spend time thinking about how the software should evolve in the future in order to fix or prevent other bugs. The person testing the software can spend his time creating automated tests, finding new bugs to be fixed, and double-checking the work of the people fixing the bugs.

Watch out, though! Because there's this other thing, of which you are probably well aware, called the 'pointy-haired boss'. Pointy-haired bosses don't like 'specialization' because they are fairly dim-witted and actually enjoy going to meetings and re-doing the same work over and over again. So, if you do decide to 'specialize' in your work, you should probably have a few websites lined up to browse and look like you're staying busy during all that extra free time you will soon have. Good luck!

Re:You see...

[Anonymous Coward]

The problem with specialization is that it limits the quality of your software to the quality of the worst person on your staff. This is why sweatshop-made software is so very very bad.

The best quality software is made by a smaller team of multi-skilled experts, herded by someone very good at herding. Yes, these people are expensive and not very common, but if you want quality, that's what you need.

IT (Project) Management 101

[Anonymous Coward]

Calculate the statistics regarding the costs that fixing the bugs already takes in your outfit (such as hours taken, damage recovery costs caused by impacts of bugs etc.) and then calculate the capital costs for setting up the processes, procedures, assets etc. for properly testing software and estimate how many bugs it could have prevented by examining old data and working out what technique, process or procedure you're planning to implement would have caught said bug(s).

Deming...

[Aiwendil]

Get him to read anything written of/about Deming about what is wrong with current management I'd say.

The book "Dr. Deming: The American Who Taught the Japanese about Quality" by Rafael Aguayo is probably the easiest to grasp for most people (and is a book I would recommend to pick up and read in general anyways).

Re:WHY do you have to prove software testing saves

[Kjella]

Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

Didn't you just answer your own question: extensive testing on top of proper design and good manufacturing practices

For the first I'll just quote Knuth: "I have only proved it correct, not tried it." And if you send piles of shit to testing/QA and expect them to be the impenetrable barrier between you and the customer, you're equally deluded.

I've found that for anything that should last a while and be stable, there should be a test case. It's so easy to subtly break something. However, I've found writing a proper test suite that deals with databases, network communication+++ and not just the application itself is pretty hard.

Re:WHY do you have to prove software testing saves

[Opportunist]

Because it is. Do you know the term "bananaware"? It's used internally by a huge German company. It means "letting the goods mature at the customer's (just as you do with bananas, picking them while green and letting them mature during transport and at the customer's shop)".

At the customer's shop, and at his expense, of course.

The core reason is that people got so used to crappy software that they don't even expect it to work "out of the box" anymore. It's also trivially easy today to replace broken software. No need to send it to the shop for replacement, we'll send you a patch via internet. You do have internet, right? You don't? How do you exist today?

Testing or no testing is only a matter of money. A product only has to be good enough to keep enough customers from throwing it back at the maker, or rather, as long as the cost from RMA process and customer complaints are lower than the cost for testing, no testing will occur.

Speak their language

[BESTouff]

However you prove the need for a test suite, express it on a shiny powerpoint. That way, even if the people to whom you'll show it are techs, they'll be able to pass it to their PHB and have him understand just what's needed: allocated time slots for tests.

Don't do that !

[eulernet]

The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money.

No, it will NEVER save money nor time. It will just improve quality.
Testing requires investment, and there is no ROI (Return On Investment) on it, except that the quality of the software improves, since you are able to avoid regression bugs.

In my company, we spent a lot of time (several man years) writing tests to strengthen our software, and I can assure you that this was very expensive, and I still believe that it was a waste of time and money (disclaimer: I worked onto writing the tests), since this could be done in a cheaper way.
In our case, we have a 7 years old project, and it was very difficult to write tests, since the code was not designed to be tested, and thus not easily testable.

Just putting the tests in place will probably require a large amount of time and effort.

If there is no development on your software, but only bugfixes, then it's better to use the cheaper approach of writing tests that fail, as somebody suggested above.
If the cost to write a single test is too expensive (in effort/time/money), you'll have to live with the current situation.

If there is still some development on the software, it's good practice to start adding tests on the NEW parts of the code.
Forget about the legacy code, it will require too much effort.

Re:Don't do that !

[Anonymous Coward]

You can get a positive ROI if you have the metrics to show a concrete reduction in lost money due to bugs that the test suite would have caught.

Re:Don't do that !

[Mr Z]

Lost money can come from multiple places:

• Man power spent fixing bugs. This is perhaps the most obvious, but it is unpredictable since you, by definition, don't know how many bugs you're going to have. However, you should be able to look at your defect rate versus time to get some idea.
• Man power spent writing tests. This is completely under your control, since you can decide simply to not write tests. Or you can decide to write copious tests. Or, anything between.
• The cost of lost sales due to poor quality. If you ship a total turd, your existing customers will start shopping elsewhere, and they won't send new customers your way. If you ship high quality product, though, that too ought to spread via word of mouth.
• The opportunity cost of lost sales due to product delays. If you ship sooner, you may get some critical customers that you'd miss if you shipped later. Of course, this breaks two ways:
  • One is that you delayed your product by being stuck in the "infinite polishing" phase. You have to be careful to avoid gilding the lily.
  • The other is that the development process itself got bogged down under the weight of its own bugs. Joel Spoelsky mentions what Microsoft called the Infinite Defects Methodology in which coding to get features took precedence over fixing bugs, and the product was delayed significantly by the weight of its own bugginess.

As you can see, it's not really a simple "testing saves money". But, laying all the pieces out there and showing how what you propose affects all these aspects might go a long way to getting at least something in place for a testing methology.

Metrics...

[theVarangian]

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

I actually had to write a paper on this subject for an SQA class I took. The result among other things was (as if we didn't know that already) that PHBs respond to numbers and little else (except maybe Blackberrys) and a scary percentage of them think software testing isn't worth doing beyond weeding out the worst of the bugs because it offers a low return of investment. To convince your PHB you have to gather metrics. How much does an error cost if that error makes it undetected into a marketed product? How much would do save if we caught it early by testing? How much of a reduction in the frequency of, say... buffer overflow or SQL injection bugs, etc... have you achieved by implementing software testing in some other project? If you don't have any SQA enabled projects to fall back on you can try to cite case studies or CS papers. Case studies are probably better since they are less theoretical and not loaded with a huge amount of statistics. Keep in mind that the level of SQA that is justifiable for projects varies according to their nature. If this is an internal product never used by external parties you can get away with much less SQA than if it is used by external customers. Metrics gathering is also a key component in pretty much everything else SQA related, root cause analysis, test coverage analysis, determining code coverage, deciding on where to focus tests... the list goes on and on.

TIMEZONE SLASHDOT

[Anonymous Coward]

I happen to read Slashdot at night when the sun rise in the US.
Now They are making a good move, somewhat like GOOGLE's targeted ads. Stories as per geographies.
Thanks slashdot.

Develop a mathematical model of costs

[Paul Johnson]

Basically you have to develop a mathematical model of the costs of the current situation, and compare it with a mathematical model of the costs of using tests. As part of this you will have to produce a plan for introducing tests, with the costs for each step itemised. Use the best numbers available, but don't worry if some of those numbers are "best guess". Just don't try to hide the fact. Put both models in a spreadsheet and come up with a number for how long it will take to recoup the initial investment (break even). Don't forget to discount future cash flows. In MBA-speak this is known as a "business case".

Re:Develop a mathematical model of costs

[Anonymous Coward]

Someone when to college too long. How many of these models turn out to be accurate?

Re:Develop a mathematical model of costs

[Paul Johnson]

Someone didn't go to college enough. Remember: "All models are wrong, but some models are useful." Any model of the future is going to be inaccurate to some degree, maybe a lot. But any model is better than "I just know that...". Even at the worst, it provides a framework for the conversation about the costs and benefits. Besides, sometimes when you plug in the numbers you find that the difference is so great, the debate is over. But you have to run the numbers to get there.

One bug 2-3 months

[ArIck]

One bug every 2-3 months. How long does it take to resolve that bug. An estimate of 4-6 bugs a year is not that severe of an issue and like the owner is better off without the extensive bug testing.

Re:One bug 2-3 months

[Shooter28]

One bug every 2-3 months. How long does it take to resolve that bug. An estimate of 4-6 bugs a year is not that severe of an issue and like the owner is better off without the extensive bug testing.

How many months are in your year?

Re:One bug 2-3 months

[nedlohs]

2 * 6 = 12
3 * 4 = 12

How many are in yours?

Re:One bug 2-3 months

[Hognoxious]

From the data given it would appear to be somewhere between 2x6 and 3x4.

A good approach?

[gedeco]

I think there might be something wrong with the problem analysis. You should consider why a bug does reappear. Wrong version in Production? Or as you stated not enough tested.
1) First of all, You have to make a inventory of all existing bugs in the application.
2) you need to describe the possible Workarounds. Here you can already start calculating the estimated cost of each incident provoked by a bug.
3) prioritize the bugs. The more urgent, the more they will cost the bussiness by not functionning and resolving time
4) Focus on some Quick wins. bugs who doesn't consume much time and money to solve.
5) Keep in mind that having a good Workaround is sometimes better
6) Make a bussiness plan. Explain the benefits versus the money lost (consider lost worktime as a los off money)would be cheaper to redesign the app.

Managers are willing to pay for improvement, but they have to be convinced. The only language they understand: MONEY.
Briefly you talk about the money you loose doing nothing, versus the Return on Investment (ROI) doing something.

bug regressions

[rhendershot]

at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite.

Many have already mentioned that you need metrics if you want to prove that a test suite would be cost effective. You don't really say if that's to be automated and that's harder. You also don't say if the intent is module unit testing or integration testing or some other slice, nor do you say what technology the system is built around. 10 years probably would be VB6 and there was a unit testing framework a few years ago available but the tool-set is decidedly sparse AFAIK.

Anyway, the statement about bugs returning caught my eye. Perhaps they really are a regressive bug but take a close look at that idea. To illustrate suppose the complaint is that "a failure was not logged to the logging file". This could be *caused* by, among other things, a misconfiguration of the logging system, a missing logging statement within the code that handles the failure, a locked logging file which could not be written, use of a module that disabled logging, or which did not re-enable logging, and so on.

Since these are all discreet sources they really are different bugs. It's unlikely (disadvantageous actually) that one test would find re-emergence of all of these even though they result in the same basic problem.

The point I'm hoping to make is to be certain you really are identifying the same bug as haunting you not just similar bugs and be sure you can faithfully relate that to your stakeholders.

And if you can make that case then it should be a no-brainer that they'd *not* wish for their teams to spend time and their resources re-doing the same thing. At that point the question becomes "What's the best way to insure that we don't?"

Sometimes it comes down to better documentation of the modules being used or training of the people involved. It's very hard to quantify the cost of training folks to create useful tests that don't actually increase costs (a lot). Also hard is predicting the point of break-even where all the costs in training, tools, and coding (and testing) the tests have started to return enough that the net starts to be positive.

We often seek perfection but removal of all regressive bugs is probably not the best first step. Another clue I got was that these bugs are found late in the process. In any case, perhaps better defensive coding in the modules would make them more apparent or obvious, and found sooner or prevented from propagating in the development process or the system itself.

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

The question really is "Are they?".

Few things I'm aware of

[ThePhilips]

How Do You Prove Software Testing Saves Money?

Generally, in short term testing doesn't save money. Though in long term, there are some benefits.

0. Software testing doesn't save money to the software developer itself - it generally saves money to the customer. Having software being tested, allows customers to cut the time and the effort in the critical phases of the larger project deployments. Some companies I know sell testing separately. You can buy their product and test the particular configuration yourself - or the ISV can do it for you. (Feature bloat is the main reason why the bogosity exists.)

1. Customer relationships. There is nothing worse than a minor design defect requiring coming up before (and causes delays of) deployment.

2. Long term software development generally builds on previous enhancements. If the enhancements were not tested properly beforehand, later development might come to the unpleasant surprise, when it would come to the light that the foundation for the new work wasn't as stable as people did tend to believe (or the foundation wasn't really satisfying the requirements it was presumed to). And it is a well known fact that software developers are poor at spotting problems in their own code. Testers having a different angle on the software complement well the development process.

Also, I would generally recommend to retrain software developers to be testers. Though not obvious to many, any effective testing requires knowledge of the source code. I had seen lots of wasted effort due to the fact that testers were not aware of the implementation peculiarities.

For smaller companies I would advise a simpler scheme: rotate primary focus of developers between coding and testing (and support). Idea is that developer should have enough time to learn particularities of each of the development phases and bring the new experience back into the other development phases.

and I can only assume it's because we don't have a formal test suite.

For internal applications, make sure that you have requirement specification. Testing is impossible if the expectations are not spelled out formally.

If you do not have a requirement specification, then try to create one before advocating for testing. It might turn out that the software has too many interested parties and finalizing requirements would prove impossible. Then testing might cause more harm than good, by breaking the unwritten unspoken balance. By becoming a tester you should be ready to make an enemy out of better half of the company.

Answers:

[Alex Belits]

once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite

No, it's because you didn't fix it. "Bug" is usually described as a set of symptoms. A very simple test can determine if the changes you made altered the behavior of a program that was recognized as a bug, however it's your responsibility to have understanding of a product that allows you to recognize and fix problems. Tests merely help to determine if some known aspect of product's behavior is unaltered (if tests could find bugs, products would never be released with them).

The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money.

You are on a fixed salary, you idiot! No one spends money or saves money depending on what you do -- it all depends on what product does (and how poorly) because it affects sales. Unless you are asking them to buy some software (what would be a wrong approach), there is nothing he is supposed to do -- it's your job to implement everything you need to do your job properly.

Re:Answers:

[SecurityGuy]

No, it's because you didn't fix it.

Not necessarily. He can fix it and some other developer can un-fix it. Microsoft famously does this. Not frequently, but more often then they should.

You are on a fixed salary, you idiot!

Partly irrelevant. You're still paying the guy $X per hour to fix the bugs. If he could do something else useful during that time, it's valid to say the time and money spent on fixing the bugs is saved (and spent on something else).

Re:Answers:

[Alex Belits]

Not necessarily. He can fix it and some other developer can un-fix it. Microsoft famously does this. Not frequently, but more often then they should.

Testing won't help with this, either. Not that Microsoft developers being able to do something extraordinarily idiotic is always a valid explanation for the same thing happening outside Microsoft.

Partly irrelevant. You're still paying the guy $X per hour to fix the bugs.

Developer is being paid regardless of what he is doing -- writing new code, documenting things, developing tests, fixing bugs or doing nothing waiting for something to happen (typical condition right before release when product is being beta-tested).

If he could do something else useful during that time, it's valid to say the time and money spent on fixing the bugs is saved (and spent on something else).

Software development does not work that way. People have to spent fixed (and significant) amount of time switching between unrelated projects, so effectiveness of developers working on multiple projects depends not as much on how much time they have to spend on projects but on how often they have to switch. Adding "something else" to overloaded developer is often the best way to bring development to a grinding halt.

Re:Answers:

[An Onerous Coward]

One developer stomping on the changes another developer makes to a very large project is "extraordinarily idiotic?" I don't see why you think so, especially given that you're railing against one proven technique for keeping such extraordinary idiocy from happening. Commenting code will not prevent it. Well managed source control will not prevent it. Good communication will the rest of the team will not prevent it. A test suite will not prevent it. But between the four of them, you have a decent shot.

"People have to spent fixed (and significant) amount of time switching between unrelated projects, so effectiveness of developers working on multiple projects depends not as much on how much time they have to spend on projects but on how often they have to switch."

Exactly! Which is why you don't want to have to be constantly context-switching back into a legacy project to fix the bugs that would be prevented by a test suite.

Re:Answers:

[An Onerous Coward]

"If tests could find bugs, products would never be released with them."

That presumes that every bug would have a test written for it. Presumably by magical, test-writing fairies.

It's the programmer's job to think of the ways that a given piece of code might break. Sure, a perfect programmer would know every possible test that could be written. But then, the perfect programmer would write the perfect code, and wouldn't need to write tests.

That is, unless he was working with imperfect programmers.

That's the point of testing. It's a reality check for a programmer's own arrogance. You can be confident in your own work. But if you can sincerely say, "I understand this large, complex project well enough to guarantee that this set of changes will not introduce any bugs," then you are either among the elite 0.5% of programmers, the changeset in question is trivial, or you're just overconfident.

Each test you write is basically a statement that says, "Dear code. You are expected to behave thusly. I will catch you if you start doing something else."

"You are on a fixed salary, you idiot!"

Not sure what you're saying here. Perhaps because I'm an idiot. But "fixed salary" generally implies a couple of things. First, it implies that under normal circumstances your work should be completable within a standard forty hour work week. Second, it implies that you have a lot of leeway in what work you do and how it gets done. If you're regularly working fifty or sixty hour work weeks because you have a very well-defined set of work that must be done, then by law you shouldn't be salaried; you should be getting overtime.

Salary is not -- as you seem to be suggesting -- an excuse for employers to dump eighty hours of work on you and then expect you to do it. In this instance, if his employer is expecting him to complete his regular forty hour workset and then to also make the up-front investments in a test suite that he feels he needs "to do his job properly," that's simply not a reasonable expectation on the employer's part.

Clearly, in this case there is something his boss can do. Give him permission to start building a test suite, or exploring the various commercial and open source testing harnesses that might suit his needs. "Salaried" doesn't mean that your bosses have no say in how you spend your time, or that you don't need to communicate with them or get buy-in from them before you make major changes to the system (especially when those changes require coordination with dozens of other people doing the same job you are).

Re:Answers:

[Alex Belits]

One developer stomping on the changes another developer makes to a very large project is "extraordinarily idiotic?"

Yes. Make a bug once, shame on you. Make the same bug twice, you are probably George Bush with a compiler. Not that "the same bug" from a human's point of view will fail a test written to catch its previous incarnation.

I don't see why you think so, especially given that you're railing against one proven technique for keeping such extraordinary idiocy from happening. Commenting code will not prevent it. Well managed source control will not prevent it. Good communication will the rest of the team will not prevent it. A test suite will not prevent it. But between the four of them, you have a decent shot.

One of those things is more worthless than others.

Exactly! Which is why you don't want to have to be constantly context-switching back into a legacy project to fix the bugs that would be prevented by a test suite.

If it's a "legacy" project that is only being maintained with bug fixes, how the Hell you will get a regression due to multiple developers' conflicting changes?

Simplicity?

[Fri13]

What if you would just simply drop all money metrics and say that the pleased client who gets working product is happy customer and happy customer brings more customers in long run.

If the boss can not understand how valued for customer is that they are happy from the product they buy, he (or she) ain't good boss.
The company will go down when there comes competition whom gives better products. If you can not gain a monopoly or dominant market position or any other way to tie customers to your services and products. But that's it. No more new customers as what the company would get with happy customers.

Been there...

[dogsbreath]

"at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite"

I assume that what you really want is to improve the development process and reduce your frustration time. A shiny test suite seems to be the answer.

Hmmmm... before you go buying a test suite you have to understand that the business should drive what you do. Business cases are all about research, analysis and documentation:

1. Metrics: What is the value of the app? What is the cost of maintaining it? ie: How many person hours are spent on development and maintenance? How many bugs are fixed per period and what is the average cost? Can you create some metrics for code quality and code brittleness based on version release stats and fault stats? You should have a handle on all of these items. If you are not in a position to know then you might start by persuading your boss to do some research on these topics. If you have an experienced crew then you should be able to estimate at a high level what the important metrics are, even if there are no formal stats being tracked. What do these metrics reveal as hot spots?

2. Processes: What are your current development, maintenance, and release processes? How are bugs tracked? If there is no formal process then what is the informal process? How do things happen? Do other people have thoughts about the processes and issues? Document how things are done and what the known or perceived issues are. If bugs are being reintroduced because of bad process then document this. If you don't have good processes then software tools will just be anchors that help drag you to the bottom. The worst thing you can do is introduce a tool that only you know, which then becomes part of a hidden, informal process that only you know.

3. Analysis: What does the research into metrics and processes show? How do you compare against standards either from industry or from inside the company? If you can't find a comparison then how do things look versus where you would like to be? What is the low hanging fruit: the obvious things that would improve quality? Lack of formal bug tracking, change control, and other metric gathering may be the obvious place to start.

4. Make a plan to improve things. This means that there has to be a sequence of changes and for each change there must be details of what the change will improve and how to measure the improvement. Be clear on what the goals are. No airy-fairy stuff. Hard, documentable, measurable changes. At this point you may identify some software that would help, such as a test suite. Remember though, software tools are just that: tools. No sense using a hammer to push in screws. Simply creating a change control process with a regression test may get rid of a lot of issues with spending a dime on tools or hardware. Simply formalizing and adhering to strict development and release processes that utilize (free) version control and (free) bug tracking may cure many problems. Similarly, standardizing methods can go a long way to improving quality and reducing problems.

5. Buy in: at every step of the way, from initiating the research into the situation to creating an improvement plan, you need support from your boss and from your peers. If nobody wants to change then fuhgetaboutit.

The above may seem like a lot but it isn't really. The initial metrics gathering, analysis, and plan development should not require a huge effort, especially if you are a relatively small shop. Some of the numbers may be sourced as WAGs from subject matter experts but I prefer to use the term "heuristics" or educated WAGs ;->

It only costs money if you need to hire help

[petes_PoV]

Although a lot of consultants like to try and pin dollar costs on "quality", coding standards, development tools and other intangibles, the inescapable truth is that unless these processes and methods actually stop additional money from leaving the company they have no real-world, measurable value. Sure they might make your life easier (though most just increase the amount of time it takes to get something done), they might reduce the number of bugs, or the time taken to fix one, However if that time is already accounted for in your working day then the only way these things could save money is if they get you fired and can save your salary.

Since you appear to have the time to spend on these support tasks, there would only be a saving (and one that would take a considerable time to realise and even longer to pay back) you could only really make a case if you can say that int he future your workload will increase to the point where another person will have to be hired to assist ot take over these tasks. That's the only solid, dollar cost you will have to offer. Even then it's going to be an uphill struggle as the zero-cost alternative is just to make you work harder for the same money.

Re:It only costs money if you need to hire help

[dogsbreath]

... they might reduce the number of bugs, or the time taken to fix one, However if that time is already accounted for in your working day then the only way these things could save money is if they get you fired and can save your salary...

Ahhh.... I can't let this go by.

First off, let me say I agree that simply bringing in a consultant and doing some "thing" (process change, tool implementation) based solely on a best practice is foolish. Absolutely there has to be a real, identified problem to fix or efficiency to be made and there has to be a real, concrete, measurable way of determining whether the change works. Almost always, the answers exist already within the core group of experienced employees.

But... I must take issue with the idea that "the only way these things could save money is if they get you fired". That statement is almost never true for any non-trivial, non-monolithic IT shop. It is important to attribute costs accurately and reducing labor spent on a product, whether support (operational expense) or development (capital expense), is important. From a business view, spending less time on a particular item is a real savings.... no matter what. That is one issue.

What the business does with the employee time that is returned to the resource pool is another issue. It is important to separate these items. Also the phrase "save money" is almost meaningless or at best a fuzzy concept that has a subjective meaning when referring to work efficiencies. It would be better to say "reduce operational/capital cost" for a particular item instead. From a business POV, the returned resource time is an opportunity to redirect the resource to another ROI generating area. The time saved becomes time that can earn more $$ on something else.

Employee time is almost always the single largest cost for an organization. Increasing productivity (reducing time spent per unit) is always a top priority. If the company's management does not understand this then there are fundamental problems in the organization. If a firm's response to reducing required hours on a project is to let go of permanent employees, then there are other issues and a smart IT worker should make sure his/her parachute is packed, IMO. Reducing consultants/contractors is one thing; reducing permanent staff means there is a willingness to write off a significant investment in training and experience.

Re:It only costs money if you need to hire help

[dogsbreath]

--
politicians are like babies' nappies: they should both be changed regularly and for the same reasons

Man, that is SO true!

Re:It only costs money if you need to hire help

[Fulcrum of Evil]

Don't they teach anything about risk these days? Testing and QA improve quality and reduce risk, making the chance of failure go down. That's worth a bit more overhead, isn't it? It's not like the CFO invested their quarterly profits in lotto, right?

Re:WHY do you have to prove software testing saves

[outsider007]

So you think American companies have a commitment to good quality for its own sake. Just out of curiosity, what do you drive?

Minimal Test Suite

[ATestR]

How do you prove software testing saves money? Not going to happen in this case. The place testing saves money is during development. Once the software is deployed, its too late.

This is a pretty much the same place I'm in with my current position. 10-15 year old application that just "grew" into its current state, generally without formal testing. The decision to introduce formal testing into the process was probably made to meet standards of external users in my case. However, as the Software QA analyst who ended up setting testing up, I quickly realized that it would be impossible to set up complete testing in any reasonable time frame. Instead, what needs to be done is to establish a basic framework for testing, and introduce some very high end tests that will run the application's key points. Then, every time a new version is generated, run the tests. Shouldn't take very long. If a bug fix is introduced (other than cosmetic), add a check for that specific item. Do not bother to check everything. As the old axiom goes, if it ain't broke, don't fix it.

Re:Minimal Test Suite

[An Onerous Coward]

Fair points. On the other hand, once you've got the testing infrastructure in place, and you have to go in and revamp an area of the code anyways, one recommended way to refamiliarize yourself with that code is to write tests for it. It's a good approach, because it forces you to more rigorously understand the functionality of the code, gives your exploration some guidance and direction, and leaves behind some path markers for the next person. It's a little closer to the "test every damned thing" mindset, but not so much that it will destroy your productivity.

But does it? Not always.

[blind biker]

Civilization V was released with a ton of bugs, many of which were (are!) showstopping crashers, as long as you got to a few hundred turns. Add a trillion smaller, non-crasher bugs, and you have one of the buggiest releases of all time. And yet, between brick-and-mortar and Valve/Steam, they sold about a million units. That's about $50M in the bank.

Testing? What testing?

Re:But does it? Not always.

[vlm]

Civilization V was released with a ton of bugs, many of which were (are!) showstopping crashers, as long as you got to a few hundred turns. Add a trillion smaller, non-crasher bugs, and you have one of the buggiest releases of all time. And yet, between brick-and-mortar and Valve/Steam, they sold about a million units. That's about $50M in the bank.

Testing? What testing?

The key difference is Civ5 is a short term experience, its just something you play until Civ6 comes out, assuming they survive.

On the other hand, the original post

We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago.

implies every sale is vital since there have only been a couple hundred, and its just one app, not a series.

In other words, everyone knows civ5 is super buggy and as long as its fun for awhile, who cares. On the other hand, over the previous decade word has gotten around about the original poster software thus only a couple hundred sales.

Quality control a decade ago might have turned them into a billion dollar company over the previous decade, rather than their current pitiful state. If, perhaps, they're planning on winding down next year, debugging for one eleventh the lifetime of the SW is probably pointless.

Overall cost may not be "in his own best interest"

[dtmos]

If you truly are working for a small software development company, cost may not be the issue. Frequently (I may even say universally) the issue facing the owners of small companies is not cost, or even profitability, but cash flow -- making sure that there is enough money coming in to make the next big expense coming up (frequently payroll). In small companies there typically aren't the cash reserves available to spend on unanticipated expenses or program delays. The boss may even agree with you that the overall cost would be reduced if software testing were introduced, but may not "want to invest the time and money" because the company does not have the cash available to make the investment.

For example, the software you're working on needs to ship by the end of January to receive payment from the customer by the end of February, so that payroll can be made in March. If bug-fixing stops in January for development of the bug-fixing program, the customer doesn't get his software until the end of February, so he doesn't pay until the end of March and so payroll is missed that month. Having fewer bugs in the long term has to be balanced by the need to stay in business until the "long term" is reached.

It's like anything else in finance: You have better options and get a better return on your investment if you invest $100,000 than if you invest $1,000, but if you don't have $100,000, it doesn't much matter -- you do what you can with the resources you have available. Similarly, you get a better return if you're able to invest your money for a year than if you must have it back in a week. (The guy who said "time is money" actually knew something.) This trade between cash flow and long-term efficiency (leading, one hopes, to profitability), i.e., knowing when and how much to invest, is the real art of business management.

The solution to your problem? IMHO, incremental investment. I agree with those above who suggest implementing tests for new bugs found. This should enable you to begin to work more efficiently over time, without substantially delaying current work, while collecting metrics (I know, I know...) that can help quantify the cost of the bug re-work. Once a substantial body of tested bugs has been collected, one can institute a formal testing program (preferably by including it in the next project, so that it is a planned expense) as a cost reduction, since by then it should speed up development work over the ad hoc method then in place.

Rewrite the fucker

[Anonymous Coward]

You don't mention what this application is written in, but anything that was written ten years ago needs rewritten from the ground up. There are probably libraries and other much-improved functions that you could take advantage of now that didn't exist ten years ago.

What is test for you?

[Parker Lewis]

See in this post comments, how people think that test is only unit tests. Testing must not be write by developers. Period. Reasons: developers are the bottleneck (about time) of the development process, they think that software as their chield, they are so much time over the software that they cannot see obvious defects, developers look for a defect how a negative point for his software (and we known that ANY software in the world, even the most simple one, has a lot of bugs), they have a lot of software know-how that average joe user do not have, so if you ask to a developer test your own software, you're wasting money. Sure, they must write and run unit testing, but this is the minor portion of testing. Testing starts when someone starts write requirements docs. Someone from testing team must read this document and check for ambiguities. At this moment, find some ambiguity, it's the most cheap testing. If you find a bug in the internal testing, it's more expensive then find it on doc. More expensive than that, is find a bug on user acceptance test. But, the really most expensive bug, is the one found in a running/deployed software: you have to take time to really understand the bug, put the analyst to re-design, programmer to fix, and testers to run it again. It's hard to find a excellent tester, cause he must really understand the application, it's good if he have a base background of develop (when automation is needed), he must understand how to write negative tests, he must have excellent communication skills (write and talk), etc And, the best documentation to start learn about testing, is the International Software Testing Qualifications Board syllabus: http://istqb.org/download/attachments/2326555/Foundation+Level+Syllabus+(2010).pdf The good thing about ISTQB: it's not related to any commercial company. And, testing is not about run an audit in the testing team. Testing team has the same objective that all the company: deliver a free bug software (free bug is impossible, but at least at an acceptable level).

Re:What is test for you?

[Parker Lewis]

Sorry, I forgot to send the comment as pure text (it lost the paragraphs).

Re:WHY do you have to prove software testing saves

[jellomizer]

Because some people and companies like to have a quality product.

Are You trying to kill Your own job !?

[hugetoon]

Let it be, if it's the same bug, you'll fix it faster and go home earlier.

Re:WHY do you have to prove software testing saves

[tibit]

Nobody puts up with bad products anymore.

-- Ha, ha, ha. The U.S. market obviously demands plenty bad, nearly worthless crap, if one were to judge from what's on the market.

There are simple things like cast-iron hand-cranked meat grinders where the market is saturated with extremely poorly made chinese crap, with absolutely no alternatives. I know, I've tried to buy a new one -- ended up getting a used Porkert instead.

Another simple thing: portable DVD players. I've already had to fix two of them because they had liquids spilled on them. It's not a rare occurence -- they are often used by kids who either don't care or just forget themselves and get unlucky. While the manufacturing quality is perfectly acceptable, the design quality just sucks big time. The clearances between many parts seem randomly chosen, so that either things that rub on each other seize, or they rattle. An no, this isn't a molding issue, it's by design. Molding on the units I looked at was by the book. You can tell that many small cross-section plastic parts, prone to breakage, were designed by people who have no clue or no experience in mechanical design. There are plenty of changes that one could make that would not increase the material costs, but would make those devices much more rugged.

It seems that many U.S. companies that brand products imported from China have no engineering on the other side of the pond, just marketing and purchasing people.

Hubris

[noidentity]

The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

Why don't you simply explain to him why YOU are so sure that it will save time and money, and what testing you have in mind that will do so. Assuming your certainty is based on strong evidence, he should find it convincing as well.

Re:WHY do you have to prove software testing saves

[gbjbaanb]

Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.

in consumer devices perhaps, but that's a different area to most software. For most software sold, especially to business users, bugs are accepted as one of those things that happen. Now, really buggy software will eventually get replaced but in most cases the important thing is to have a good (or excellent) customer service that responds to, and fixes the bugs. If you have that, your customers will not care so much about bugs that they find. you just need to be responsive, and considerate in your communication with them. Once you have that in place, you'll often find that its an asset to you, the more bugs the customer finds, the more they feel loved. And customers like feeling that you care for their business - especially when its some executive/supervisor reporting them, and some grunt who has to actually use your PoS software. Its how the big names can make such poor enterprise software and still have it selling in billions of dollars a year in maintenance/support payments.

I think I might have just explained how buggy software is good for your business. I feel dirty.

John_anon

[Anonymous Coward]

100 people using software A 1hour per day;
software A has a new bug caused by lack of testing;
it takes 3 days to fix the bug; the result is: 100ppl * 1hour * 3days= 300hours;
lets say u pay ur staff 20$/h. so 300 * 20$ = 6.000$ per bug in software A.
now think about what would happen if the deadline was 2days.. that would realy be expensive.

btw: ppl hate bugs and they will hate sw A if it has bugs. if u hate A u dont want to work with A, and as u have to, u will do ur work much slower because ur in a bad mood.

Question without an answer

[macraig]

Your question can never be answered definitively enough, especially to satisfy suits and bean-counters. As evidence I offer you the ONGOING tug-of-war over tech support: is it an expense to be minimized and avoided at all cost, or does it actually aid the bottom line by boosting goodwill, etc? Even after decades that question hasn't been answered, and customers often suffer because of it.

The question you ask is just as insoluble. You might as well be debating religion.

Validate nothing is broken / development tool

[Fuzzums]

One of the worse things that can happen while fixing bugs, is introducing a new one in an unexpected part of the software hidden somewhere deep inside the code.

With a (big) set of automated tests, you can check everything is still working as expected and if a test fails, you know where to look. That is if you've made enough tests.

One problem I found in Scrum is refactoring. If at any point you want to change a piece of the code, you'll find you'll have to rewrite the tests as well.
Another thing is writing tests takes time. It will probably depend on the type of software you're writing, but I found it takes about the same amount of time it takes to write the new software.

You'll also see that you';; get faster over time. First of all because you'll have more experience writing good tests and second because a lot of test data and helper functions have been written already.

Then there is test-driven development. It's easier to run a test that specifically targets your new method, than starting up the application, loading the data, process the data and finally get to your breakpoint after a few minutes of calculations.

Re:WHY do you have to prove software testing saves

[dbIII]

Exactly. So why not test things properly and give it to them instead of the usual situation?
Testing to determine if something is fit for a purpose applies to pretty well everything apart from basket weaving and software.

Identify the real problem

[JustNiz]

If you're repeatedly seeing the same bugs come back that you've already fixed, it strongly suggests you have sloppy developers repeatedly merging old/outdated code back in to the current trunk over fixes.

It seems clear your real problem is around a lack of software process and/or version management so testing is after-the-fact in this case. you need to fix your development process first. Adding a formal test environment on a bad process is just putting expensive lipstick on a pig.

Really 2 issues

[Antique Geekmeister]

The first issue is that you have to convince people that their business practices are wrong. This is not easy, and often requires lying through your teeth with statistics. Testing costs time and money and slows down releases. _Foolish_ testing, with lots of business case meetings and Powerpoint slides and expensive test configurations that don't actually test anything but demoware, is even more expensive and wasteful. The testing needs to actually prevent support calls, save engineering time re-fixing the same bugs, prevent support calls, and especially _help sales_. Ideally, they also help prevent liability: look up the phrase "always mount a scratch monkey" for the canonical story of failing to test a change.

The second issue sounds like a basic source control issue. It sounds like developers or engineers are reverting fixes after you've made them and committed them, and this should _never_ happen. Switching to a better source control process sounds vital to your issues, and may solve a lot of your issues without expensive test suites. Which source control suits your local programming styles is your own choice, but having a central trunk or repository that changes, and logging of changes, is committed to might at least help you cover your backside the next time this happens.

Bug, or Business Logic?

[Algorithmnast]

One thing I haven't seen mentioned/asked - is it an actual bug, or is an area of ambiguity?

I once worked on a project with ambiguous requirements (there's a shocker), and at least two programmers had a different interpretation of what the requirements meant. Since they didn't write code that interacted directly, it didn't normally matter. But every few months there'd be a bit of a circus where their implementations would cause the software to behave poorly in a third area. And it would take a few days to get the Tech Director involved to require them to "play nice".

So - Is it a mis-implementation (a bug) or is it a result of ambiguous requirements? If the latter, then software testing won't help.

Depends on your business model

[RogueWarrior65]

You have two choices: 1) Test now or 2) Fix the bugs later. Okay, technically you could blow them off entirely but you won't be in business much longer. But I digress. The choice between 1 or 2 could depend on your business model. If your company charges for maintenance releases then it's in the company's best interest to not fix them and wait until it's getting paid to fix them. This could also apply to the annual major release business model since you can blow off bug fixes until next year. If you subscribe to the older method of software releases, one could argue that releasing a bulletproof app boosts your perception in the marketplace. The tradeoff is taking forever to actually release a product.

These days it seems that major software companies have adopted the annual release model (with minimal real new features I might add). And more companies are sticking you for maintenance every year. And some companies are trending and would dearly love to run software over the net or in the cloud because then the customer never owns it but instead rents it in perpetuity which means the company is guaranteed a steady stream of income instead of getting a big spike after a major release and tapering off until the next one.

I think the only exception to stalling on bug fixes would be showstoppers. Bugs that really piss of the majority of your customers.

Because clients don't like things that work unexpe

[Anonymous Coward]

I work for a large bank. All software gets thoroughly tested. That is because were it not tested, customers would get unexpected results. And unexpected results could mean not being able to log in to Internet banking. Or the wrong authorisation code applied to a credit card transaction causing the wrong level of interest to be charged. Or even their address not being input as expected meaning it was hard to verify. These errors would cause the bank's customers to leave en-masse, and then there would be no business.

Depending on your tolerance for things going wrong, your tolerance for leaving it to chance also varies.

We have zero tolerance. So we test. Thoroughly.

Be careful with the efficiency thing...

[Anonymous Coward]

The only way your boss is going to save money is by reducing head-count... (Increasing sales isn't his department.)

Solution

[Charliemopps]

Just stick the word "Beta" at the end of the software version but sell it anyway. When bugs come back say "Thank you for your beta testing!" and then promise them some free, but useless addition to the software when it finally comes out of beta (which it never will.) It seems to work for everyone else out there, why not you?

remoteshell

[remoteshell]

In my experience, the answer is nothing. You've tried to convince him, and he can't see your point. Until the owner suffers a loss of revenue or reputation you're just annoying him, and frustrating yourself. The problem is that you said 'the owner'. If you were in a bigger pond, and you enjoyed corporate gamesmanship, you could spend a year or two, and present metrics up the chain. Now get off his lawn.

Lose The Job

[assertation]

If you are catching shit when a bug happens and they refuse to set up formal testing get out of the job now.

If you aren't catching shit of any kind, then look at the bugs as helping you keep employed. If it annoys you, go find another job.

I've worked for similar cheapskates. They aren't likely to change.

Maybe something from this NIST report?

[mpsmps]

The Economic Impacts of Inadequate Infrastructure for Software Testing finds an average ROI for software testing somewhere between about 100% and 1000%.

Possible outcomes of bad software

[digitalhermit]

A couple things could happen:

1) Code related issues lead to critical failure. For a revenue generating system this can have immediate effects on the bottom line. For non-revenue systems this can affect how many support hours are burned fixing problems.

2) Financial impact from lawsuits or non-compliance can be costly. If a security bug or design flaw in your application leads to PCI non-compliance, you may find your cost of doing business with credit providers can rise dramatically.

You can't test quality into a product

[Just Brew It!]

Depending on how bad the code is, it might be more cost-effective to rewrite it from scratch (assuming you've got the resources to do it right this time).

What a comprehensive test suite can do for you is verify that each new release of the product meets specs (i.e. prevent regressions). But for the tests to have real value, you need to do it right:

1. If you don't have a comprehensive system specification, create one. The system specification should be written with an eye towards verification -- e.g. it needs to be decomposed into simple, easily testable requirements of the form "When A occurs, the system does B." Make sure you cover what the system is supposed to do under unexpected conditions as well (invalid inputs, network connectivity problems, etc.).

2. For each requirement in your system specification, develop a test (or tests) that verify it. The tests should preferably be as automated as possible, because if they require too much manual labor they will get blown off.

3. Create a written test procedure. The tests are worthless if people don't understand how to run them and interpret the results!

4. Keep the system specification, tests, and test procedures up-to-date as the system evolves.

If it sounds like a lot of work, that's because it is! I'm not saying you shouldn't have formal tests, but incorporating them after the fact -- especially if the system specification is incomplete (or even non-existent) -- is going to require a lot of effort. Without knowing the state of the existing code, the complexity of the system, the approximate cost of finding/fixing bugs in the field (is this a mission critical app that results in significant lost productivity when it breaks?), and a myriad of other factors, I don't think anyone can conclusively answer the question "Is it worth it?"

I've been this owner...

[salesgeek]

What you are dealing with is a situation where the company is using "bench time" where the programmer isn't producing revenue, or isn't doing the job that his salary is allocated against. To the company, bench time is basically free (yes, it does have a real cost, but it's a sunk cost, so it's perceived as being already paid for), and there isn't enough of it for a major project. So the application is updated by whoever is free, and the QA guy gets to test and bugfix when he is available. The situation sucks, but you are unlikely to get the owner to change his mind because once you turn maintaining the system into a major project, the owner may want to hire lower cost people to do the work or may even rather scrap the system and invest in something better.

If you want to discuss it with the owner, start the conversation with a simple question: is the system worth investing in? If the answer is yes, ask why more resources haven't been invested in the software. Sometimes business owners know the answer is yes, but have higher priorities - like revenue producing work so you can stay employed.

If you are asking this

[VincenzoRomano]

there's no chance for real software testing and related money saving.

precausions

[Anonymous Coward]

In order to write tests, your app must be written in a test-friendly way first. For example, if it's a GUI with nested DB access, automated testing is impossible. GUI front-ends are hard if not impossible to test, depending on the technology. Logic with DB access is only testable if you can easily supply mock objects for your data source.

Given your code is testable, then just start to write tests as you fix bugs. Consider writing the tests part of writing code. It's part of your job. If you don't write tests, you are not doing your job right. So no need to get permission from anybody. You should need to get permission for omitting tests!!!

Case studies

[athlon02]

Ask Intel how much F0 0F cost them. While in college, we had someone in a seminar class discuss how Intel had an engineer who didn't update the floating point division test code when they modified the FDIV instruction and POOF! there goes $500 million (iirc).

Sounds like a source code repository problem

[DigitalReverend]

Seems to me that if you are seeing bugs come back after they've been fixed, that you don't have a good repository for the source code. I would start there. That's cheaper.

Show how you avoided a nasty bug

[__roo]

This is an fight I've had to win many times over the last ten years. The most effective way to do that is to sell better software testing to the lead developer, and this is especially true in a small company that's owned by the person who started it -- they almost always defer to the lead programmer on everything he does. (If the owner is the lead developer, then you need to convince the rest of the team.)

The best way I've found to convince a good developer that testing is useful is to help them experience that sense of relief that we've all had when testing turns up a nasty bug, had it gotten out in the wild, would have caused us to spend days trying to reproduce and track down. That gut feeling -- "Whoa, we never would have found that, and it would have been a nightmare if it got out!" -- has helped me convince more developers to completely embrace unit testing, test-driven development, and working with a good QA team than anything else.

I feel you pain!

[Putr]

I work in a small Accounting company with it's own accounting web (php, javascript) based accounting app. It's used by 30 client companies but most of them are big names in Slovenia.

As a php programmer working for them i could sum up the development and maintainenc of the application like this:

No version management (no git, svn etc), no testing, no coding guidelines, 8 year old code with totaly new code, No OPP, mostly just copy/paste (i've seen basic functions on 30+ places in the code), HTML&CSS not even HTML4 (invoice for printing is wrapped with a ), works only in Firefox 2.0, the code loads all the objects (thousands lines of code, php and javascript) into $Session ($_SESSION['something'] -> new Something();), half of the code is in Slovenian, half in english ... etc etc etc.

It's basically just a pile of ~300k lines of code nobody understands how it works, when it works.. sometimes.

O and all the programers are students from the local collage and are payed 4,5€/h (about the same as a maid, waitresses are paid more, lot more). (One beer = 2,5€)

So ye, i feel your pain, i tried to implement GIT but our boss couldn't understand it so we're not using it :) ... i suggested selenimum(testing) and he didn't what to hear it.

Your 5 point plan

[TheChrisCarroll]

1) Write a simple cost-benefit calculation.
Estimate the cost of your test suite and estimate the cost of your current bug-fixing. Business folks aren't mugs when it comes to profit and loss calculations.
2) Don't pre-judge the outcome.
You _may_ be wrong. You may find that the cost of building the test suite is more than the cost of the bug fixing
3) Guesstimate the cost of business reputation and lost sales caused by customers finding bugs.
Ask someone in sales to help you with this. (No, don't believe Scott Adam's view of marketing people. They may not be engineers but they know the value of a lost customer).
4) Briefly justify all your guesstimates
You're don't know any of the numbers really but you can outline the reasoning they are based on and let the owner replace your reasoning & estimates with his own.
5) Let him convince himself.
the owner may have some hard-to-express reason why he doesn't buy it, and that reason my be simple short term cashflow. If so, redo your calc taking his point on board (e.g. do a 1-month profit/loss not a 1 year) and pass it back. You're responsible for doing the best you can, he's responsible for making sure there's money to pay salaries.

But at the end of the day, don't complain at someone who's paying you to fix bugs in a sluggish economy. At least he's keeping you in work for the year.

Re:WHY do you have to prove software testing saves

[hopeless case]

However, I've found writing a proper test suite that deals with databases, network communication+++ and not just the application itself is pretty hard

Writing a comprehensive test suite that did all of that would be an enormous undertaking. Writing a single automated test dealing with network comm or with a database interaction, however, is not a huge task, although it does take more time than manually testing for the existence of some bug you are in the middle of fixing. If you only automate a small fraction of the bugs you manually test for, however, those automated tests will grow in their usefulness over time, as some of the bugs they test for will resurface in the future and be found immediately thanks to the test script. This usefulness can be observed by teammates and managers and inspire imitation. At developers meetings, the next time a bug resurfaces in one of your colleagues work, a manager might even ask "Wouldn't you have found that faster if you had written an automated test for it like Steve over there has been doing for his work? Maybe you should spend some time with Steve so he can show you how to write automated tests."

Because it's already being tested

[b4dc0d3r]

You're looking at this as a hardware vs. software issue I think. Crap electronics get basic tests like 'are the battery leads connected' and 'does it turn on'. Same as the bug fix or patch is tested. OK, it worked, get it out there. So what you're talking about is already being done, basic testing.

OP is talking about setting up an entire QA process that tests every possible case that can be thought of. This is not basic 'it works' testing, this is full-on 'let no part be unbroken' quality assurance. Sell that idea to the chinese and see how far it goes.

Re:Because it's already being tested

[Fulcrum of Evil]

OP is talking about setting up an entire QA process that tests every possible case that can be thought of.

No, he's talking about a formal test suite. This means testing that isn't ad-hoc, but not much more.

Re:Because it's already being tested

[dbIII]

entire QA process that tests every possible case that can be thought of

You don't get that in manufacturing either and I never pretended that happens. You get things like dimension tests using go/nogo gauges to see if parts fit within tolerences. It's a simple test suite in physical form.
My question is: why do people think software is so special that you can get away with cutting corners that would be considered too dodgy for the cheapest manufacturers if it was a physical product?
Do they really think conduct considered unprofessional in other areas is acceptable?

Re:Because it's already being tested

[HeadlessNotAHorseman]

I bought a dvd burner a few years back. It has bugs. If you are recording a program to DVD, and watching a different program off the DVD, it is wise to avoid fast-forwarding because the unit is liable to freeze. The only way to unfreeze it is to unplug it and plug it back in again. Also, the clock loses about 5 minutes per month. What the heck is with that? It's surely not that expensive to build a reasonably accurate clock! Not only that, the UI is so bad it takes over 20 button presses to navigate to the point where you can change the time settings.

Simple...

[guybrush3pwood]

Have two or three critical bugs split into production and crash the system, and you will enjoy this piece of theatre:

Manager: What happened???
Programmer: There's a bug.
M: Why?
P: This artifact wasn't tested.
M: Why not?
P: Because you said it was too expensive.
M: ??!!

Re:Simple...

[John Hasler]

Congratulations. You just made your manager hate you.

Re:Simple...

[guybrush3pwood]

He hates me already.

Re:Overall cost may not be "in his own best intere

[Anonymous Coward]

This a thousand times!

Also lets look at it this way. If it is say costing the secretary a half days work and she makes 25k a year. The owner is not out much and a few people are a bit annoyed. But it will cost 4 devs, 2 QA's people full time for 2 months solid to go rework plus possible new hardware and retraining. Then the half day inconvenience every 2-3 months is cheaper.

Now if it say takes out the business every 2-3 months then it is worth looking at. You will need to make some graphs in excel and show *WHEN* the ROI is. Even given that you may not be able to do it just to like what the parent here is say. There may not be the cash flow to even hire those other 5 people...

Re:Overall cost may not be "in his own best intere

[Anonymous Coward]

For example, the software you're working on needs to ship by the end of January to receive payment from the customer by the end of February, so that payroll can be made in March. If bug-fixing stops in January for development of the bug-fixing program, the customer doesn't get his software until the end of February, so he doesn't pay until the end of March and so payroll is missed that month. Having fewer bugs in the long term has to be balanced by the need to stay in business until the "long term" is reached.

I like OpenBSD's way of doing things: fixed release shedules. They spend 'x' months integrating new code, and then 'y' months testing it. (Often x = y.)

Similarly for Joel Spolsky's company (to take one documented example), they get a list of features that people want (people = developers, marketing folks, customers) and rank them. They start coding things up from the top of the list and work their way down completing as many features (which may include 'internal' things like "refactor the parser") until a cut-off date. After a certain point new features stop going in and the focus goes to QA and testing (which also happens in parallel with coding).

At the end of it you have a set of new features but also a fairly predictable schedule. If a feature isn't ready in that time it will have to wait for the next cycle (which is six months later in OpenBSD's case).

If you base releases on features, then you can hold up a release (and revenue) for an inordinate amount of time just for one or two things when you've already got a decent product ready to go. (This is what caused FreeBSD issues in with their 5.0 release—they tried to do too much.)

Quality of the code is the issue

[tg123]

The issue you are having does not sound like it will be solved by better testing. All you are doing is fixing mistakes which is basically rework.

You need to increase the quality of the code.

This is where your biggest cost savings are made as poor quality leads to higher costs.

http://en.wikipedia.org/wiki/W._Edwards_Deming W. Edwards Deming - Famous for helping the Japanese rebuild there industrys during the 1950's.

Mod parent up.

[John Hasler]

What is your version control system and how are you using it? Your build system? As someone else mentioned, what are your rules for commenting and documenting bug fixes? Perhaps you can eliminate many of these regressions by improving your process (not that regression testing is a bad idea if you can justify the cost).

Re:Mod parent up.

[dogsbreath]

Good questions. Some regression testing can be done through simple scripting, especially if it is a web based app.

From the original question, my main worry would be that the app is getting brittle.

Cheers

Been there, done it with stone knives

[TomasChymiste]

The Guy with the Gold (who makes the Rules) _never_ wants to spend an hour or a cent on something he can't bill for. But you still need to control the code you're stuck with or keep rolling the same rock uphill every day.

You probably also can't say rigorously what this app is _supposed_ to do -- only what it's done for the people who've complained that it doesn't anymore. So rewriting it from a real design is improbable, as well as impractical. Attack it as part of your workflow: every time you have to fix something, capture the problem you're fixing as a test case. It doesn't have to be elegant, or 100% automated; just a set of vectors to validate "correct" (what it done did before) behavior. This should, at least, minimize regression. Don't make any test too elaborate, or you won't want to run it, and that's a waste.

Even a well-structured SW product costs 10% to design, develop, document, and release, and 90% to maintain, over its life-cycle. And there has never been a well-structured one, only attempts, and mostly miserable failures. Do what you can, and watch out for Murphy!

If you did your job right.....

[Anonymous Coward]

If you did your job right the first time, had proper use cases, had proper documentation, understood the requirements, met with people that used the tool, spent a few hours testing the application...well.....you guessed it, it would have worked the first time.

Why should your boss spend money because of your lack of programming and debugging abilities? Use your head before you post stupid stuff on slashdot.

Why are you asking?

[computational super]

Dude, you just made the cardinal mistake of dealing with bosses - you told him what you were doing! Just do it already. When you're done, show him what you did. He'll think it was cool.

Exactly the same way you prove anything like this

[jimicus]

You need to sit down and work out how much the lack of a testing framework is costing you right now. Which means you need to figure out a way to estimate how many bugs you're seeing (should be easy enough - you do at least have bug tracking?), how many are silly little bugs that keep on re-cropping-up and how many wouldn't exist (or would have been squashed once and for all a long time ago) with a proper test suite.

Then you work out how much they're costing. Every person costs money - in salary, benefits etc. So you work out how many man-hours are wasted re-fixing these bugs, encountering them in the first place when they shouldn't exist any more.

Another question that significantly impacts cost: how do your clients feel about it? Do your clients use the quality of it as a gauge to estimate what your other products are like? What sort of an advert is that for your company? Much of this you may have to fudge unless you have access to sales figures.

This lot gives you an idea of how much the product is costing you in terms of lost revenue that you could potentially get back.

Then you work out how many man-hours will be taken in developing a proper test suite. Will you want to buy a commercial product for some of this or will you roll your own? How much would the commercial product cost? Bear in mind that every hour spent on the suite is an hour not spent on something else, so if your company is running a reasonably tight ship (who isn't these days?), something will have to give.

Once you've got all those numbers added up, you have a good idea of how much it'll cost to get that revenue back. Bear in mind you won't get 100% of it back - you may never see more than a fraction back - so if the cost to fix is more than - or even in the same ballpark as the lost revenue - you'll be unlikely to get it back and so you can't make the business case. This is something you'll have to accept from the moment you start to do the arithmetic - it may never add up, and if it doesn't - you were mistaken. Put it behind you and worry about something else.

Wrong assumptions?

[The End Of Days]

Maybe you should assume your fix wasn't correct? It's kinda presumptuous to jump to the conclusion that some other problem must be happening when it's your work coming back to you as insufficient.

It's a good rule to exhaust the possibility that you fucked up first.

Simple answer

[HackHackBoom]

BDD/TDD

Start now, today on your next piece of code you need to write. Make it a goal to not put out any further code without a test harness that BUILDS the code for you. Treat legacy code as a black box. This won't help you today. But it will help you in six months.

rewrite

[PJ6]

Sorry, but if an application wasn't designed with testing in mind, *especially* when it's had a bunch of developers doing whatever they want, to cover it with testing it's faster to write it from scratch (this will include re-designing the database). I've seen no exceptions. If you can't justify a complete rewrite, you can't in good conscience justify adding tests in general.

If you do test, don't cheap out and just say "NUnit" - all the free products have really crappy interfaces. BUY something. If you're a Microsoft shop, go with the built-in testing features.

Re:Overall cost may not be "in his own best intere

[Anne Honime]

I miss a "Most useful comment ever" button to click on. But this is /. so consider yourself happy if you achieve "+4 funny".

!funny

[Anonymous Coward]

Sheesh. I search for posts tagged "funny" and... nothing. I'm closing this window now.

Just sneak it in

[Anonymous Coward]

Every time you have to fix a bug, write a regression test first, make sure the test demonstrates the bug, then fix the bug and verify that the test passes. Every time you add new functionality, write unit tests and an integration test. Over time you'll get that test suite.

Lawsuit

[Anonymous Coward]

Big lawsuits seems to motivate management!

Um

[Anonymous Coward]

Roll out your untested software and see the OT your IT staff works to fix the bugs you didnt find at all because you didnt test your product.

Is this article real?

Testing is Insurance

[Anonymous Coward]

Testing should be viewed like insurance. You spend money on insurance (testing) to avoid unexpected costs to the company. You need to look at the cost of bugs to determine if the insurance is worth it -- and how much insurance is needed.

Selenium Testing and jsUnit testing

[houbou]

I've written Selenium and jsUnit test suites and unit test frameworks in the past. All I can say is that when designed correctly, these unit test save deployment headaches. When an app evolves with new features and/or the modification of existing features, unit testing is a must in order to catch bugs before you head for production. Does it save money? well, yes, because, once a piece of software is in production, the people who depend on it are using it. Introducing new bugs because of bad software implementation will disrupt the app, causing hours of user time. That is money saved. But even in development phase, good unit testing are a solid way of ensuring your code is robust. It really becomes vital as you work with a large amount of developers.

Write some bugs

[Anonymous Coward]

And I mean bugs, not bug reports. Make them hard to spot, innocent-looking in the source code and erratic. When everybody is at their wits' end, wave your magic wand and produce the test suite that nails them.

Simple Answer

[npsimons]

Unless you have a really good manager, all the evidence in the world won't convince him or her. If, on the other hand, you can at least talk to cost/benefit ratio and the manager will *listen*, have them think about this: can you build, test and package your code for distribution to your users with one button press (or the equivalent)? If it takes you more than 15 minutes after applying a bugfix to get it tested and packaged, you will almost certainly save money by automating, and this includes testing. I say *almost* because this is where the cost/benefit ratio comes in: how much will it cost to "fix" things? How often do you have to make changes?

I know the OP posted some of this already, but there's no simple answer and they need to do the thinking; the above is merely how I would think about it. The manager, if they are doing their job, will listen to cost/benefit ratio analyses; no manager wants to get fired for not pursuing something that will cut costs or raise profits.

to answer your question

[houbou]

If I were to try and find out if software testing is worth, think of the following questions to ask:

Recurring bugs?
how long does it take to fix? (programmer time)
how much productivity was lost because of it? (user time)
Did the bug fix initiate a cascade of other fixes in other modules?
Did the recurring bug fix cause database changes?

New Bugs?
Same questions, but, with the added, how did it get introduced?


Now, Software testing, should be done first and foremost based on establish scenarios which the feature you are coding is design to perform..
This means that once a programmer has implemented new code, there should be a set of X scenarios, which accomplishes specific tasks, which the app should be run against in order to see if the new feature will behave accordingly and existing features are not impaired.
That is software testing 101.
Selenium framework do this type of testing.
If you need to know that you aren't mucking with your existing code, then you need actual unit testing, jUnit, jsUnit, etc..
Bad code practices tend to be hard to unit test thoroughly.
when code is well designed, the unit test code is easy to write
\ The heavy cost of designing unit test, is when you must refactor your existing code base for unit testability.
Still, for the most part, "software testing" as usability/goal oriented tests, that should always be done.
For the most part, the question is how to implement this?
Depending on the complexity of the application, I would recommend that tests are designed around the more heavily used features first, then the areas where bugs are found. There should be no areas untested, but prioritizing the work helps greatly.

How do you prove it saves money?

[Chris Mattern]

The same way you prove anything saves money: you sit down and work out the figures. Estimate how much money (Largely staff salaries, but don't forget to estimate how much business you're losing because of buggy software) is spent in fixing things. Estimate how much of that would be saved if you test your software properly. Estimate how much the testing will cost (don't forget to include testing hardware and also delay to release time). If A > B, then it saves you money. If you have a convincing analysis and you work for rational bosses (always a concern), you should be able to talk them into it.

Testing is a Way of Life

[wrfelts]

Honestly, you may never be able to convince a budget pusher that testing will save money. What they see is a financial constraint that they may not be able to get around. As a coder, however, you can make YOURSELF more valuable by learning test methodology as a way of life. It has changed the way I think and code. My code is cleaner. I now deliver finished and stable product much faster.

First, let's look at the types of testing you may encounter. The most important, and easiest to introduce, is unit testing. Unit testing effects your approach to building clean, reusable modules. The second is regression testing, which is basically just the practice of keeping all your unit tests and running them ALL every time you add a new module or change an existing one. This keeps your bug fixes or functionality from "regressing". The next, if needed, is stress testing. This is important if you have an app that deals with a lot of data or a lot of user traffic. It helps you determine is there might be an issue with the chosen platform, database engine, even your network "pipes" clogging traffic. The last, and hardest to implement, is UI testing. That route usually requires a set of libraries that compile into your app. There aren't a lot of good ones out there, but the are some new stars rising in the field.

Let's get to the most important one now, unit testing. Unit testing, for a lot of languages can be implemented free and incrementally, which is how I started. If you are running C#, look into NUnit or even Microsoft's testing suite, built into pro versions of VS 2005 or greater. If C++, look at CppUnit. For Java, there is JUnit. If you are using SmallTalk (not very likely) there is the granddaddy of unit testing, SUnit, from which the rest derived. Even Flex and Flash from Adobe has FlexUnit. These are all free (or included in the development tools.) There are other projects out there for other languages but you'll have to research that yourself.

If you have never done unit testing get a book or three on Test Driven Development. It's a pretty large bite to chew, but give it time. This is more about improving your career than fixing a single problem. The basic idea is that, when you start solving a programming problem, break it down into small chunks and write a set of tests that will prove the code for each small chunk. Keep your classes small and focused. This will improve not only testability, but reusability, and inheritability (if those are even words). Don't tie UI code directly to the functionality behind it. This improves testability, but it also makes it easier to change out UI components when a wrong UI decision is corrected. For instance, if you program in a progress bar but the customer wanted a fuel gauge look, the functional code can service the different looks without changing.

Even with unit tested code you will still find bugs. The trick here is to make sure that 1) the classes are small and focused (mentioned earlier), 2) the class (and relevant unit tests) are well documented so that there is no guessing as to why you did what you did, and 3) the unit tests are well organized. When you run into a bug, sometimes just because of bad data in the real world, build a new test that duplicates the condition and fails (throws an error or somesuch), then modify the class(es) to handle the condition. Run that test again to make sure it worked. Then, lastly, run the full suite of tests that you built to see if you brake anything else with your change. This last test (regression testing) is the magic that makes unit testing shine.

I know that this post doesn't directly answer your question, but sometimes, a little understanding goes a long way to helping you ask the right question instead of guessing. Currently, I write about twice as much code as I used to. Half of it is unit tests (maybe more). The quality of my code, however, has greatly improved. I am also able to deliver cleaner code faster. Just like project management, where, if done right, you

Manager's introduction to Test-Driven Development

[Randlaeufer1998]

Dave Nicolette gives a great talk called Manager's Introduction to Test-Driven Development in which he explains to non-technical people why software tests matter. You might want to skip the TDD-with-Excel example at the beginning and start with Dave's points about technical debt. Good luck.

Proving testing value

[Maxo-Texas]

Before hand.

1) Do you have a ticket/incident system? How many high/critical defects are reported?
You can argue, "We spend 3600 hours a year dealing with these defects- many of which are duplicate reports. " Arguing potential savings is tricky-- how can you prove testing will reduce defects? You need to find a study showing that having a QA department reduces defects to production by "X%". Then you could apply that X% to the 3600 hours and say "We would save 540 hours a year of time and that would cost us $30,000 at $60 per hour"-- so there is your testing budget and a reasonable justification.

2) Have you lost any business? You can argue, "We lost $1 million in revenues (about $100k in profits) because of the lack of testing"

---

There are two kinds of testing
Regression testing- Did we break something. Using a tool like Mercury, or doing it manually, you restore data, run the test steps (automatically or manually) and then compare the database changes to those made prior to the change. Any changes must be explained by some new feature or change or it is a defect.

QA testing- i.e. hostile testing. Most programmers test "ideal path". Good QA tries to break your code and catches errors before a clueless user catches them.

---
Just remember, perfect is the enemy of good. You shoot for perfection, it will never see production. At my company- after we put in QA, it cost us about 2 weeks on a formerly 1 day process. Add on SOX-- and it went to 4 weeks for a formerly 1 day process. A lot of defects and enhancements no longer made financial sense to make unless they were bundled with others to justify the 4 week overhead. (in the end, they got it up to 47 days to make a 1 line change-- at that point they declared the old software obsolete and paid a huge amount of money to have new software developed (without any change control or SOX until it hit production).

The real question: is QA needed?

[davek]

I asked the same question on stack overflow and got some good responses:

http://stackoverflow.com/questions/432512/how-to-convince-management-that-qa-is-important

Of course, there is no silver bullet. It really comes down to you being able to convince them with your charisma and charm (+1 charisma is big bonus). These things rarely show up on the bottom line. If they do, it's 6 months down the road and management will just attribute it to some new rock-star salesman.

Sorry to be a downer, but that's the life of an engineer.

therac 25

[Anonymous Coward]

http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html

Re:Overall cost may not be "in his own best intere

[Anonymous Coward]

durrrrrrrrrrrrrrrrrrrrrrrrr no.

You get a better return if you invest your money for a microsecond than if you invest it for a milisecond, and you get much, much better returns if you invest a million than if you invest a thousand.

You're working at the wrong company

[Rogerborg]

The fix is to work for smarter people.

adding regression tests

[fuliginous]

Your best bet is to simply add tests for each thing you fix. Don't bother with wider testing. But do add a new test every time you break something (for the thing you broke) when fixing another bug.

And if you are adding tests make sure they are for run automatically after changes to the code base so that they are always run shortly after changes are submitted (I'm assuming it's under source control, if not get that sorted first).

business game

[bigdavex]

There's a business game in which people who want to something X way demand that the people who want to do it Y way provide a business case. What's the business case for continuing to develop code without a test suite?

Do you comment your code? Why? was there a business case developed for that?
Did the electricians install circuit breakers in your building? Why? Did anyone sit down and figure out if this was cost effective? Of course not, because circuit breakers are best practice.

Sure, quantify costs and benefits when you can and when the task is large enough to justify it. But when then costs and benefits are not rigorously defined, it's innappropriate to give the status quo a free pass.
   

TDD from here on out

[wonder]

I've been working for a company which does a lot of Test Driven Development (TDD), but quite honestly, we don't develop new products all that often. We have legacy products (10 years old on average) that we work on day in and day out which in some cases are abhorrent. However, we don't try to fix the world by taking a year or two off to add comprehensive tests to the company's products, we just test anything we add or modify. Slowly but surely, you will add value to your products and the stuff you're complaining about will start to happen less often.

I would submit that you don't need to ask for permission to do this. YOU are the developer and it is YOUR mandate and responsibility to produce quality software. YOU know how best to do this, so just do it, and build it into the cost of your estimates. At first, while you get used to it, things may slow a little (not as much as you might worry about). However, once you get rolling in practice we see that things don't take any less time than they did before, and eventually some things will take less time than before. Did i just say that right? In fact I did. The difference is that the same old bugs for the same old reasons simply don't occur anymore. We just have fewer bugs coming in against our products. We KNOW that once we fix something it will stay fixed tomorrow, next release, next year, and for all of time until the spec changes (don't be deluded into thinking it won't...). We have tests that say so and will not let the build pass if at any point that feature no longer behaves as it should. In practice, whenever we add tests to legacy code that we just happen to be modifying, we aren't surprised to find out that it doesn't work the way the specifications say that it should work.

Testing isn't there to speed you up. It's there to not slow you down (among many other wonderful benefits). It's there so that you can say that you *KNOW* that your product works in certain ways. It can be hard to start writing tests, but someone above mentioned a book "Working Effectively With Legacy Code" (Michael C. Feathers), which is an excellent resource to get you started. Here are some others:

- Test Driven Development: By Example (Kent Beck)
- Clean Code: A Handbook of Agile Software Craftsmanship (Robert C. Martin)
- Refactoring: Improving the Design of Existing Code (Martin Fowler et al)

You don't *have* to adopt TDD as a daily life routine to reap the benefits. Some of these books and other reference it as an excellent tool to enable you to write better tests and better code, which i think is true. However, these books also teach you how to tackle existing code and to get it under control by writing tests for it. You're doing yourself and your products, and your employer a disservice if you don't write tests for the code you write, even if your employer cannot see the benefit. I'm sure surgeons could get stuff done a lot faster and cheaper if they took some shortcuts and didn't bother to sterilize their equipment, but they know better than to do that. You should know better as a developer than to just hack something together and run it by some visual inspection of limited "use cases" and then think that you've done your job properly. Your code may not work at all, and you won't even know it until a customer tells you so.

testing is big waste of time

[tp_xyzzy]

In a small company, testing is just waste of time. If you cannot trust your co-workers in a small company enough that they do not break the code, then you will have bigger problems than just lack of testing. Testing is for situations where your programmers are idiots, and you don't trust them enough to keep the code working. It's always a bad situation in small organisation. In large organisation testing is necessary. But below some treshold, once you know everyone involved, testing gives nothing new to your process.

It's programmers responsibility to ensure their code works perfectly in all situations. If a programmer cannot check that all execution paths are actually working, it'll take large amount of work for other people to find the paths and manually test every one of them.

My alternative to testing in small organisation is to teach people to write code correctly. Assuming you're in bad situation where you cannot trust your programmers already -- obviously trust needs to be deserved -- once you start finding errors in the field, the programmer will obviously lose the trust for short period of time.

How do you know it's due to lack of testing?

[DerekLyons]

"Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite"

Why do you assume it's due to a lack of a formal test suite? Why can't it be inadequate communications between team members? (Programmer 'A' fixes a bug, programmer 'B' puts it back while 'fixing' or 'restoring' a different feature.) Why can't it be inadequate commenting? (A bug keeps recurring because nobody quite understands why/how a chunk of code works and keep 'fixing' it.)

Etc... etc...

That you 'assume' it's due to the lack of testing implies to me that you haven't looked very deeply at the whole process and have latched onto 'testing' as some form of a magic wand.

You didn't fix it

[Anonymous Coward]

If the bug keeps coming back, then buddy, you didn't fix it. 'Fixed' means, 'not broken'. If it comes back, then the situation that causes it is ongoing and recurring. Damage control is not the same as making sure it doesn't happen again. You need to fix the bug. And what do you mean by 'undergrads'. Do you have a PhD? And what were they undergrads in: business? Accounting? Some other has-no-business-touching-the-innards-of-a-computer discipline? For that matter, what are your qualifications? How is it that the bug keeps coming back even though you describe it as 'Fixed'? Do you have any business touching the innards of a computer?

Wrong question.

[edw]

"Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"

What bullshit. Perhaps the question you should ask is, "How can I determine whether setting up a test suite is in his best interest?" But if you insist on pursuing an answer to your original question, you might want to dedicate some time to other questions, like figuring out how to convince people that humans and dinosaurs roamed the earth together, or that people's astrological signs play an important role in determining their personalities, or that ingesting colloidal silver is a solution to a wide range of health problems.

You are part of the problem.

Re:WHY do you have to prove software testing saves

[freaq]

Nobody puts up with bad products anymore. Software is one of the last kind of products where it's still somewhat accepted

Judging from the evidence (Windows on damn near every PC sold), it's not somewhat expected, it's demanded. Microsoft has made a very successful business by defining 'beta testing' to mean 'ship it and see how many people sue us'.

Then be prepared

[Anonymous Coward]

to be fired after spending lots of the companies money, while failing to solve any of the issues you have just described.

Craftsmanship & Maintaining Legacy Code

[nukeevry1]

As others have pointed out, your question's premise is inherently flawed because you are begging the question. Any proposed answer could be considered a self-fulfilling prophecy. Here's a more logical way to think about how to get to a cost-effective test suite.

Creating an automated test suite after-the-fact is difficult for a number of reasons, primarily because a well tuned and effective test suite requires constant feedback to determine what works and what doesn't. You can't get that unless you have adopted a "pay as you go" model where your test feedback influences your codebase and vice versa.

By Michael Feathers' definition, a legacy codebase is one without a test suite. I highly recommend you take a look at his book on the topic. http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052/

If you're interested in the most effective way to maintain your legacy code base, you'll build in the appropriate level of testing (including supporting code, infrastructure, etc.) and force yourself to refactor the parts of it that need to be refactored to support testablity. It will drive out the tests for you that do provide value, and steer you away from lines of thought that do not help you catch regressions or drive refactoring. It gets you to "pay as you go." Not only will this provide you the best argument for management to give you time to write tests, it is the most professionally responsible one as well.

Speaking of professionalism, have a look at Bob Martin's arguments for software craftsmanship and the ethical responsibility software developers have to produce quality work. This might help form your opinion on when and why software quality is important to professional responsibility. There are more to ethics than just the bottom line.

cvs annotate

[zigamorph]

Most version control systems have a function to keep track of who did what. Use it to find the idiot and educate it in the error of its ways. If repeated education fails, terminate the idiot or promote it to a non-coding position, depending on corporate culture.

Revision control?

[amorpheous]

Perhaps if fixed bugs periodically re-appear in the product then the revision control system is broken? Software testing is important (regression testing seems especially needed in this case) but so is revision conrol.

It Doesn't Save Money

[stewbacca]

Testing doesn't "save" money any more or less than "training" saves money. They both cost the company money.

Without either, however, you are less likely to receive future revenue because your software will garner the reputation of being bad (because it wasn't tested) and being hard to use (because there is no training).

There's always the old adage that you've got to spend money to make money. I think that holds true in this case.

Re:It Doesn't Save Money

[wrook]

This is really untrue. There are many ways to get from point X to point Y. Some are faster (and hence cost less) than others. Very frequently writing tests (whether unit tests or acceptance tests) will lead to faster development than not writing tests. This is because you can wade into hard to understand code without worrying so much about breaking things. You can be less careful about replacing code that doesn't make sense any more. You can maintain a smaller, easier to understand codebase with less duplication of functionality. The result is that your average coding throughput is higher (even if occasionally your instantaneous productivity might be lower). When measured over a few months, I can definitely write new code with tests faster than I can write new code without tests.

However, taking a large legacy application and putting it under sufficient tests to improve your throughput is likely to cost time and money in the short to medium term. It may not be feasible for the business case that they have.

Re:WHY do you have to prove software testing saves

[stewbacca]

I'm frequently amused at such great test procedures that catch things like my 12 cup coffee resevoir on my 10 cup coffee pot. I love me some 2 cups of over filled coffee in the mornings!

And how about the "testing" that goes into ovens/ranges/microwaves? It's as if those things get a pass when it comes to usability. Shit, those things can blow your house up and kill your entire family if not used correctly, but they have the lowest UI standards (tied with cars!) on the market.

As one of my design expert coworkers frequently says--it doesn't cost any more money to design something to work correctly, but it does cost more to fix a bad design. Example, my 12 cup coffee maker with a 10 cup pot. It wouldn't have cost any less money to limit the resevoir to 10 cups.

Re:WHY do you have to prove software testing saves

[stewbacca]

My company makes software for the government. It's amazing at how high the government's tolerance level is for crap.

AC asks how to learn testing?

[adriccom]

Software testing is an entire profession including having its own graduate programs, but there are lots of resources to help you get started from books and online, just poke around.

There are books just about testing (TCS), books about integrating testing into a development methodology(Agile and Scrum include testing), and plenty of books on specific testing technologies (JUnit, Cucumber, ...). Most modern languages/toolkits include at least some support for basic software testing (unit or functional) such as Perl, Python, Ruby or have it readily available such as JUnit for Java, NUnit for C#. For testing web applications go look at Selenium, a great package of tools for web testing that includes browser plugins.

And *plug* I've had great experiences with the online resources including low-cost online classes available from the AST, at http://www.associationforsoftwaretesting.org/ The BBST courses are very informative and quite challenging. */plug*

hth,
adric

Try a short pilot project. Get real data quickly.

[Hexawise]

1) Track the amount of time it takes to create and execute a set of tests. (A few days' worth, not a few weeks' worth) 2) Measure how many bugs were identified. 3) Do a quick calculation: what was the approximate amount of time spent? What was the approximate value of finding those bugs then vs. in production (or a later stage of testing)? It is fine to use rough estimates here, but make your assumptions explicit. The findings will be useful to a manager making a decision about where to spend money; where to devote limited resources. My advice is to put your faith in hard data, not "HiPPO's" / the "Highest Paid Person's Opinion." Amazon took that approach when it came to their automatic recommendations of products for users; it worked pretty well for them. For more on the HiPPO point, see, e.g., http://hexawise.wordpress.com/2009/08/18/learning-using-controlled-experiments-for-software-solutions/ Also, at the risk of stating the obvious, not all kinds of testing will deliver equal benefits. Good test design (the approach you take to identifying which software tests to execute when), will have a huge impact vs. poor test design approaches. The URL above has information on that point too. - Justin Hunter

Not in the real world.

[dtmos]

In the real world, the longer you let someone hold your money, the better return you get. (Note the increasing coupon with increasing maturity.) This isn't some kind of academic use of the Law of Compound Interest, or some Wall Street quant's way of finding a microsecond's inequality in the market (which is, of course, the exception that proves the rule; it's called an "inequality" precisely because it's an unusual situation in which the short-term return is unexpectedly high).

Also, I did say that "[y]ou have better options and get a better return on your investment if you invest $100,000 than if you invest $1,000." It's even more true if you invest a million -- assuming you have one, of course, which is the subject under discussion.

Re:Overall cost may not be "in his own best intere

[Anonymous Coward]

This is the most eloquent explanation I have read on Slashdot for money a moon.

Well done, sir.

Re:WHY do you have to prove software testing saves

[tibit]

Another idea that I just don't see all that often: continuous product improvement. One doesn't have to spend the money and effort on coming up with a completely new piece of crap that does exactly what a previous model did or didn't do. Too many consumer products are fire and forget. Once the production run is done, it's done. And I'm not even talking about the electronics. Simple things like strollers: they seem to absolutely have to introduce 'new models', where some design flaws surface every couple of product generations. Like if the design teams had amnesia or something.

Test smarter, not harder

[Hexawise]

One method of test design that I have personally seen work much better than standard manual methods of determining what tests to execute is pairwise and combinatorial test design. In my view, not nearly enough developers or software testers know about it. Here are some good articles about it here: http://www.combinatorialtesting.com/clear-introductions-1 This article, published in IEEE Computer, describes this test design approach and the results of a ten project study that showed this test design selection method found more than twice as many defects per tester hour as traditional, manual test case design methods. http://app.hexawise.com/Combinatorial-Softwar-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf - Justin Hunter

Re:Overall cost may not be "in his own best intere

[dtmos]

I like that development method, too, but it does move the variable from "ship date" to "customer features". Often, as Spolsky promotes, it's a good way of doing things, since the features are not promised in advance. Sometimes, however, projects start with an agreement on just what the features are to be, and it's difficult some time later to make the unilateral decision to delay them to later releases. Without, of course, affecting payment schedules and company cash flow....

I have seen variants in which a company has two development teams, "June" and "December". Team June starts its project every 1 July, and ships every 30 June. Team December starts its project every 1 January, and ships every 31 December. If a team runs into a development snag that threatens to delay it past its due date, the feature or technology is excised from the project and placed with the following team, which then has an additional six months to work the problem. This works well for both hardware and software development (frequent new product introductions, snazzy features that work when introduced, etc.), if one has the cash to simultaneously fund two development teams.

Re:Overall cost may not be "in his own best intere

[dtmos]

Merci pour vos commentaires, et toutes mes excuses pour mon mauvaise connaissance de la langue française.

The Overhead

[SuperKendall]

All the developer has to do is keep the pass rate at 100%, which is only overhead when they reintroduce a bug or rework the design

Or the test is making an assumption that is invalid in relation to some new code change, so instead of just fixing one bug you fix two. That happens even when you aren't re-working, just repairing.

That's why I advised him to try maintaining a small test suite first to see if it catches bugs more frequently than you have to maintain the tests.

There's also the overhead for same test suites of keeping fully in sync with what goes in the main build, when you add new files etc.

Way cheaper than waiting for someone to find the bug later on.

Actually that part is free.

And if it's a bug they guy has seen before, the developer can fix it very quickly. That's really low cost too.

It's hard to find where it pays off to spend a ton of effort building a formal test suite when all that might just happen anyway...

Re:The Overhead

[Fulcrum of Evil]

Or the test is making an assumption that is invalid in relation to some new code change, so instead of just fixing one bug you fix two. That happens even when you aren't re-working, just repairing.

Sounds like it's documenting how things should actually work, which is something you should be doing anyway.

That's why I advised him to try maintaining a small test suite first to see if it catches bugs more frequently than you have to maintain the tests.

I don't recall saying anything about size.

There's also the overhead for same test suites of keeping fully in sync with what goes in the main build, when you add new files etc.

So you update your tests when the underlying code changes. Not seeing a problem.

And if it's a bug they guy has seen before, the developer can fix it very quickly. That's really low cost too.

Until the dev switches departments or jobs or just forgets. These things happen.

It's hard to find where it pays off to spend a ton of effort building a formal test suite when all that might just happen anyway...

So don't spend a ton. And you really get to choose - either the code is tested or you don't know what the hell it's doing.

Re:The Overhead

[SuperKendall]

Sounds like it's documenting how things should actually work, which is something you should be doing anyway.

Yes, but still extra work beyond what they WOULD be doing. There are many, many things in software development that SHOULD be done, where only a percentage of them are.

Forced work is NEVER a good thing because it takes resources from more important tasks.

So you update your tests when the underlying code changes. Not seeing a problem.

You have to (a) see the tests are failing, (b) repeat the failure (c) determine the failure is a false positive. All that takes time, again it's taking time from potentially much higher priority things. Instead of working on code that DOES SOMETHING FOR USERS, you are repairing code that never would have broken if you didn't write it and wouldn't have mattered if it were not there (in the case of a false positive). That is a negative aspect, and that is what you must monitor for. If that becomes too extreme that test is a vampire, and should be eliminated.

The thing about the testing people, is that they never can comprehend the negative drain things like false positives take. I worked on some projects that were destroyed because the people working on them were so adherent to making sure there were many tests that all passed... while the software never worked. If they'd taken the time to produce working software instead of working tests the product would have been fine (as it was once the test-focused people were ejected).

Re:The Overhead

[Fulcrum of Evil]

Yes, but still extra work beyond what they WOULD be doing.

You mean like their job? Or are you suggesting that docs and specs are for other people?

You have to (a) see the tests are failing, (b) repeat the failure (c) determine the failure is a false positive.

a, b are automatic. It sends you mail and you can run the test locally. C is a judgment call, and if you're getting too many of those you have too many tests. Write fewer tests that exercise important stuff that can be broken.

Instead of working on code that DOES SOMETHING FOR USERS, you are repairing code that never would have broken if you didn't write it and wouldn't have mattered if it were not there

Users like not having the same thing break that broke last month, for instance. This is maintenance level work, akin to keeping the lights on. Now then, if you weren't writing a bug, you wouldn't have to repair it, would you?

The thing about the testing people, is that they never can comprehend the negative drain things like false positives take.

The thing about you is that you make all sorts of assumptions in order to avoid having to consider a new way of doing things. I never said that tests need to be heavy or complete. That's all you. I have a test framework, it's light and easy to deal with, and it allows me to proceed with confidence in the face of changing code and new devs.

If they'd taken the time to produce working software instead of working tests the product would have been fine (as it was once the test-focused people were ejected).

To paraphrase, you had some idiots who were big into testing, so now testing is a waste of time. Right.

Re:The Overhead

[SuperKendall]

You mean like their job? Or are you suggesting that docs and specs are for other people?

The job is to make software for whoever uses it. Docs and specs are good but are secondary to the purpose of work, and there is not always time for them.

The whole reason Apple succeeds is because they realize this one fact, they are designing software for people to use. Not designing software to be a perfect artifact in and of itself.

a, b are automatic. It sends you mail and you can run the test locally.C is a judgment call,

One hour.

If you're getting too many of those you have too many tests. Write fewer tests that exercise important stuff that can be broken.

Reduction of tests, half a day. Time to write tests it turned out you didn't need, three days.

Congratulations, you have found your overhead.

Users like not having the same thing break that broke last month, for instance.

That depends on what has broken. If it's in a part rarely used, they don't really care. If it's a visual flaw, they may not care.

This is maintenance level work, akin to keeping the lights on. Now then, if you weren't writing a bug, you wouldn't have to repair it, would you?

You have to repair it regardless of if a user finds it or if testing finds it.

I never said that tests need to be heavy or complete. That's all you.

Nor did I. I'm telling you what some people think testing is.

I have a test framework, it's light and easy to deal with, and it allows me to proceed with confidence in the face of changing code and new devs.

So do I, although often I can proceed with confidence in untested areas simply because I understand the code well enough. That too is an option.

To paraphrase, you had some idiots who were big into testing, so now testing is a waste of time. Right.

You are too sensitive. I never said testing was a waste of time, for everyone. I said only that testing CAN be a waste of time, that time CAN possibly be better spent on other tasks.

The thing about gung-ho testing people is they seem rather unwilling to admit there's any scenario in which testing can be bad. YOu are not one of them; you pointed out there can be too many tests, that testing can get too heavy. That's good, but you are unusual. Most people that are very keen on testing are unwilling to admit to any possible downside.

I am only pointing out that I have seen testing have a negative effect in the real world, so it is possible. I have also seen working testing systems that helped a great deal, so I know that is possible also (in fact my first "real" job outside of college was writing tests in a dedicated QA department that did a great job keeping software stable).

I am also saying that you need to evaluate the conditions at hand to see what level of testing makes sense, and that sometimes the level is zero.

Write tests for new code

[nut]

If you can't convince your boss to spend the time and money to write tests for old working code, just start writing tests for any new code you write.

When you fix a bug, write a test for it.

When you add a feature write a test suite for it.

Your tests will also incidentally test old code near the new code, and your coverage will increase surprisingly quickly.

I had a team of 6 developers doing this over ~400,000 LOC over the course of about 18 months and got 60% code coverage over the product.

Poor Quality Doesn't last

[Anonymous Coward]

Testing is about making a quality product and it does cost money. However tech is littered with million and billion dollar companies that are worth pennies with what they once were because of poor quality. Gateway had great branding, a good business model but couldn't build a quality box (Dell slowly ran them over). Netscape owned most of the browser market and but just doesn't exist anymore. It's possible to coast for awhile with bad quality, but that doesn't last.

depends on the testers

[Anonymous Coward]

I currently have a problem where I'm doing development AND testing because the "testers" we have are useless. A good tester is worth their weight in gold. Good testers will save your company money for sure - bad ones will just be expensive.

I guess the same can be said for programmers.

If you tell them that it will "save the money" and they get idiots as testers -- everything will fall apart.

You can't measure that w/ metrics -- but I can tell you that is what I've learned from being in this industry for the past 15 years.

The boss may be right

[chrismcb]

One thing you might want to look at is why are those bugs coming back? Did you miss something when you fixed it, or is it a regression (that is did another bug fix break it?) Unit tests will help with the later. Even without permission you can write unit tests as you fix new bugs. As far as the former goes, as long as your customers aren't complaining, your boss is probably right. You refix a bug every couple of months? If your users don't seem to mind, the cost of setting up a formal test suite is probably more than the cost of you fixing a bug every few months.

Not necessarily in "best" interests...

[lpq]

If he always fixes bugs and provides support 'gratis', then sure, it might be in his best interests. But that's not a requirement of the market and not something most companies do. Look at Microsoft. If you get Windows w/your computer and have a Windows problem, MS will charge you either per-incident or per-hour, depending on customer type on type of service you choose. MS shrugs off responsibility and costs for support by saying that any support for the over 90% of windows copies that come on OEM machines is supposed to come from your machine manufacturer.

Dell -- charges hundreds of dollars for software (i.e. MS Windows) incidents. You can buy single or multi packs but they aren't cheap.

Last company I worked for -- I was told by my manager NOT to fix any bugs -- (I was marked down on a review for fixing some while I was working on the code for other reasons). Bugs were only to be fixed when they were reported by a support-paying customer. If I fixed them in my spare time -- I was hurting the maintenance-team's support revenues.

You see, in most software companies, support is treated as its own profit and loss center. They only way it can make money is by getting customers to pay support and pay for the bug fixes. If there are too few bugs to fix, (or too customers reporting bugs), it can mean layoffs and cost jobs.

So your idea of creating a test suite, while it might have been a good idea in the 70's and early 80's. after Reagan's "every man for themselves, every department for themselves, corporate profit before all else (including employees and customers)" philosophy was adopted into the culture (with support of repealed worker and customer protections) in the early 80's, making quality software just wasn't profitable and didn't make sense in the business market.

So.... You want quality SW? You might consider working for a place where quality is required as part of the final product (usually due to financial or other incentives) such as, perhaps, the medical market (where bugs might cost lives and mean large lawsuits), or places that are not simply motivated by next quarter's financial results (maybe government funded?)...

Recurring Bugs = bad CM

[ghalasdie]

Recurring bugs usually means that you're failing to properly manage source code commits. Testing is just another way to discover when you've screwed that up. You should have a process that informs developers where they should pull their code from and how they can go about pushing their changes back into the code base without undoing someone else's work. Of course, even if you pulled from the right source, your changes can break things, so you still should be testing. I agree with what's been said above about slowly building up an automated test suite. Just bake it into your development. If the recurring bugs are genuinely due to poorly engineered products, then, well... You're in a tough spot. You can spend a lot of time polishing a turd, but there's only so much shine you're going to get out of it.

Apple

[dgriff]

Hey, even Apple don't bother testing their iPhone alarm software so good luck convincing your owner!

Well intentioned, yes...

[gillbates]

The problem is not that testing is bad, but that the break even point for test suites is rather elusive.

A general rule of thumb is that you will write ten lines of code for every line of code under test. While writing test suite code does proceed faster than the main code, it still can take long enough to sink a project.

My particular experience involved writing a test suite for just a dozen or so requirements. The test cases written took just 4 hours. Learning how to create a makefile with the broken emake system (another rant...) took more than 8. Writing the test suite framework took about 180 to 200 hours.

In retrospect, the test suite took far more time to develop than the code which it tested. More importantly, even though adding test cases is now trivial, it will likely never be done in the lifetime of the software, as the firm has a policy against writing reusable code. Worse still, the entire feature set would have taken less time to test manually than it took to write the test suite.

The takeaway? It takes a *very* large project to make a test suite worthwhile. Typically, you need a project so large that miscommunication between groups of programmers is a very likely source of bugs. A small software shop will typically not produce software of the complexity or importance to justify the expense of a test suite. (Aviation and other safety applications excluded, of course.)

My gut feeling, based solely on my experience, is that the break-even point for writing a test suite is when you expect to produce greater than 250k of new code on a project, or are maintaining a codebase of greater than 10M lines of code and expect to add more than 50k lines per quarter. At this level, you can probably dedicate a programmer or two to doing nothing but writing the test suite.

Re:Exactly the same way you prove anything like th

[An Onerous Coward]

Wouldn't it be faster to just make the numbers up?

I wish I were kidding.

Automated Testing

[LifesABeach]

Fingers can get tired, one can forget certain tests. An automated tester can save your fingers, and do every test one can conceive; including the ones that your clients have created. These reasons are enough for one to save time and money with. And programmers like myself can go home knowing that most the bugs will be accounted for.