Automated Software QA/Testing?

nailbite writes "Designing and developing software has been my calling ever since I first used a computer. The countless hours/days/months spent on imagining to actualizing is, to me, enjoyable and almost a form of art or meditation. However, one of the aspects of development that sometimes "kills" the fun is testing or QA. I don't mind standalone testing of components since usually you create a separate program for this purpose, which is also fun. What is really annoying is testing an enterprise-size system from its UIs down to its data tier. Manually performing a complete test on a project of this size sucks the fun out of development. That's assuming all your developers consider development as fun (most apparently don't). My question is how do you or your company perform testing on large-scale projects? Do you extensively use automated testing tools, and if so, can you recommend any? Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)"

Comments

You shouldn't be doing it

[dirk]

The first thing you need to learn is that you shouldn't be doing large scale testing on your own systems. That is just setting yourself up for failure, since the only real testing is independent testing. Preferably you should have full-time testers who not only design what needs to be tested, but how the testing will be done and who will do the actual testing. Where I work, we have 2 testers who write up the test plans, and then recruit actual users to do the testing (because they can then not only get some exporsure to the system, they can suggest any enhancements for the next version). Testing your own work is a huge no-no, as you are much more likely to let small things slide than an independent tester is.

Re:You shouldn't be doing it

[wkitchen]

Testing your own work is a huge no-no, as you are much more likely to let small things slide than an independent tester is.

Yes. And also because you can make the same mental mistakes when testing that you did when developing.

I once worked with a dyslexic drafer. He generally did very good work, except that his drawings often had spelling errors. When he looked at his own drawings, the errors were not obvious to him like they were to everyone else. Most people don't have such a pronounced systematic tendency to make some particular error. But we all occasionally make mistakes in which we were just thinking about something wrong. And those mistakes are invisible to us because we're still thinking the same wrong way when we review it. So having work checked by someone other than the one who created it is a good practice for just about any endeavor, not just software.

As a former professional software tester ...

[eddison_carter]

Nothing can compare to having a dedicated test staff. At the last software place I worked (part-time, in school, while getting my engineering degree), we had 3-6 college students working on testing most of the time (we would also be given some small projects to work on).

Testing goes far beyond what any automated system can test, if you have a user in there somewhere. You also need to check things like "How easy is it to use?" and "Does this feature make sense?". We also suggested features that the program did not have, but from our experiance using it, thought that it should have.

QA is a separate function

[drgroove]

Outside of unit testing and limited functional testing, developers should be doing QA on their own code. That's a bit like a farmer certifying his own produce as organic, or a college student awarding themselves a diploma. It misses the point. QA function, automated, regression et al testing is the responsibility of a QA department. If your employer is forcing you to perform QA's functions, then they obviously don't "get it".

Test Matrix

[Pedro Picasso]

I've used auto-test thingies, ones that I've written, and packaged ones. Some situations call for them. Most of the time, though, it's just a matter of doing it by hand. Here's what I do.

Create a list of inputs that includes two or three normal cases as well as the least input and the most input (boundaries). Then make a list of states the application can be in when you put these values into it. Then create a graph with inputs as X and states as Y. Print your graph and have it handy as you run through the tests. As each test works, pencil in a check mark in the appropriate place.

Now that you've automated the system to the point where you don't need higher brain functions for it, get an account on http://www.audible.com, buy an audio book, and listen to it while you run through your grid. It still takes a long time, but your brain doesn't have to be around for it.

This is going to sound incredibly elementary to people who already have test methodologies in place, but when you need to be thorough, nothing beats an old fashioned test matrix. And audiobooks are a gift from God.

(I'm not affiliated with Audible, I just like their service. I'm currently listening to _Stranger in a Strange Land_ unabridged. Fun stuff.)

Automation is difficult

[TheSacrificialFly]

It's extremely difficult to develop and maintain on any enterprise size system. One of the big problems management has with automation I've found is that once they've put the money into initally developing the automation, they think it will run completely automatically forever more.

From my personal experience at one of the world's largest software companies, automation maintenance for even a small suite (200-300 tests, 10 dedicated machines) is a full time job. That means one person's entire responsibility is making sure the machines are running, the tests aren't returning passes and fails for reasons other than they are actually running the tests, and making changes to the automation both when either the hardware or software changes. This person must know the automation suite as well as the tests attempting to be performed intimately, and must also be willing to spend his days being a lab jockey. It's usually pretty difficult to find these people.

My point here is that even after spending many dev or test hours developing automation, in no way is it suddenly complete. There is no magic bullet to replace a human tester, the only thing you can do is try and improve his productivity by giving him better tools.

-tsf

Re:Automation is difficult

[Bozdune]

Parent has it right. Most automation efforts fail because the test writers can't keep up with the code changes, and not many organizations can pay QA people what you need to pay them if you expect them to be programmers (which is what they need to be to use a decent automation tool). Plus, one refactoring of the code base, or redesign of the UI without any refactoring of the underlying code, and the testers have to throw out weeks of work. Very frustrating.

Even in the best case, automation scripts go out of date very quickly. And, running old scripts over and over again seldom finds any bugs. Usually nobody is touching the old functions anyway, so regression testing is largely pointless (take a lude, of course there are exceptions).

I think the most promising idea on improving reliability I've seen in recent years is the XP approach. At least there are usually four eyes on the code, and at least some effort is being put into writing unit test routines up front.

I think the least promising approach to reliability is taken by OO types who build so many accessors that you can't understand what the hell is really going on. It's "correctness through obfuscation." Reminds me of the idiots who rename all the registers when programming in assembly.

6 year experience in QA

[cemaco]

I worked 6 years as a Quality Assurance Specialist. You cannot avoid manual testing of a product. Standard practice is to manually test any new software and automate as you go along, to avoid having to go over the same territory each time there is a new build. You also automate specific tests for bugs found after they are fixed, to make sure they don't get broken again. My shop used Rational Robot from IBM. There are a number of others, Silk is one I have heard of, but never used. Developers often have an attitude that Q.A. is only a necessary evil. I think part of it is because it means admitting that they can't write perfect code. The only people I have seen treated worse are the help desk crowd. (another job I have done in the past). The workload was terrible and when layoff time came, who do you think got the axe first? As for developers doing their own testing? That would help some but not all that much. You need people with a different perspective.

Automation versus Manual Testing

[kafka47]

At my company, we have a small QA group that tests several enterprise client-server applications, including consumer-level applications on multiple platforms. To exhaustively test all of the permutations and platforms is literally impossible, so we turn to automation for many of the trivial tasks. We've developed several of our own automation harnesses for UI testing and for API and data verif. testing. The technologies that we've used :
- Seque's silktest
- WinRunner
- WebLoad
- Tcl/Expect

There are *many many* problems with large-scale automation, because once you develop scripts around a particular user interface, you've essentially tied that script to that version of your application. So this becomes a maintenance problem as you go forward.

One very useful paradigm we've employed in automation is to use it to *prep* the system under test. Many times its absolutely impossible to create 50,000 users, or 1,000 data elements without using automation in some form. We automate the creation of users, we automate the API calls that put the user into a particular state, then we use our brains to do the more "exotic" manual testing that stems from the more complex system states that we've created. If you are to embark on automating your software, this is a great place to start.

Hope this helps.

Use a QA team and test-driven development

[William Tanksley]

You need two things: first, people who are dedicated to testing and aren't concerned merely to uphold their pride in the code they wrote (this is a long way to say that you need a dedicated testing team that doesn't report to the coding team); and second, testable code. The best way to get the second needed item, in my experience, is to have your developers write their automated unit tests BEFORE they write the unit they're developing.

This is generally called "test-first" development. If you follow it, you'll find some nice characteristics:

1. Each unit will be easily testable.
2. Each unit will be coherent, since it's easier to test something that only does one thing.
3. Units will have light coupling, since it's easier to express a test for something that depends only lightly on everything else.
4. User interface layers will be thin, since it's hard to automatically test a UI.
5. Programmers will tend to enjoy writing tests a bit more, since the tests now tell them when they're done with their code, rather than merely telling them that their code is still wrong.

You can go a step further than this, and in addition to writing your tests before you write you code, you can even write your tests as you write your design. If you do this, your design will mutate to meet the needs of testing, which means all of the above advantages will apply to your large-scale design as well as your small-scale units. But in order to do this you have to be willing and able to code while you're designing, and many developers seem unwilling to combine the two activities in spite of the advantages.

-Billy

Testing

[Jerf]

Lemma One: It is impossible to comprehensively test the entire system manually.

Proof: For any significantly sized system, take a look at all the independen axes it has. For instance: The set of actions the user can take, the types of nouns the user can manipulate, the types of permissions the user can have, the number of environments the user may be in, etc. Even for a really simple program, that is typically at least 5 actions, 20 nouns, (lets estimate a minimal) 3 permission sets (no perm for the data, read only, read & write), and well in excess of 5 different environments (you need only count relevant differences, but this includes missing library A, missing library B, etc.). Even for this simple, simple program, that's 5*20*3*5, which is 1,500 scenarios, and yes, you can never be sure that precisely one of those will fail in a bad way.

Even at one minute a test, that's 25 hours, which is most of a person-week.

Thus, if you tested a enterprise class system for three days, you did little more than scratch the surface. Now, the "light at the end of the tunnel" is that most systems are not used equally across all of their theoretical capabilities, so you may well have gotten 90%+ of the use, but for the system itself, a vanishing fraction of the use cases. Nevertheless, as the system grows, it rapidly becomes harder to test even 90%.

(The most common error here is probably missing an environment change, since almost by definition you tested with only one environment.)

Bear in mind that such testing is still useful, as a final sanity check, but it is not sufficient. (I've seen a couple of comments that say part of the value of such testing is getting usability feedback; that really ought to be a seperate case, both because the tests you ought to design for usability are seperate, and because once someone has fuctionally tested the system they have become spoiled with pre-conceived notions, but that is better than nothing.)

How do you attack this problem? (Bear in mind almost nobody is doing this right today.)

1. Design for testability, generally via Unit Tests. There are architectures that make such testing easy, and some that make it impossible. It takes experience to know which is which. Generally, nearly everything can be tested, with the possible exception of GUIs, which are actually provide a great example of my "architecture" point.

   Why can't you test GUI's? In my experience, it boils down to two major failings shared by nearly all toolkits:

   1. You can not insert an event, such as "pressing the letter X", into the toolkit programmatically, and have it behave exactly as it would if the user did it. (Of the two, this is the more importent.)
   2. You can not drive the GUI programmatically without its event loop running, which is what you really want for testing. (You need to call the event engine as needed to process your inserted events but you want your code to be in control, not the GUI framework.) One notable counterexample here is Tk, which at least in the guise of Python/Tk I've gotten to work for testing without the event loop running, which has been great. (This could be hacked around with some clever threading work, but without the first condition isn't much worth it.)

   The GUIs have chosen an architecture that is not conducive to testing; they require their own loop to be running, they don't allow you to drive them programmatically, they are designed for use, not testing. When you find a GUI that has an architecture at least partially conducive to testing, suddenly, lo, you can do some automated tests.
   
   And in my case, I am talking serious testing that concerns things central to the use of my program. I counted 112 distinct programmatic paths that can be taken when the user presses the "down" key in my outliner. I was able to write a relatively concise test to cover all cases. Yes, code I thought was pretty good turned out to fail two specific cases (

The Mythical Man-Month

[DragonHawk]

When Fred Brooks published his book, The Mythical Man-Month, one of the things he noted was that testing should acount for *more then half* of the budget of a software project. Actual design and coding should be the minority. This is because software is complex, inter-related, easy to do wrong, and not obvious when it is done wrong.

Of course, nobody wants to do that, because it's expensive and/or boring. Thus we have the state of software today. Just like we had the state of software back in 1956 when he wrote the book.

It never ceases to amaze me that we're still making the same exact mistakes, 50 years later. If you work in software engineering, and you haven't read The Mythical Man-Month, you *need* to. Period. Go do it right now, before you write another line of code.