Automated Software QA/Testing?

nailbite writes "Designing and developing software has been my calling ever since I first used a computer. The countless hours/days/months spent on imagining to actualizing is, to me, enjoyable and almost a form of art or meditation. However, one of the aspects of development that sometimes "kills" the fun is testing or QA. I don't mind standalone testing of components since usually you create a separate program for this purpose, which is also fun. What is really annoying is testing an enterprise-size system from its UIs down to its data tier. Manually performing a complete test on a project of this size sucks the fun out of development. That's assuming all your developers consider development as fun (most apparently don't). My question is how do you or your company perform testing on large-scale projects? Do you extensively use automated testing tools, and if so, can you recommend any? Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)"

Comments

Manual Testing

[Egonis]

Although I haven't personally used many testing tools:

I created an Enterprise Application composed of Client/Server Apps -- the best test was a small deployment of the Application to users who are apt to help you conduct the test, over a few weeks, I found bugs I never caught with my own manual tests.

Applications that test your code, etc are great from what I have heard, but will not catch Human Interface related issues, i.e. GUI Mess-Ups, Invisible Buttons, etc.

Test Matrix

[Pedro Picasso]

I've used auto-test thingies, ones that I've written, and packaged ones. Some situations call for them. Most of the time, though, it's just a matter of doing it by hand. Here's what I do.

Create a list of inputs that includes two or three normal cases as well as the least input and the most input (boundaries). Then make a list of states the application can be in when you put these values into it. Then create a graph with inputs as X and states as Y. Print your graph and have it handy as you run through the tests. As each test works, pencil in a check mark in the appropriate place.

Now that you've automated the system to the point where you don't need higher brain functions for it, get an account on http://www.audible.com, buy an audio book, and listen to it while you run through your grid. It still takes a long time, but your brain doesn't have to be around for it.

This is going to sound incredibly elementary to people who already have test methodologies in place, but when you need to be thorough, nothing beats an old fashioned test matrix. And audiobooks are a gift from God.

(I'm not affiliated with Audible, I just like their service. I'm currently listening to _Stranger in a Strange Land_ unabridged. Fun stuff.)

Re:nothing to see here, move along.

[Anonymous Coward]

How about we read books on the subject written by software engineering researchers and not programming language researchers? See the Dynamic Analysis lecture notes.

Use a QA team and test-driven development

[William Tanksley]

You need two things: first, people who are dedicated to testing and aren't concerned merely to uphold their pride in the code they wrote (this is a long way to say that you need a dedicated testing team that doesn't report to the coding team); and second, testable code. The best way to get the second needed item, in my experience, is to have your developers write their automated unit tests BEFORE they write the unit they're developing.

This is generally called "test-first" development. If you follow it, you'll find some nice characteristics:

1. Each unit will be easily testable.
2. Each unit will be coherent, since it's easier to test something that only does one thing.
3. Units will have light coupling, since it's easier to express a test for something that depends only lightly on everything else.
4. User interface layers will be thin, since it's hard to automatically test a UI.
5. Programmers will tend to enjoy writing tests a bit more, since the tests now tell them when they're done with their code, rather than merely telling them that their code is still wrong.

You can go a step further than this, and in addition to writing your tests before you write you code, you can even write your tests as you write your design. If you do this, your design will mutate to meet the needs of testing, which means all of the above advantages will apply to your large-scale design as well as your small-scale units. But in order to do this you have to be willing and able to code while you're designing, and many developers seem unwilling to combine the two activities in spite of the advantages.

-Billy

Testing

[Jerf]

Lemma One: It is impossible to comprehensively test the entire system manually.

Proof: For any significantly sized system, take a look at all the independen axes it has. For instance: The set of actions the user can take, the types of nouns the user can manipulate, the types of permissions the user can have, the number of environments the user may be in, etc. Even for a really simple program, that is typically at least 5 actions, 20 nouns, (lets estimate a minimal) 3 permission sets (no perm for the data, read only, read & write), and well in excess of 5 different environments (you need only count relevant differences, but this includes missing library A, missing library B, etc.). Even for this simple, simple program, that's 5*20*3*5, which is 1,500 scenarios, and yes, you can never be sure that precisely one of those will fail in a bad way.

Even at one minute a test, that's 25 hours, which is most of a person-week.

Thus, if you tested a enterprise class system for three days, you did little more than scratch the surface. Now, the "light at the end of the tunnel" is that most systems are not used equally across all of their theoretical capabilities, so you may well have gotten 90%+ of the use, but for the system itself, a vanishing fraction of the use cases. Nevertheless, as the system grows, it rapidly becomes harder to test even 90%.

(The most common error here is probably missing an environment change, since almost by definition you tested with only one environment.)

Bear in mind that such testing is still useful, as a final sanity check, but it is not sufficient. (I've seen a couple of comments that say part of the value of such testing is getting usability feedback; that really ought to be a seperate case, both because the tests you ought to design for usability are seperate, and because once someone has fuctionally tested the system they have become spoiled with pre-conceived notions, but that is better than nothing.)

How do you attack this problem? (Bear in mind almost nobody is doing this right today.)

1. Design for testability, generally via Unit Tests. There are architectures that make such testing easy, and some that make it impossible. It takes experience to know which is which. Generally, nearly everything can be tested, with the possible exception of GUIs, which are actually provide a great example of my "architecture" point.

   Why can't you test GUI's? In my experience, it boils down to two major failings shared by nearly all toolkits:

   1. You can not insert an event, such as "pressing the letter X", into the toolkit programmatically, and have it behave exactly as it would if the user did it. (Of the two, this is the more importent.)
   2. You can not drive the GUI programmatically without its event loop running, which is what you really want for testing. (You need to call the event engine as needed to process your inserted events but you want your code to be in control, not the GUI framework.) One notable counterexample here is Tk, which at least in the guise of Python/Tk I've gotten to work for testing without the event loop running, which has been great. (This could be hacked around with some clever threading work, but without the first condition isn't much worth it.)

   The GUIs have chosen an architecture that is not conducive to testing; they require their own loop to be running, they don't allow you to drive them programmatically, they are designed for use, not testing. When you find a GUI that has an architecture at least partially conducive to testing, suddenly, lo, you can do some automated tests.
   
   And in my case, I am talking serious testing that concerns things central to the use of my program. I counted 112 distinct programmatic paths that can be taken when the user presses the "down" key in my outliner. I was able to write a relatively concise test to cover all cases. Yes, code I thought was pretty good turned out to fail two specific cases (

Re:You're not alone.

[anonymous cowherd (m]

Heh. I'm a software tester, and my boss, who actually is a Native American, put it to us this way: You guys are the cavalry at Little Bighorn, and the whole rest of IT is the Indians.

Basically, what he was trying to get across to us was that as long as we can keep the BS somewhat at bay, we can do our jobs, but the second it gets out of hand, 2000 Indians are gonna be on your ass.

To be fair, not all the development guys are like that, but some of them are.