Slashdot Mirror
Automated Software QA/Testing?
nailbite writes "Designing and developing software has been my calling ever since I first used a computer. The countless hours/days/months spent on imagining to actualizing is, to me, enjoyable and almost a form of art or meditation. However, one of the aspects of development that sometimes "kills" the fun is testing or QA.
I don't mind standalone testing of components since usually you create a separate program for this purpose, which is also fun. What is really annoying is testing an enterprise-size system from its UIs down to its data tier. Manually performing a complete test on a project of this size sucks the fun out of development. That's assuming all your developers consider development as fun (most apparently don't).
My question is how do you or your company perform testing on large-scale projects? Do you extensively use automated testing tools, and if so, can you recommend any? Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)"
how about we go back to basics and read the proper books on computer science? no need for your shmancy-fancy-'voice debugged'-automagically-'quality assured' offerings, thanks.
i'll stick with The Practice of Programming. at the very least i trust the people who wrote it to have a better judgement.
The first thing you need to learn is that you shouldn't be doing large scale testing on your own systems. That is just setting yourself up for failure, since the only real testing is independent testing. Preferably you should have full-time testers who not only design what needs to be tested, but how the testing will be done and who will do the actual testing. Where I work, we have 2 testers who write up the test plans, and then recruit actual users to do the testing (because they can then not only get some exporsure to the system, they can suggest any enhancements for the next version). Testing your own work is a huge no-no, as you are much more likely to let small things slide than an independent tester is.
"Information wants to be expensive" - Stewart Brand, the same guy who said "Information wants to be free"
Nothing can compare to having a dedicated test staff. At the last software place I worked (part-time, in school, while getting my engineering degree), we had 3-6 college students working on testing most of the time (we would also be given some small projects to work on).
Testing goes far beyond what any automated system can test, if you have a user in there somewhere. You also need to check things like "How easy is it to use?" and "Does this feature make sense?". We also suggested features that the program did not have, but from our experiance using it, thought that it should have.
I always prefer to start the year off with a bang - or, to be more precise, a series of loud hums, a crackle or two, and
Outside of unit testing and limited functional testing, developers should be doing QA on their own code. That's a bit like a farmer certifying his own produce as organic, or a college student awarding themselves a diploma. It misses the point. QA function, automated, regression et al testing is the responsibility of a QA department. If your employer is forcing you to perform QA's functions, then they obviously don't "get it".
I worked 6 years as a Quality Assurance Specialist. You cannot avoid manual testing of a product. Standard practice is to manually test any new software and automate as you go along, to avoid having to go over the same territory each time there is a new build. You also automate specific tests for bugs found after they are fixed, to make sure they don't get broken again. My shop used Rational Robot from IBM. There are a number of others, Silk is one I have heard of, but never used. Developers often have an attitude that Q.A. is only a necessary evil. I think part of it is because it means admitting that they can't write perfect code. The only people I have seen treated worse are the help desk crowd. (another job I have done in the past). The workload was terrible and when layoff time came, who do you think got the axe first? As for developers doing their own testing? That would help some but not all that much. You need people with a different perspective.
I came across this when I recently read the book by Erich Gamma and Kent Beck, Contributing to Eclipse. They do TDD in this book all the time, and it sounds like it's actually fun.
Not that I have done it myself yet! It sounds like a case where you have to go through some initial inconvencience just to get into the habit, but I imagine that once you've done that, development and testing can be much more fun altogether.
At my company, we have a small QA group that tests several enterprise client-server applications, including consumer-level applications on multiple platforms. To exhaustively test all of the permutations and platforms is literally impossible, so we turn to automation for many of the trivial tasks. We've developed several of our own automation harnesses for UI testing and for API and data verif. testing. The technologies that we've used :
- Seque's silktest
- WinRunner
- WebLoad
- Tcl/Expect
There are *many many* problems with large-scale automation, because once you develop scripts around a particular user interface, you've essentially tied that script to that version of your application. So this becomes a maintenance problem as you go forward.
One very useful paradigm we've employed in automation is to use it to *prep* the system under test. Many times its absolutely impossible to create 50,000 users, or 1,000 data elements without using automation in some form. We automate the creation of users, we automate the API calls that put the user into a particular state, then we use our brains to do the more "exotic" manual testing that stems from the more complex system states that we've created. If you are to embark on automating your software, this is a great place to start.
Hope this helps.
Parent has it right. Most automation efforts fail because the test writers can't keep up with the code changes, and not many organizations can pay QA people what you need to pay them if you expect them to be programmers (which is what they need to be to use a decent automation tool). Plus, one refactoring of the code base, or redesign of the UI without any refactoring of the underlying code, and the testers have to throw out weeks of work. Very frustrating.
Even in the best case, automation scripts go out of date very quickly. And, running old scripts over and over again seldom finds any bugs. Usually nobody is touching the old functions anyway, so regression testing is largely pointless (take a lude, of course there are exceptions).
I think the most promising idea on improving reliability I've seen in recent years is the XP approach. At least there are usually four eyes on the code, and at least some effort is being put into writing unit test routines up front.
I think the least promising approach to reliability is taken by OO types who build so many accessors that you can't understand what the hell is really going on. It's "correctness through obfuscation." Reminds me of the idiots who rename all the registers when programming in assembly.
However, if you go into beta testing too early, then major basic features will be broken from time to time, and you'll only irritate your testers and make them stop helping you. This is where automated tests shine, because they help you ensure that major changes to the code have not broken anything.
Put another way, automated test can verify compliance to a specification or design. User testing can verify compliance to actual needs. Neither can replace the other.
When Fred Brooks published his book, The Mythical Man-Month, one of the things he noted was that testing should acount for *more then half* of the budget of a software project. Actual design and coding should be the minority. This is because software is complex, inter-related, easy to do wrong, and not obvious when it is done wrong.
Of course, nobody wants to do that, because it's expensive and/or boring. Thus we have the state of software today. Just like we had the state of software back in 1956 when he wrote the book.
It never ceases to amaze me that we're still making the same exact mistakes, 50 years later. If you work in software engineering, and you haven't read The Mythical Man-Month, you *need* to. Period. Go do it right now, before you write another line of code.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.