Slashdot Mirror
Automated Software QA/Testing?
nailbite writes "Designing and developing software has been my calling ever since I first used a computer. The countless hours/days/months spent on imagining to actualizing is, to me, enjoyable and almost a form of art or meditation. However, one of the aspects of development that sometimes "kills" the fun is testing or QA.
I don't mind standalone testing of components since usually you create a separate program for this purpose, which is also fun. What is really annoying is testing an enterprise-size system from its UIs down to its data tier. Manually performing a complete test on a project of this size sucks the fun out of development. That's assuming all your developers consider development as fun (most apparently don't).
My question is how do you or your company perform testing on large-scale projects? Do you extensively use automated testing tools, and if so, can you recommend any? Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)"
Laugh, it's good for you.
If my answers frighten you, stop asking scary questions.
Although I haven't personally used many testing tools:
I created an Enterprise Application composed of Client/Server Apps -- the best test was a small deployment of the Application to users who are apt to help you conduct the test, over a few weeks, I found bugs I never caught with my own manual tests.
Applications that test your code, etc are great from what I have heard, but will not catch Human Interface related issues, i.e. GUI Mess-Ups, Invisible Buttons, etc.
how about we go back to basics and read the proper books on computer science? no need for your shmancy-fancy-'voice debugged'-automagically-'quality assured' offerings, thanks.
i'll stick with The Practice of Programming. at the very least i trust the people who wrote it to have a better judgement.
The first thing you need to learn is that you shouldn't be doing large scale testing on your own systems. That is just setting yourself up for failure, since the only real testing is independent testing. Preferably you should have full-time testers who not only design what needs to be tested, but how the testing will be done and who will do the actual testing. Where I work, we have 2 testers who write up the test plans, and then recruit actual users to do the testing (because they can then not only get some exporsure to the system, they can suggest any enhancements for the next version). Testing your own work is a huge no-no, as you are much more likely to let small things slide than an independent tester is.
"Information wants to be expensive" - Stewart Brand, the same guy who said "Information wants to be free"
Nothing can compare to having a dedicated test staff. At the last software place I worked (part-time, in school, while getting my engineering degree), we had 3-6 college students working on testing most of the time (we would also be given some small projects to work on).
Testing goes far beyond what any automated system can test, if you have a user in there somewhere. You also need to check things like "How easy is it to use?" and "Does this feature make sense?". We also suggested features that the program did not have, but from our experiance using it, thought that it should have.
I always prefer to start the year off with a bang - or, to be more precise, a series of loud hums, a crackle or two, and
I agree about programming. I prefer the design phase. I like to design a system to the point that programming it is a cinch. What really sucks about software development is not testing it is meetings. Meetings suck the fun out of programming. Stupid senseless timewasting meetings. Scott Adams hits the nail on the head about meetings every time.
-- Each tock of the Planck clock is a new world and here we are still life. --
Outside of unit testing and limited functional testing, developers should be doing QA on their own code. That's a bit like a farmer certifying his own produce as organic, or a college student awarding themselves a diploma. It misses the point. QA function, automated, regression et al testing is the responsibility of a QA department. If your employer is forcing you to perform QA's functions, then they obviously don't "get it".
We use certification analysts. They handle the testing. Unfortunately, the functional analysts that are supposed to write the tests rarely know the software well enough to do that, so some of my time is spent helping to write the black box tests. Writing good tests has become especially important since most of the cert analyst jobs are being sent to India and aren't on site anymore.
bhj
I've used auto-test thingies, ones that I've written, and packaged ones. Some situations call for them. Most of the time, though, it's just a matter of doing it by hand. Here's what I do.
Create a list of inputs that includes two or three normal cases as well as the least input and the most input (boundaries). Then make a list of states the application can be in when you put these values into it. Then create a graph with inputs as X and states as Y. Print your graph and have it handy as you run through the tests. As each test works, pencil in a check mark in the appropriate place.
Now that you've automated the system to the point where you don't need higher brain functions for it, get an account on http://www.audible.com, buy an audio book, and listen to it while you run through your grid. It still takes a long time, but your brain doesn't have to be around for it.
This is going to sound incredibly elementary to people who already have test methodologies in place, but when you need to be thorough, nothing beats an old fashioned test matrix. And audiobooks are a gift from God.
(I'm not affiliated with Audible, I just like their service. I'm currently listening to _Stranger in a Strange Land_ unabridged. Fun stuff.)
--
(sourceCode == freeSpeech)
It's extremely difficult to develop and maintain on any enterprise size system. One of the big problems management has with automation I've found is that once they've put the money into initally developing the automation, they think it will run completely automatically forever more.
From my personal experience at one of the world's largest software companies, automation maintenance for even a small suite (200-300 tests, 10 dedicated machines) is a full time job. That means one person's entire responsibility is making sure the machines are running, the tests aren't returning passes and fails for reasons other than they are actually running the tests, and making changes to the automation both when either the hardware or software changes. This person must know the automation suite as well as the tests attempting to be performed intimately, and must also be willing to spend his days being a lab jockey. It's usually pretty difficult to find these people.
My point here is that even after spending many dev or test hours developing automation, in no way is it suddenly complete. There is no magic bullet to replace a human tester, the only thing you can do is try and improve his productivity by giving him better tools.
-tsf
I worked 6 years as a Quality Assurance Specialist. You cannot avoid manual testing of a product. Standard practice is to manually test any new software and automate as you go along, to avoid having to go over the same territory each time there is a new build. You also automate specific tests for bugs found after they are fixed, to make sure they don't get broken again. My shop used Rational Robot from IBM. There are a number of others, Silk is one I have heard of, but never used. Developers often have an attitude that Q.A. is only a necessary evil. I think part of it is because it means admitting that they can't write perfect code. The only people I have seen treated worse are the help desk crowd. (another job I have done in the past). The workload was terrible and when layoff time came, who do you think got the axe first? As for developers doing their own testing? That would help some but not all that much. You need people with a different perspective.
I came across this when I recently read the book by Erich Gamma and Kent Beck, Contributing to Eclipse. They do TDD in this book all the time, and it sounds like it's actually fun.
Not that I have done it myself yet! It sounds like a case where you have to go through some initial inconvencience just to get into the habit, but I imagine that once you've done that, development and testing can be much more fun altogether.
At my company, we have a small QA group that tests several enterprise client-server applications, including consumer-level applications on multiple platforms. To exhaustively test all of the permutations and platforms is literally impossible, so we turn to automation for many of the trivial tasks. We've developed several of our own automation harnesses for UI testing and for API and data verif. testing. The technologies that we've used :
- Seque's silktest
- WinRunner
- WebLoad
- Tcl/Expect
There are *many many* problems with large-scale automation, because once you develop scripts around a particular user interface, you've essentially tied that script to that version of your application. So this becomes a maintenance problem as you go forward.
One very useful paradigm we've employed in automation is to use it to *prep* the system under test. Many times its absolutely impossible to create 50,000 users, or 1,000 data elements without using automation in some form. We automate the creation of users, we automate the API calls that put the user into a particular state, then we use our brains to do the more "exotic" manual testing that stems from the more complex system states that we've created. If you are to embark on automating your software, this is a great place to start.
Hope this helps.
We evaluated a bunch of tools for testing the front end systems, and after a year long study of what's in the marketplace, we settled on the Mercury Interactive [note: I do not work for them] tools: QuickTest Professional for regression testing, and LoadRunner for stress testing.
No one product will test both the front and back ends, so you will need to use a mixture of the tools, open source tools, system logs, manual work, and some black magik.
Our applications are global in scope. We test against Windows clients, mainframes, UNIX boxes and mid-range machines.
The automated tools are definitely a blessing, but are not an end-all-be-all. Many of the testing tool companies just do not understand "global" enterprises, and working to integrate all the tools amongst all your developers, testers, operation staff, etc can be difficult.
Getting people on board with the whole idea of automated tools is yet another big challenge. Once you have determined which toolset to use, you have to do a very serious "sell" job on your developers, and most notably, your operations staff, who have "always done their testing using Excel spreadsheets".
Another big problem is no product out there will test the front and the back end. You will have to purchase tools from more than one vendor to do that. A tool for the backend iSeries, for example, is Original Software's TestBench/400. Again, this does not integrate with Mercury's tools, so manual correlation between the two products will be needed.
You can only go so far with these tools; they will not help you to determine the "look and feel" across various browsers - that you will have to do yourself.
QuickTest Professional does have a Terminal Emulator add-in (additional charge), that allows you to automated mouse/keystrokes via Client Access 5250, and other protocols.
The best way to determine your needs, is call up the big companies (CA, IBM, Mercury) and have them do demos for your staff. Determine your requirements. Setup a global team to evaluate the products. Get demo copies and a technical sales rep to help you evaluate in your environment. Compare all the products, looking at the global capability of the product, as well as support 24/7.
No tool is perfect, but it is better than manual testing. Once everybody is on board, and has been "sold" on the idea of the tools, you won't know how you lived without them.
Also, make sure that you have a global tool to help in test management to manage all your requirements and defects and test cases. Make sure it is web-based.
Expect to spend over a year on this sort of project for a global company. Make sure you document everything, and come up with best practice documentation and workflows.
Good luck!
Why do you think g0d made interns?
Cig? No, thank you.
I would like to know how people in other systems companies divide up testing work between Dev and QA. I would also be interested in learning more about the kind of tools people use to develop and track QA.
You need two things: first, people who are dedicated to testing and aren't concerned merely to uphold their pride in the code they wrote (this is a long way to say that you need a dedicated testing team that doesn't report to the coding team); and second, testable code. The best way to get the second needed item, in my experience, is to have your developers write their automated unit tests BEFORE they write the unit they're developing.
This is generally called "test-first" development. If you follow it, you'll find some nice characteristics:
1. Each unit will be easily testable.
2. Each unit will be coherent, since it's easier to test something that only does one thing.
3. Units will have light coupling, since it's easier to express a test for something that depends only lightly on everything else.
4. User interface layers will be thin, since it's hard to automatically test a UI.
5. Programmers will tend to enjoy writing tests a bit more, since the tests now tell them when they're done with their code, rather than merely telling them that their code is still wrong.
You can go a step further than this, and in addition to writing your tests before you write you code, you can even write your tests as you write your design. If you do this, your design will mutate to meet the needs of testing, which means all of the above advantages will apply to your large-scale design as well as your small-scale units. But in order to do this you have to be willing and able to code while you're designing, and many developers seem unwilling to combine the two activities in spite of the advantages.
-Billy
Proof: For any significantly sized system, take a look at all the independen axes it has. For instance: The set of actions the user can take, the types of nouns the user can manipulate, the types of permissions the user can have, the number of environments the user may be in, etc. Even for a really simple program, that is typically at least 5 actions, 20 nouns, (lets estimate a minimal) 3 permission sets (no perm for the data, read only, read & write), and well in excess of 5 different environments (you need only count relevant differences, but this includes missing library A, missing library B, etc.). Even for this simple, simple program, that's 5*20*3*5, which is 1,500 scenarios, and yes, you can never be sure that precisely one of those will fail in a bad way.
Even at one minute a test, that's 25 hours, which is most of a person-week.
Thus, if you tested a enterprise class system for three days, you did little more than scratch the surface. Now, the "light at the end of the tunnel" is that most systems are not used equally across all of their theoretical capabilities, so you may well have gotten 90%+ of the use, but for the system itself, a vanishing fraction of the use cases. Nevertheless, as the system grows, it rapidly becomes harder to test even 90%.
(The most common error here is probably missing an environment change, since almost by definition you tested with only one environment.)
Bear in mind that such testing is still useful, as a final sanity check, but it is not sufficient. (I've seen a couple of comments that say part of the value of such testing is getting usability feedback; that really ought to be a seperate case, both because the tests you ought to design for usability are seperate, and because once someone has fuctionally tested the system they have become spoiled with pre-conceived notions, but that is better than nothing.)
How do you attack this problem? (Bear in mind almost nobody is doing this right today.)
Why can't you test GUI's? In my experience, it boils down to two major failings shared by nearly all toolkits:
The GUIs have chosen an architecture that is not conducive to testing; they require their own loop to be running, they don't allow you to drive them programmatically, they are designed for use, not testing. When you find a GUI that has an architecture at least partially conducive to testing, suddenly, lo, you can do some automated tests.
And in my case, I am talking serious testing that concerns things central to the use of my program. I counted 112 distinct programmatic paths that can be taken when the user presses the "down" key in my outliner. I was able to write a relatively concise test to cover all cases. Yes, code I thought was pretty good turned out to fail two specific cases (
To clarify: The dev team only tests on the code level (memory leaks, code fault tolerance, etc.). We have a separate QA team that performs customer-level testing. Some of them are technically inclined, some are not. All of them, however, understand the mechanisms of the system on a user level. Currently, they manually test each operation of each component in the system using test procedure lists and guidelines. I was thinking if there was anyway to, even partially, automate their processes using software. I believe that in any job/work there must be a fun aspect, and lately I've seen that they have lost that spark. Perhaps by introducing a new toy (software) it would both help their productivity and ignite their spirits. Thus the reason for the post. I would like to thank everyone who replied.
Seems to me that there are really a few levels of automated testing: there's automatically generating and running test cases, and there's having the ability to automatically repeat test cases you've manually created and run in the past.
Just being able to repeat manually created test cases is a big help. It sounds really simple -- create a harness that you can plug test cases into and start writing test cases -- but scheduling and coordinating that sort of thing starts to get really difficult on large projects.
Where I work (a certain large software company) we have about as many full-time testers as we have developers, and it takes work from all of us to keep the test framework up to date and running. Our testers actually write a lot of code as part of their job; their code isn't shipped as part of the product -- it's used to test it. They write test cases and create infrastructure for running test cases. The developers also create test cases that can plug into the test harnesses. It actually took a lot of work to get all of that running smoothly for a project that is large and has a lot of people checking in every day.
Before every checkin we can submit our change to a server farm that runs a smoke build (to verify that things build) and also runs a suite of basic functionality tests (written by developers) to make sure nothing is outright broken by the checkin. It really goes a long way toward ensuring that you have a good build almost every day.
We also run separate automated test passes that take a lot longer but are much more thorough. They include a lot of manually developed cases with some written according to a plan and others arising as specific regression tests. (Having a big regression suite is very important when you're supporting/maintaining previous versions of your software with service packs and hotfixes!)
Automated deployment is also a biggie! Since the software product we work on is intended to be deployed as a distributed system, we also do a lot of testing with multiple nodes deployed. My group spent a lot of man-hours preparing a system than can automatically wipe and install multi-server topologies and then run test cases across them. So basically you get two things: hands-free setup of all installation and config for a large number of machines, and test case coordination between nodes (e.g. something runs on node A, then waits for node B to do something, then checks a result on node C). If you need to verify a fix that only affects scenarios with at least 5 servers involved it can really take a lot of time if you have to set up and test the scenario manually! (Also it is a pity that some software can get complicated enough that new problems will only appear after you have at least a certain number of nodes, but that is another story.)
As far as automatic test case generation goes, we don't have a lot of that. It has its place, but I have to agree with something another poster stated which is that as long as there is a human user involved you need a human tester at the other end. (There's no one-size-fits-all tool that you can run against your software that will magically return "true" iff it is flawless.)
One thing that automatic tools are good at is checking fundamental things at code level, and we do use some tools that do that. One of our tools instruments our binaries by marking every code block so that when you run it you can get "code coverage" output. This helps you see a couple of things: you can get a feel for the complexity of your code (you'll see the number of blocks and arcs for each function or group of functions), and you can see which code gets hit and how many times when you run a particular test case. This allows you to do a few things -- target complex areas of code to try and make simplifications, and target potential dead code areas for removal. This recognizes a couple of fundamental rules of software: complicated code is generally more bug-prone, and dead code is an accident waiting to happen. In the spirit of using automatic tools, this information by itself doesn't mean anything -- it's only good if it's used as feedback to the development process.
D
When Fred Brooks published his book, The Mythical Man-Month, one of the things he noted was that testing should acount for *more then half* of the budget of a software project. Actual design and coding should be the minority. This is because software is complex, inter-related, easy to do wrong, and not obvious when it is done wrong.
Of course, nobody wants to do that, because it's expensive and/or boring. Thus we have the state of software today. Just like we had the state of software back in 1956 when he wrote the book.
It never ceases to amaze me that we're still making the same exact mistakes, 50 years later. If you work in software engineering, and you haven't read The Mythical Man-Month, you *need* to. Period. Go do it right now, before you write another line of code.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Expect is a Tcl thing, that is also a module in Perl.
Autoexpect is a script generator of sorts for Expect.
Using that combination, you can create a test harness that can also test things that were not designed to be tested, such as GUIs, Browsers, SSH and Password sessions.
However, it is not easy and yes it works on Windoze too...
Oh well, what the hell...
The solutions to your problem is trained monkeys. it has been shown in trained monkeys that wide-dynamic-range neurons in the cerebral cortex participate in the encoding process by which monkeys perceive (Kenshalo et al., 1988). There has not been so large a demand for the little guys since Mustacioed italians wandered the steets of New York with them almost a centruy ago.
I find that ring-tailed monkeys (from South America, principally Brazil) are the best for software testing. They are usually shipped in lots of twenty-five. They will be classed by customs as with the capuchin, which means the value-add taxes will be very reasonable.
If you are having trouble importing primates, you can always try checking local campuses for desperate college students. They may not be good at sitting still for verylong, but their wages will be even lower than the monkeys.
I've been in QA for a number of companies and have found one consistent problem... re-invention of the wheel...
I attribute this to two things
1) Commercial tools are over complicated and not very good
2) There's no tool that exists to do X Y Z
3) Our software can't be automatically tested !
4) A lot of time is taken up with reporting !
1).. Totally agree... have you ever seen Test Director ? It's a nightmare to use, takes ages to import testcases, and it's automation interface sucks... plus it costs a FORTUNE.
Consequently people end up storing testcases in excel sheets, word docs or spreadsheets...
2) In some cases thats true, but in most its not.. I've seen companies write there own test environments that address problems that have been addressed (often much better) millions of times before... I'm talking automating starting / stop processes and test scripts, reporting, logging, seeding with test data etc... ALL COMMON problems...
The answer to this is SIMPLE
Take a look at STAF (Software Test Automation Framework) by IBM. It's free and GPL. It gives a common framework for witing tests scripts and applications in java/perl/python/c and a few other languages. It consists of a STAF process running and a set of APIs and a command line interface. Services exist to start / stop processes on remote boxes, transfer files, global variables, semaphores, logging.... in fact EVERYTHING you would want to do in automation.
Take a look at
http://staf.sourceforge.net/
This tool saves many man-months of coding in a test environment and gives a consistent way of doing things. No more tester X has this cool script, tester Y has this, dept Z does this....
If you are a tester you need to look at this !
3) This argument normally doesn't hold water anymore... most things can be automatically tested. The question is IS IT PRACTICAL ? If you are only going to test a product once (think end customer doing acceptance testing on your product) then often it's not.
However if you are a software house, then there is no excuse for not doing automated testing, especially when you've got access to source code. You can also buy lots of expensive analysers and script generators for stimulating systems under test.
4) Reporting... yes management wants to know how you are getting on.... A lot of companies do this by email... Each QA person has to send an email every day detailing what they've tested, problems they've had, progress etc . etc.
This is UNNECESSARY. This is the perfect excuse for automation.
An analysts' time should not be taken up with this crap. They should enter this info into a centralised test management system and management should be able to query and manage the test cycle using this tool. Again Test Director tries to do this, but it's TOO complicated.
Now for the shameless Plug....
I have writen a GPL testcase management system called Testmaster which does most of the above and integrates with STAF, allowing test scripts access to the testcase database via STAF.
It provides web front end which holds all testcases and proceedures for multiple projects and departments, imports testcases from word docs, CSV files or directly into the DB and is the primary interface for everyone involved in the QA process.
So now you have your testscripts running, automatically marking tests as passed/failed etc, automated emails going off to management on progress, a web based system for testers and managers to use and the ability with STAF to stop/start/pause testing at the click of a button on multiple remote systems.
It's easy to use, free, under the GPL, eerything is held in a database and runs on almost any Unix system...
Take a look at
http://testmaster.sourceforge.net/
and see if it will make your job easier.
End shameless plug
Alanp
I have had many years of experience in QA departments over my career. My observation is that it is difficult to attract good talent to a QA depratment. Many developers and technically inclinded folks see QA as menial labor. This mentality misses the value add and complexities of a true QA department/function. Ideally, you would hire dedicated and technically experiences individuals that: + Can analyize requirements into test plans (not by following the programming logic, but by following the business logic) + Understand the application architecture and environments so they can design tests to get at those components and risks + Develop automation tools, test harnesses and tests data loaders. You really want the technical expertise in your QA department to think about certifying or trying to break your application from a different perspective. When developers guide test plan development too closely, QA can never really be sure they are getting the best test coverage. Do thses skills sound like a "college entry level job"? I think not. Companies that hire inexperienced QA analysts are missing the real benefits of an objective QA department. Depending on the size of the organization, it is helpful to have a QA department report into a centrallized org structure, like the PMO (Project Management Office), or have policies requiring "hard signoff" on quality from QA. This allows them a level of objectivity and ability to ensure the quality of the product. I challenge developers to think of QA differently than in the past. Look for talented independant technical professionals in your QA department, and you will truely assure quality.
...has shown me that most of the commercial tools available are of extremely limited use. If you're working in any cutting edge environment or pushing existing technology to its limits the tools will not be able tohelp you.
Home grown test suites built to suit the job at hand have always turned out to be the bets testing solution. Automation is the only way to keep up with the test effort required for a large project. Especially when QA is seen as an afterthough or a barrier to release. Automation will kepp the code from regressing and serves as a great check for problems duing large merges of patches.
Load test tools are invaluable for verifying the stability of a system. The quality of these tools vary greatly, but things like Apache Flood and httpload have done wonders for stressing web services.
FWIW YMMV
As seen on Wired: Get a free desktop PC
The example GUI application is a simple game, but the methodology could be used for any GUI application.
From the article:
The author goes on to discover, after searching with Google, that he had developed "a pattern known as Model-View-Presenter (MVP), a variant of Model-View-Controller (MVC)."
Martin Fowler has a good description of how the Model View Presenter works.
The example source code is available on the site. It utilizes an automated Java test suite called JUnit
Software Development Magazine is a magazine targeted at software developers, and there is no charge to subscribe to it.
Me: "Just like we had the state of software back in 1956 when he wrote the book."
AC: "It was actually 1975. But your point still holds true."
That was a deliberate over-simplification. I should have known somebody here would call me on it. Full version:
The original edition of TMMM was copyright in 1975, yes. However, Brooks based his writings on his professional experience. He was working on projects relating to IBM's mainframe computer products in 1956. He became manager of the OS/360 project in 1964. He actually wrote most of the essays in TMMM from 1965 to 1974. I don't have exact dates for original publication, but I know some or all of the essays (including the "The Mythical Man-Month") were published separately, albeit in much less widely distributed media, prior to the book.
Then, of course, in the "20th Anniversary" Edition, we have a renewed copyright of 1995, an essay "No Silver Bullet" first published in 1986, and a retrospective chapter written in 1994.
I, of course, used the 1956 date because it sounds the most impressive. However, I honestly believe it is a valid "starting date" for the field of personal experience on which Brooks bases his essays.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Drop "it." from the URL to get rid of
Thanks for the tip. Whoever designed the new scheme probably has to have his wife pick out his clothes for him, so the colors will match.