Slashdot Mirror
Automated Software QA/Testing?
nailbite writes "Designing and developing software has been my calling ever since I first used a computer. The countless hours/days/months spent on imagining to actualizing is, to me, enjoyable and almost a form of art or meditation. However, one of the aspects of development that sometimes "kills" the fun is testing or QA.
I don't mind standalone testing of components since usually you create a separate program for this purpose, which is also fun. What is really annoying is testing an enterprise-size system from its UIs down to its data tier. Manually performing a complete test on a project of this size sucks the fun out of development. That's assuming all your developers consider development as fun (most apparently don't).
My question is how do you or your company perform testing on large-scale projects? Do you extensively use automated testing tools, and if so, can you recommend any? Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)"
And I hate it. Looking for tips here myself.
I keep getting errors following links on slashdot today.
Everyone that disagrees with me is a paid shill
We do it the old fashioned way - we don't.
Seriously, why bother?
Laugh, it's good for you.
If my answers frighten you, stop asking scary questions.
Although I haven't personally used many testing tools:
I created an Enterprise Application composed of Client/Server Apps -- the best test was a small deployment of the Application to users who are apt to help you conduct the test, over a few weeks, I found bugs I never caught with my own manual tests.
Applications that test your code, etc are great from what I have heard, but will not catch Human Interface related issues, i.e. GUI Mess-Ups, Invisible Buttons, etc.
how about we go back to basics and read the proper books on computer science? no need for your shmancy-fancy-'voice debugged'-automagically-'quality assured' offerings, thanks.
i'll stick with The Practice of Programming. at the very least i trust the people who wrote it to have a better judgement.
The first thing you need to learn is that you shouldn't be doing large scale testing on your own systems. That is just setting yourself up for failure, since the only real testing is independent testing. Preferably you should have full-time testers who not only design what needs to be tested, but how the testing will be done and who will do the actual testing. Where I work, we have 2 testers who write up the test plans, and then recruit actual users to do the testing (because they can then not only get some exporsure to the system, they can suggest any enhancements for the next version). Testing your own work is a huge no-no, as you are much more likely to let small things slide than an independent tester is.
"Information wants to be expensive" - Stewart Brand, the same guy who said "Information wants to be free"
www.empirix.com --- blatent plug
Nothing can compare to having a dedicated test staff. At the last software place I worked (part-time, in school, while getting my engineering degree), we had 3-6 college students working on testing most of the time (we would also be given some small projects to work on).
Testing goes far beyond what any automated system can test, if you have a user in there somewhere. You also need to check things like "How easy is it to use?" and "Does this feature make sense?". We also suggested features that the program did not have, but from our experiance using it, thought that it should have.
I always prefer to start the year off with a bang - or, to be more precise, a series of loud hums, a crackle or two, and
I agree about programming. I prefer the design phase. I like to design a system to the point that programming it is a cinch. What really sucks about software development is not testing it is meetings. Meetings suck the fun out of programming. Stupid senseless timewasting meetings. Scott Adams hits the nail on the head about meetings every time.
-- Each tock of the Planck clock is a new world and here we are still life. --
Outside of unit testing and limited functional testing, developers should be doing QA on their own code. That's a bit like a farmer certifying his own produce as organic, or a college student awarding themselves a diploma. It misses the point. QA function, automated, regression et al testing is the responsibility of a QA department. If your employer is forcing you to perform QA's functions, then they obviously don't "get it".
We use certification analysts. They handle the testing. Unfortunately, the functional analysts that are supposed to write the tests rarely know the software well enough to do that, so some of my time is spent helping to write the black box tests. Writing good tests has become especially important since most of the cert analyst jobs are being sent to India and aren't on site anymore.
bhj
Automated in house testing tools (for the ui all the way down), combined with an army of testers walking through every criteria manually. So to answer your question we do it both ways. And that's all testers, developers don't get involved with the tests. Having developers test their own code is like having an author edit his own work.
I've used auto-test thingies, ones that I've written, and packaged ones. Some situations call for them. Most of the time, though, it's just a matter of doing it by hand. Here's what I do.
Create a list of inputs that includes two or three normal cases as well as the least input and the most input (boundaries). Then make a list of states the application can be in when you put these values into it. Then create a graph with inputs as X and states as Y. Print your graph and have it handy as you run through the tests. As each test works, pencil in a check mark in the appropriate place.
Now that you've automated the system to the point where you don't need higher brain functions for it, get an account on http://www.audible.com, buy an audio book, and listen to it while you run through your grid. It still takes a long time, but your brain doesn't have to be around for it.
This is going to sound incredibly elementary to people who already have test methodologies in place, but when you need to be thorough, nothing beats an old fashioned test matrix. And audiobooks are a gift from God.
(I'm not affiliated with Audible, I just like their service. I'm currently listening to _Stranger in a Strange Land_ unabridged. Fun stuff.)
--
(sourceCode == freeSpeech)
It's extremely difficult to develop and maintain on any enterprise size system. One of the big problems management has with automation I've found is that once they've put the money into initally developing the automation, they think it will run completely automatically forever more.
From my personal experience at one of the world's largest software companies, automation maintenance for even a small suite (200-300 tests, 10 dedicated machines) is a full time job. That means one person's entire responsibility is making sure the machines are running, the tests aren't returning passes and fails for reasons other than they are actually running the tests, and making changes to the automation both when either the hardware or software changes. This person must know the automation suite as well as the tests attempting to be performed intimately, and must also be willing to spend his days being a lab jockey. It's usually pretty difficult to find these people.
My point here is that even after spending many dev or test hours developing automation, in no way is it suddenly complete. There is no magic bullet to replace a human tester, the only thing you can do is try and improve his productivity by giving him better tools.
-tsf
http://www.segue.com/index.asp
Jesus used to be my co-pilot, but we crashed in the mountains and I had to eat him.
is an oxymoron.
Difficult if there is no logic and no interactions.
Scripts will be of some help.
Probably best will be long complicated sequences of operations whose net effect must logically be precisely nothing.
If you're lazy like me, integrate first, and every module is responsible for not going beserk in the presence of buggy neighbors. Too much software seems designed to assume that everything else is perfect. The old game of "gossip" shows that even if everything is perfect, your idea of perfect and my idea of perfect may not perfectly coincide.
I dunno about anybody else, but my dad works on AUIML(Abstract User Interface Markup Language Toolkit), a large gui back-end for iSeries and any other java front end, web back-end admin type tools. Written in entirely in java there is an eclipse plugin to output the xml gui code (You can find it on the alpha works IBM web site for a free trial). My dad does all the testing with XDE tester from Rational Tools (owned by IBM, how nice!). XDE can test both the web and the java side of this. The tester is really cool, test cases are written in XML. XDE has some odd problems with the Internet Explorer java plugin and he has horror stories about trying to get it to work (He seems to be the only person around capable of getting it to run :) ). He now runs all his test cases over night on a headless machine (They take a couple hours), checks the error reports in the morning to know what he has to fix.
The fun development part is once you've built the model, you can use any technology to develop the automation to run the resulting test cases.
We too use dedicated test engineers who write the test plans and at least oversee the testing (they do most of it themselves). We are working on several different automated test ideas at the present. Automatic regression testing (our code is in Java and each class and package has JUnit test code) will be accomplished during each build cycle.
As you noted, it is more difficult to test the entire system end-to-end. Our problem is complicated by the fact that we don't have the infrastructure to completely load test the system. Load testing of the web interface is another class of problem that can be solved fairly easily ... and if the tests are selected properly, you can exercise your middle-ware and back-end DB server at the same time.
Here is the standard management response to automating anything:
"We don't have time for that. Just get the QA testing complete so we can start the layoffs."
This basically makes the entire question of automating processes academic. Now, if automating processes can lead to massive job loss, salary savings and bonuses, it might actually be approved.
Long-term value is never EVER approved instead of short-term pocket-stuffing, EVEN IF a business case can be made for it. I've seen near-perfect business cases (complete financials, charts, graphs, blow-dried corporate phone-flipping management prick with a light pointer hosting Wheel of Buzzwords in the conference room) made for automating very expensive work schedules, and they were a) ignored or b) shouted down.
Based on this, it's very possible that even if an automated tool could be built, and worked, it would still be ignored because it was non-standard. Yes, I've seen this happen too. Five people assigned to a project that a Perl script could do in a half hour. The Perl script completes the job, but management refuses to believe the results are accurate, so they keep the five people working on the same project for four days... and produce the exact same results.
Now let's all sing the company song...
Business isn't willing to pay for products, innovation and careers, so we get brands, mortgage commercials and layoffs.
I worked 6 years as a Quality Assurance Specialist. You cannot avoid manual testing of a product. Standard practice is to manually test any new software and automate as you go along, to avoid having to go over the same territory each time there is a new build. You also automate specific tests for bugs found after they are fixed, to make sure they don't get broken again. My shop used Rational Robot from IBM. There are a number of others, Silk is one I have heard of, but never used. Developers often have an attitude that Q.A. is only a necessary evil. I think part of it is because it means admitting that they can't write perfect code. The only people I have seen treated worse are the help desk crowd. (another job I have done in the past). The workload was terrible and when layoff time came, who do you think got the axe first? As for developers doing their own testing? That would help some but not all that much. You need people with a different perspective.
I came across this when I recently read the book by Erich Gamma and Kent Beck, Contributing to Eclipse. They do TDD in this book all the time, and it sounds like it's actually fun.
Not that I have done it myself yet! It sounds like a case where you have to go through some initial inconvencience just to get into the habit, but I imagine that once you've done that, development and testing can be much more fun altogether.
duh, open source
best testing there is
they are called coops ;)
At my company, we have a small QA group that tests several enterprise client-server applications, including consumer-level applications on multiple platforms. To exhaustively test all of the permutations and platforms is literally impossible, so we turn to automation for many of the trivial tasks. We've developed several of our own automation harnesses for UI testing and for API and data verif. testing. The technologies that we've used :
- Seque's silktest
- WinRunner
- WebLoad
- Tcl/Expect
There are *many many* problems with large-scale automation, because once you develop scripts around a particular user interface, you've essentially tied that script to that version of your application. So this becomes a maintenance problem as you go forward.
One very useful paradigm we've employed in automation is to use it to *prep* the system under test. Many times its absolutely impossible to create 50,000 users, or 1,000 data elements without using automation in some form. We automate the creation of users, we automate the API calls that put the user into a particular state, then we use our brains to do the more "exotic" manual testing that stems from the more complex system states that we've created. If you are to embark on automating your software, this is a great place to start.
Hope this helps.
As a bonus you'll end up with a scriptable app that advanced users will love.
We evaluated a bunch of tools for testing the front end systems, and after a year long study of what's in the marketplace, we settled on the Mercury Interactive [note: I do not work for them] tools: QuickTest Professional for regression testing, and LoadRunner for stress testing.
No one product will test both the front and back ends, so you will need to use a mixture of the tools, open source tools, system logs, manual work, and some black magik.
Our applications are global in scope. We test against Windows clients, mainframes, UNIX boxes and mid-range machines.
The automated tools are definitely a blessing, but are not an end-all-be-all. Many of the testing tool companies just do not understand "global" enterprises, and working to integrate all the tools amongst all your developers, testers, operation staff, etc can be difficult.
Getting people on board with the whole idea of automated tools is yet another big challenge. Once you have determined which toolset to use, you have to do a very serious "sell" job on your developers, and most notably, your operations staff, who have "always done their testing using Excel spreadsheets".
Another big problem is no product out there will test the front and the back end. You will have to purchase tools from more than one vendor to do that. A tool for the backend iSeries, for example, is Original Software's TestBench/400. Again, this does not integrate with Mercury's tools, so manual correlation between the two products will be needed.
You can only go so far with these tools; they will not help you to determine the "look and feel" across various browsers - that you will have to do yourself.
QuickTest Professional does have a Terminal Emulator add-in (additional charge), that allows you to automated mouse/keystrokes via Client Access 5250, and other protocols.
The best way to determine your needs, is call up the big companies (CA, IBM, Mercury) and have them do demos for your staff. Determine your requirements. Setup a global team to evaluate the products. Get demo copies and a technical sales rep to help you evaluate in your environment. Compare all the products, looking at the global capability of the product, as well as support 24/7.
No tool is perfect, but it is better than manual testing. Once everybody is on board, and has been "sold" on the idea of the tools, you won't know how you lived without them.
Also, make sure that you have a global tool to help in test management to manage all your requirements and defects and test cases. Make sure it is web-based.
Expect to spend over a year on this sort of project for a global company. Make sure you document everything, and come up with best practice documentation and workflows.
Good luck!
We have a special group of people at our company to test our new releases. They're called "users".
At our company the developers use TextTest for automatic regression testing. TextTest is developed in house by 2 of our developers, but opensource. More info at http://www.carmen.se/research_development/programm ing.htm and http://sourceforge.net/projects/texttest.
1. Write code ....
2. Post to windowsupdate.com
3.
4. Profit
Hundreds of books and resources have been written/made available on the subject. You won't find the "right" answer for "you" here. No one here knows your environment/situation/funding/etc. I suggest following the leads the folks here are pointing you to and go from there.
Word of wisdom; don't assume automated testing is the best choice for everything.
I am a test engineer at a very large company. I work in the integration and test department. My job is to test the componets of the system working together as a whole ensuring that everything has been implemented in a compatible way. Developers are responsible for unit test so what they give us should function in the way it is suppose to(not that it does).
First there are the system engineers who spend their times writing specs, thousands of pages of specs detailing in exact detail every part of the system. The developers then take these specs and implement the systems. The system is very large consisting of many software and hardware componets using systems from VAX/VMS mainframes to windows ce clients and diffrent groups responsible for different portions of the system. the developers implement the parts how they interpet the specs and throw out things here and there that arent implementable. then we get the releases and setup everything in the lab completely mirrioring a production enviroment. We also have a tools team which write tools to give us automation, most of these tools are perl modules which us testers can use to write perl scripts to exercise parts of the system. The developers in coordination with our tools and test teams try to give us hooks or api's into the system in order for us to be able to test at all layers of the system, from ui, to service layers, transport layers etc. Most of the time they give us a network socket that we can send XML to that the application will interpet and call the approriate methods. ( I am trying to convince them to use something like SOAP or xml-rpc, but some parts of the system are embedded and http might be too much overhead)
Us test engineers write test plans based of the specs and go through and make sure every part of the system is to spec and that all parts of the system interpet the spec in the same way preventing incompatabilities.
The goal is that during development we come up with a complete suite of test cases and perl scripts and other tools for automation which can be used in the maintence portion of the life of the system for regression test.
You might ask how i like this job, its pretty cool. I just graduated with a B.S. in computer engineering i am making 55k a year and i get to break stuff all day and play with cool stuff. I hope to eventually move to the tools team and then development but for now this is a great way to get familliar with the company and their processes.
I suggest you beta test it with the end user and see what they have to say. Remember that THEY will be the ones that will have to use it on a day-to-day basis.
It seems to be a fait accompli that most businesses will accept IT bullshit, but the engineering companies (umm, people that build bridges and refineries, not engineering wannabees) are a lot less malleable when some computer specialist comes calling.
Why do you think g0d made interns?
Cig? No, thank you.
To be honest many of the things they do could easily be done by something else, but QA/Testing may not seem to be the most interesting for open source developers.
See my journal, I write things there
Virtually serving coffee
I've been programming since 1965, when I started doing cryptology in assembly language (AUTOCODER) on IBM 1401's. EACH segment, module or object needs to be tested for internal consistency and a table of results should be generated by each programmer before delivering the responsible section. Then, as has been mentioned, a separate testing team should test at each stage of integration before delivering the system for customer/client testing.
I've struggled with a methodology for development of Object-Oriented systems. When I first started to learn to program we planned the system first and developed flowcharts before we actually did any code. If the design was right, coding from the flowchart was very easy, and testing was pretty easy also. I have been struggling, trying to get the same results from UML, and finally found a book called, "Tried and True Object-Oriented Development: Practical approaches with UML" by Aalto, et al, and this is how I choose to develop software today. (I do more formal use-case diagrams, and I'm not above letting Rose or Visio generate the first couple of code iterations.) Their method, developed at NOKIA, is a logical intgration of development and testing.
Mike
"The mind works quicker than you think!"
It's designed to stress test web pages, and analyse the load on web servers, database servers and operating systems.
There is also a new company - Cyrano that has risen from the ashes of the old one, and provides many other testing tools, including regression testing.
What's the silver bullet for such?
I've tried WR, Silk, and just about every automated tool available. The biggest drawbacks were the price and learning curve. In the end, I developed an app (completly in Perl) that tested our product (which consisted of a web portal and windows app on a multi-teired setup).
:)
I started with a web frontend where test could be launched and results could be kept (using mysql). The client machine (which runs the test) would poll the mysql db to see if any tests are available. The tests themselves were also in Perl. I used the following perl modules to accomplish testing:
Win32::GuiTest (for windows testing)
Win32::IE::Mechanize (testing with IE)
I used a host of other modules, but those are the main ones. Since Perl is a pretty easy language to pick up, the entire QA staff began writting tests into the framework almost immediatly.
The only area that I cant test with this Perl based framework is Java apps/servelets (yet).
Moral of the story: Just roll your own.
I do not think a single approach is fit for every aspect of software testing, as you well say one can consider at least user interface and business logic to be two separate independent items (and it is a good programming technique to keep them as separate as possible as well.) anyhow, I certainly beleive programming bussines logic is better tested via automated software, since the rules for it's input and responses are well defined mathematical and logical rules that can be very thoroughly and quickly analized via software catching errors in the infinitessimal that the human eye would just not catch at first glance.
Currently I do not use any pre-made testing software for my development projects and I would do well to find one, what I do is to give the bussiness rules and the api or ways to call my program to a friend programmer to code a quick test program to try different valid and invalid values plus border values (the frontier between valid and invalid) and then check if the result is correct.
I give this to an indipendant programer because if any flawed logic on my part (other than typing bugs) causes erroneous data it is very possible I'll apply the same flawed logic to test the code and thus consider a really invalid response as a valid one, using a different codder to come with the test code should apply some safeguard to this situation.
User interfaces themselves are another totally different aspect.. there is a data correctness aspect which can be tested automatically or by a codder who understands the system in which you'd basically test if the data entered on the UI is passed correctly to the system, and if the data returned by the system is displayed correctly by the UI. But the UI also has very different aspects which are ergonomics, and ergonomics can only be tested by humans, and by the final users of the system per se. As programmers we do not always create software for programmers (in fact less than 5% of my projects are ever intended to be used by programmers), for the rest we create software for administrators, HR, ITs, Engineers, Managers, Cashiers, etc etc etc.. the work flow and tought process of different professions are different as well, also as non experts on the area of application there are details either small or huge we can miss, the order of work might differ from what is needed, and also responses could come in places where they are not usefull. For that it is always better to have a small group of end users, prefferably ones you can trust in (beleive me otherwise it can QUICKLY turn into hell for you and them) to test the system and come back to you with any suggerences and errors they find.. also more people on the system produce a higer chance of finding mistakes (misplaced items, misspellings, etc), than a single person would miss (man hours count a lot for a large system testing.).
Once you separate the business logic and other standalone components - just do unit testing on them.
Your UI/network code should not contain wiring logic spaghetti! Separate into validation etc logic that also can be unit-tested. If a button handler has 50 lines of code... - you have much more problems in the first place, solve them first.
- Arwen, I'm your father, Agent Smith.
- Well, you're just Smith, but my father is Aerosmith!
I would like to know how people in other systems companies divide up testing work between Dev and QA. I would also be interested in learning more about the kind of tools people use to develop and track QA.
You need two things: first, people who are dedicated to testing and aren't concerned merely to uphold their pride in the code they wrote (this is a long way to say that you need a dedicated testing team that doesn't report to the coding team); and second, testable code. The best way to get the second needed item, in my experience, is to have your developers write their automated unit tests BEFORE they write the unit they're developing.
This is generally called "test-first" development. If you follow it, you'll find some nice characteristics:
1. Each unit will be easily testable.
2. Each unit will be coherent, since it's easier to test something that only does one thing.
3. Units will have light coupling, since it's easier to express a test for something that depends only lightly on everything else.
4. User interface layers will be thin, since it's hard to automatically test a UI.
5. Programmers will tend to enjoy writing tests a bit more, since the tests now tell them when they're done with their code, rather than merely telling them that their code is still wrong.
You can go a step further than this, and in addition to writing your tests before you write you code, you can even write your tests as you write your design. If you do this, your design will mutate to meet the needs of testing, which means all of the above advantages will apply to your large-scale design as well as your small-scale units. But in order to do this you have to be willing and able to code while you're designing, and many developers seem unwilling to combine the two activities in spite of the advantages.
-Billy
We use SilkTest for automated testing as well as monitoring. Our QA guys love it and it free's them up from doing regular system testing to focus on devising new and devise tests to confound the engineering teams!
Downside? Its Windows stuff AND its hellaciously expensive..
Well, I have to say that everything here seems to make sense. Personally, I use a not-quite-purist version of extreme programming, in which I write unit tests like a madman. I also take the approach that everything I write is an API that will be used by others (even if it won't because I know I'm just stupid enough to forget how I coded something months ago, and introduce some misuse of it). If you make sure all of your pieces are air-tight, then you should be okay. Then integration testing is used to test how well the little blocks play with one another.
As far as end-to-end testing, we have a testing staff that takes care of that. We perform load testing that is derived from actual user usage (as captured in the access log files), but that's about it. If we wanted to be really thorough, we could run light versions of load tests in the integration environment to simulate user behavior and see if that breaks things, but we don't currently.
No matter what test tools, methodologies or approaches you take, testing should always about these things: An independent look at the product (not performed by the developer). A spec with discrete functions defined. A plan that explains what you need to start testing, what you are going to test and how you will report what you find. A documented method to assign severities to bugs. A database to store bug reports in. A set of test cases that focus on testing discrete functions. AND most importantly - a commitment on the part of the project management to review the bug list regularly and assign issues back to developers. Lot's of testing gets done and then results are ignored until someone yells about a particular problem. As for automation, it's great, but it will always be behind the current development curve, too limited in scope and dependent on the person who wrote it. The tests that are automated should just be there to prove that stuff that is already working didn't get broken in the last iteration. I also think it's best to have as many users as possible interact with the product as early as possible. You'll learn so much more from watching them try to use the thing than you can any other way. Plus, the fear that the product is going to actually be used by customers is a great motivator for the team to focus on what matters - shipping a satisfying product.
We've had good luck with Jameleon. We use JUnit for the low level stuff and Jameleon to test the web front end. Of course, it's probably a good idea to use human testers before you go to production, but automated testing can cut down on the bugs that make it to the human testers.
You need to buy a copy of the Pragmatic Programmer's starter kit
The third book in the series is about project automation, where they teach you how to repeat in a controlled manner all the stuff you learned in the first two books. The basics:
1) Write unit tests before writing your methods
2) Your unit tests should follow the inheritance tree of the classes under test to avoid duplicate test code.
3) Write a vertical "slice" of your application first (all the layers, none of the functionality). This will prove out communications and give the QA people something to work with while you flesh out the app.
4) Build & unit test nightly. Any build or unit-test errors need to be fixed the next day, and no later.
5) Release to QA as often as things get semi-stable, and when they have time to test.
6) Try not to ship with any known bugs. How do you know if you've got bugs? Your unit tests, integration tests, QA tools, and end-users tell you via a bug-tracking tool like BugZilla or FogBugz
Do we do all this at my current employer? No. But we're working towards it.
Chip H.
the users did the QA testing, just like Microsoft has the users do. We had QA people, but I am not sure what exactly they were doing becuase many flaws and mistakes got past them. Just not from my programs, but ones my coworkers wrote. I did my own QA testing, and took longer to develop code, hence I was let go.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Proof: For any significantly sized system, take a look at all the independen axes it has. For instance: The set of actions the user can take, the types of nouns the user can manipulate, the types of permissions the user can have, the number of environments the user may be in, etc. Even for a really simple program, that is typically at least 5 actions, 20 nouns, (lets estimate a minimal) 3 permission sets (no perm for the data, read only, read & write), and well in excess of 5 different environments (you need only count relevant differences, but this includes missing library A, missing library B, etc.). Even for this simple, simple program, that's 5*20*3*5, which is 1,500 scenarios, and yes, you can never be sure that precisely one of those will fail in a bad way.
Even at one minute a test, that's 25 hours, which is most of a person-week.
Thus, if you tested a enterprise class system for three days, you did little more than scratch the surface. Now, the "light at the end of the tunnel" is that most systems are not used equally across all of their theoretical capabilities, so you may well have gotten 90%+ of the use, but for the system itself, a vanishing fraction of the use cases. Nevertheless, as the system grows, it rapidly becomes harder to test even 90%.
(The most common error here is probably missing an environment change, since almost by definition you tested with only one environment.)
Bear in mind that such testing is still useful, as a final sanity check, but it is not sufficient. (I've seen a couple of comments that say part of the value of such testing is getting usability feedback; that really ought to be a seperate case, both because the tests you ought to design for usability are seperate, and because once someone has fuctionally tested the system they have become spoiled with pre-conceived notions, but that is better than nothing.)
How do you attack this problem? (Bear in mind almost nobody is doing this right today.)
Why can't you test GUI's? In my experience, it boils down to two major failings shared by nearly all toolkits:
The GUIs have chosen an architecture that is not conducive to testing; they require their own loop to be running, they don't allow you to drive them programmatically, they are designed for use, not testing. When you find a GUI that has an architecture at least partially conducive to testing, suddenly, lo, you can do some automated tests.
And in my case, I am talking serious testing that concerns things central to the use of my program. I counted 112 distinct programmatic paths that can be taken when the user presses the "down" key in my outliner. I was able to write a relatively concise test to cover all cases. Yes, code I thought was pretty good turned out to fail two specific cases (
It never ceases to amaze me what strange and unexpected scenarios a tester can conjure, often exposing flaws and weaknesses in the code. A good tester can intercept many bugs before the product goes out the door. Congratulate and reward them.
Summer students also bring a different perspective, and can evince whole new categories of problems with the code.
We have found that automated testing tools require the skills (and salary) of a programmer, so what's the point? Pay the programmer to program, and hire more testers.
Slashdot entertains. Windows pays the mortgage.
In our 'enterprise environment' type company(3000+ users) we only do manual testing.We gather up some real techies and some real non tech people.Its amazing how much better the non techs are at testing and letting you know of various bugs.
They will let you know flaws that you never imagined were possible!All in all nothing beats manual testing.We dont have any dedicated testing staff so we just gather up a few end users.
Lord of the Binges.
The company I work at developed their own framework for the software product. There are about 100 developers on this product and 20 QA. The product has VBA macros built in, so all the testing is done by executing these macros. But there are thousands of different auto tests.
A significant portion of the testing is completely automatic -- every build goes through a smoke and regression test as it comes through the build chain. There's no user interaction at all on those. The smoke test is about 45 minutes -- smoke test failures are very serious, and everyone in QA gets email when they fail. The regression tests take several hours, and failures in those are less serious but still important.
Each QA member also manually starts certain automated tests when the product is nearing release. They all know how to start the tester and choose files or scripts to test. Output from these tests is automatically put on a database and accessible from the intranet.
I just started working there a few months ago -- I have been continually impressed with the quality of the automated testing. The problem is that this testing does not cover significant portions of the product, since the only tests that can be run are ones that are executable through VBA. So UI performance is not easily automated. I did some research but I wasn't able to find a product that did a decent job UI testing -- the product is extremely complex and the stuff I looked at was all inadequate.
FIT - Framework for Integration Test, is a cool way of wrapping your app so that people can test it more easily. It's open source, and very powerful.
http://fit.c2.com
Anything you can test with a command line, do. Text in, text out, use 'diff result.log official.log' to find if you broke anything. Anything that requires a mouse, though, I hear there's products for that but I've never used them.
I have a pet project (jenny) that generates testcases for cross-functional testing, where the number of testcases grows with the log of the number of interacting features rather than exponentially. It often allows you to do a couple dozen testcases instead of thousands, and it lets you cover more interactions too. It's of no use for pull-down menu interfaces, though, where the goal is pretty much to test every node in the tree and there's no interactions between nodes.
When it comes down to human testing, I'd say hire someone to be the "user from hell". Even if your automatic testing finds all the real bugs, this user should be able to find unanticipated use cases, annoying interfaces, performance problems, etc.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
Don't let developers check in code until it has passed the entire regression suite. Otherwise, SQA (the test maintainers) will be in a constant catch-up mode and ultimately the disconnect will grow until the whole test suite is abandoned.
Make tests easy to create. This is harder than it sounds. Tools that simply record user inputs and take screenshots are way too brittle. You need hooks at the functional level so that you can drive and monitor the application.
Prefer standardized languages and open-source tools to homegrown stuff. Much easier to maintain, especially across a large organization.
Test 1 2 3 4
-
A dedicated QA staff. You should have as many testers as you have developers.
- Tools for the QA staff to create their own automation. They don't like doing manual testing much, either, so they'll have incentive to use the tools.
:-) I'll talk about the tools we use in a bit.
- Training for the QA staff on the tools. Hire QA people capable of at least a little shell programming. And the tools they use should be not much harder than writing shell scripts.
- A good SCM (source code management) system that provides atomic commits, so that when you fix a bug, you can tell your testers exactly what revision number it's fixed in, and they can get exactly that revision verify it in the same system you had when you fixed it.
- A bug tracker. It doesn't have to integrate with the SCM, but if it doesn't, you should make it a hard policy that your commit log messages should say what bug number they are a fix for, and when you resolve a bug, you must say what revision the fix went into. I can't even estimate how much time this policy saves.
- Automated rebuilds of every revision of the software. Spend a lot of time on this, it's key. It lets your testers test things the minute you fix them. That means, if you failed to fix it correctly, you'll find out SOON while the fix is still fresh in your mind, and you'll save even more time by not having to get back into the mindset of that bug. You will need special software to do it, so read on.
So here's how those break down:It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I've been working on a simple idea for a while, a programming language that *provides* the testing capability, implementing the testing story. Its not a horribly difficult idea, just something I've been playing with. My goal is create a test language you can then compile against various languages to see if the implementation in any given language behaves in the same way.. also to make unit tests a little easier to read. You can read about it on the project blog: http://www.bigattichouse.com/thoughtbrew.php?BREWI D=METATEST
meh
The correct URL for Trac is http://www.edgewall.com/products/trac/ sorry.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
While it is true that companies start to realize that automation can help them with the quality of their software, companies that lay off testers or don't perform manual ad-hoc testing on top of what's automated are most likely poorly managed.
The thing is, once you automate something your automation will walk the same code path that was fixed after you logged your bugs after you ran this test case first time. It is very likely that the code paths that you're testing will never be touched again and therefore will never be broken.
There is a possibility that interaction of the API that you've tested with other APIs will cause it to fail, though, and automation will catch that (if it's good) so any time spent on automation is time well spent.
My point here is, there MUST be ad-hoc testing performed on the product, and the only purpose of automation is to free up testers' time to do more ad-hoc stuff, or automate large matrices of testing that would be prohibitively expensive to test manually more than once. Anyone who relies completely on automation is NOT TESTING THEIR PRODUCT WELL!
To add to that, any sufficiently sophisticated automation infrastructure requires serious maintenance. It is not uncommon to have a dozen test case failures every time you run it, and half of the time these failures are test case's fault.
So it's not like you can fire everyone and have an untrained monkey do everything else. Anyone who thinks like this is a fucking moron.
Here are some lessons learned the had way in my 10+ years of automated testing experience: 1. Large scale automation is a very complex project. Think of it not as a testing activity, but as a full scale development project. Start with setting goals, then gather the requirements, and continue as you are writting product. 2. Get at least one person on the team which has participated in large test automation project. There are many common mistakes you can avoid. 3. Make an extensive analisys what you need to test. Some areas are very suitable for automation, others does not make it cost effective. 4. Always think of reducing the maintanance tasks. maintanance is the killer of the automated test. A good framework design will save you thousands of hours of work. One of the most commong approaches is to have separate interface components, so most changes in the UI will leave your tests unchanged. 5. Select the right tool for the job - there are hundreds of them on the market. Check the license price, the power of the scripting language, how it supports our interfaces, can it be integrated with other products. I especially like TestComplete, because is cheap and powerfull. But time for training is also an expences, and many companies select the most popular products there (from Mercury, Rational and Segue). 6. Participate in mailing lists, newsgroups, etc. It gives you the ability to get fast advise from some of the best people in this area. And last, IMHO, the test automation, if done right, is always a winning strategy. But it is not a silver bullet.
To clarify: The dev team only tests on the code level (memory leaks, code fault tolerance, etc.). We have a separate QA team that performs customer-level testing. Some of them are technically inclined, some are not. All of them, however, understand the mechanisms of the system on a user level. Currently, they manually test each operation of each component in the system using test procedure lists and guidelines. I was thinking if there was anyway to, even partially, automate their processes using software. I believe that in any job/work there must be a fun aspect, and lately I've seen that they have lost that spark. Perhaps by introducing a new toy (software) it would both help their productivity and ignite their spirits. Thus the reason for the post. I would like to thank everyone who replied.
Example from older project: developed module/stress/unit test frameworks for my components, including self-built re-usable testing classes and frameworks: allowed me to easily create (in the unit tests) instantiations of the entire component, either with dummy test data, or with component parts replaced by stimuli/receptor/emulator/test code.
Not only that, entire set of test code (larger than the code in the actual module
The proof of the value of this:
1. components were near stress bullet-proof;
2. faults from the field could be easily reproduced and proven by building a new test case/verify into the framework;
3. occasional upgrades of base system or third-party libraries or O/S or compiler caused tests to fail on the daily build because of some now-invalidated assumption -- better to find this out on the daily build than further down in QA where it gates the release, or even worse, on the customer site.
Seems to me that there are really a few levels of automated testing: there's automatically generating and running test cases, and there's having the ability to automatically repeat test cases you've manually created and run in the past.
Just being able to repeat manually created test cases is a big help. It sounds really simple -- create a harness that you can plug test cases into and start writing test cases -- but scheduling and coordinating that sort of thing starts to get really difficult on large projects.
Where I work (a certain large software company) we have about as many full-time testers as we have developers, and it takes work from all of us to keep the test framework up to date and running. Our testers actually write a lot of code as part of their job; their code isn't shipped as part of the product -- it's used to test it. They write test cases and create infrastructure for running test cases. The developers also create test cases that can plug into the test harnesses. It actually took a lot of work to get all of that running smoothly for a project that is large and has a lot of people checking in every day.
Before every checkin we can submit our change to a server farm that runs a smoke build (to verify that things build) and also runs a suite of basic functionality tests (written by developers) to make sure nothing is outright broken by the checkin. It really goes a long way toward ensuring that you have a good build almost every day.
We also run separate automated test passes that take a lot longer but are much more thorough. They include a lot of manually developed cases with some written according to a plan and others arising as specific regression tests. (Having a big regression suite is very important when you're supporting/maintaining previous versions of your software with service packs and hotfixes!)
Automated deployment is also a biggie! Since the software product we work on is intended to be deployed as a distributed system, we also do a lot of testing with multiple nodes deployed. My group spent a lot of man-hours preparing a system than can automatically wipe and install multi-server topologies and then run test cases across them. So basically you get two things: hands-free setup of all installation and config for a large number of machines, and test case coordination between nodes (e.g. something runs on node A, then waits for node B to do something, then checks a result on node C). If you need to verify a fix that only affects scenarios with at least 5 servers involved it can really take a lot of time if you have to set up and test the scenario manually! (Also it is a pity that some software can get complicated enough that new problems will only appear after you have at least a certain number of nodes, but that is another story.)
As far as automatic test case generation goes, we don't have a lot of that. It has its place, but I have to agree with something another poster stated which is that as long as there is a human user involved you need a human tester at the other end. (There's no one-size-fits-all tool that you can run against your software that will magically return "true" iff it is flawless.)
One thing that automatic tools are good at is checking fundamental things at code level, and we do use some tools that do that. One of our tools instruments our binaries by marking every code block so that when you run it you can get "code coverage" output. This helps you see a couple of things: you can get a feel for the complexity of your code (you'll see the number of blocks and arcs for each function or group of functions), and you can see which code gets hit and how many times when you run a particular test case. This allows you to do a few things -- target complex areas of code to try and make simplifications, and target potential dead code areas for removal. This recognizes a couple of fundamental rules of software: complicated code is generally more bug-prone, and dead code is an accident waiting to happen. In the spirit of using automatic tools, this information by itself doesn't mean anything -- it's only good if it's used as feedback to the development process.
D
When Fred Brooks published his book, The Mythical Man-Month, one of the things he noted was that testing should acount for *more then half* of the budget of a software project. Actual design and coding should be the minority. This is because software is complex, inter-related, easy to do wrong, and not obvious when it is done wrong.
Of course, nobody wants to do that, because it's expensive and/or boring. Thus we have the state of software today. Just like we had the state of software back in 1956 when he wrote the book.
It never ceases to amaze me that we're still making the same exact mistakes, 50 years later. If you work in software engineering, and you haven't read The Mythical Man-Month, you *need* to. Period. Go do it right now, before you write another line of code.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
1. Strict adherence to the test first methodology (JUnit). It isn't enough to just write test cases, without nightly (or frequent) builds JUnits loose a lot of their effectiveness - especially in large groups. In addition, having regular code reviews to make sure people aren't just blowing smoke is essential.
2. Early and often QA cycles. Not only will the development team love this, so will your product owners. Here's a good starter for the http://www.extremeprogramming.org/rules/releaseoft en.html release often concept. Doing this will make your QA cycles much more effective and manageable.
3. Don't over automate. Automation can be a very effective tool; however, there must balance.
As for tools (sorry pretty Java centric here), I'd suggest starting with:
- http://htmlunit.sourceforge.net/
- http://jakarta.apache.org/cactus
- http://httpunit.sourceforge.net/
The article is about software testing, and the parent post complained that slashcode isn't tested enough. I second that. Why do we keep getting those 503 errors? Why does the lameness filter forbid posting a short snippet of Perl code to illustrate a point? If slashdot did a halfway decent testing of their code we wouldn't have to live with petty annoyances like those.
Customers. We just ship it and see what happens.
Luckily, my boss has tasked me with looking for better automating solutions. The product being a J2EE web app, I've found a few useful tools from Apache's Jakarta project:
- Cactus (functional verification of J2EE components)
- JMeter (for performance- based evaluation, more from an HTTP front-end, but you can write Java plugins for it).
I've used JMeter a bit, writing a Java extension to make requests to an XML interface to the product. Works well for functional verification, as well as performance- testing the business layer.Since in practice you wont be able to review each ballot cast all you can do is look a totals. So to do this you need to encode you entries--vote the first option 1 time, the second option 2 times and so on. That way you can detect if two buttons have been accidentally swapped.
And of course you are error intolerant to as I said the less than 1% level. So were talking billions of button presses assuming you always press the right one and dont make a mistake.
So how do you pull this off?
is there any way to gaurd against other kinds of errors like buttons offset from their images too?
is there any analogous industry that has this issue?
Some drink at the fountain of knowledge. Others just gargle.
There are two parts of testing:
Validation - testing requirements, paper testing designs, code reviews, usability hueristics etc...
Verification - unit testing, integration testing, regression testing, user testing etc........
The key is to balance each type, although Validation is the best since bugs are caught early and don't cost as much. Verification can be automated as much as possible. What bothers me are people who say that user and manual testing is the way to go? By the time something gets to user testing, the bug is going to cost a lot of money. And manual testing is a waste of skilled time.
Most Video Games release a demo version or have beta testing group. A way this could translate is to have some clients who run a beta version. You dont have to say its beta, but rather tell them that they are the first to try out this new software. Let them use it for free for a while and say that you are letting them use it for feedback on functionality and interface. Then while they tell you what they do and dont like about the software, they'll aslo bug you about the bugs that pop up in the program. then...PROFIT!
For most testing, I pass my code to my team mates. My boss has an especially easy time finding funny and stupid bugs. Keep in mind I'm actually complimenting his ability here. Saves me a bit of embarassment is he catches it before a user ever sees it.
So testing really needs to be done by people who need to use the application in it's environment (and their environment too).
SBC stands for Stupid Bell Company
AT&T stands for All Telephones Tapped
Expect is a Tcl thing, that is also a module in Perl.
Autoexpect is a script generator of sorts for Expect.
Using that combination, you can create a test harness that can also test things that were not designed to be tested, such as GUIs, Browsers, SSH and Password sessions.
However, it is not easy and yes it works on Windoze too...
Oh well, what the hell...
I go even further, and automate the integration test during the development phase. When the code is submitted, I update the general UI input/output routines and UI definintions, and have a working automated test.
The solutions to your problem is trained monkeys. it has been shown in trained monkeys that wide-dynamic-range neurons in the cerebral cortex participate in the encoding process by which monkeys perceive (Kenshalo et al., 1988). There has not been so large a demand for the little guys since Mustacioed italians wandered the steets of New York with them almost a centruy ago.
I find that ring-tailed monkeys (from South America, principally Brazil) are the best for software testing. They are usually shipped in lots of twenty-five. They will be classed by customs as with the capuchin, which means the value-add taxes will be very reasonable.
If you are having trouble importing primates, you can always try checking local campuses for desperate college students. They may not be good at sitting still for verylong, but their wages will be even lower than the monkeys.
In my experience, automated functional testing is good for 1 situation: functional GUI regression testing on systems using a classic "waterfall model" setup where the GUI dosen't change much and there are more than about 3-4 releases per year.
In any other situation, you usually don't get the payback. The software is expensive (I use Mercury Interactive's Winrunner), often running into $100k range. The skill set required is quite specific - you can't just throw 12 arts student graduates at it and expect it to work (which is what happened to me for y2k testing). You need people with development and scripting skills, because it is a development effort. And developers frequently don't want to "lower" themselves to QA (where I don't want to lower myself to dev... heh)
Automated gui testing dosen't work with XP (tried it) or probably any other RAD approach where the gui is constantly changing. Also, an app needs to be a cerain size (or really really frequent release cycles) before it's worth automating. Sometimes it's cheaper and faster to hire test monkeys.
The biggest benefit I see with automated testing is with performance testing. The apps are still $100-150k (MI Loadrunner), but in most cases it's worth it. I can simulate 3000 users on a web server farm after writing a few scripts, making it as simple or as complicated as you like. You usally only write 10 scripts max so one person can finish the scripting in 2-6 weeks, depending on complexity. Compare this with functional testing which ideally needs at least 1 person full time per 100 test scripts, just to maintain/run/write new scripts/report bugs.
During the dotBoom, I coughed up $500 out of my own pocket for Visual Test. I thought it would help me with the hard parts of testing my code: the NT services and the database connectivity. Unfortunately, VT was only good at testing UI pieces, of which there were very little (although one of the Access applications had about 300 forms). The latest version runs about $4k.
My current employer knows we need to do some better testing and version control, but won't allocate time or resources to installing, oh, say, cvs or subversion or visual source safe. We also can't take the time to install and use NUnit (current shop uses microsoft dev tools exclusively). So all the time used learning NUnit has to be done on my own.
Of all of the employers I had in the last 5 years, none actually used the automated test tools they had, instead all depending on human activity (button pushing), with excel spreadsheets to record the results. One place actually fired testers that didn't pass software.
One testing app created for the first Macintosh by Steve Capps was Monkey Lives, which created random desktop events in order to find bugs.
Story here at folklore.org: Monkey Lives
Folklore.org is a really neat site made apparently by Andy Hertzfeld, one of the original Macintosh team. It has lots of anecdotes, of which one of my favorites is I'll Be Your Best Friend, including the choice humor of convincing a waitress to divide toppings in thirds on a pizza.
Check out Canoo WebTest. This is my favorite tool for the kind of high-level functional testing you're talking about (assuming your app is a web app).
I have used a few tools, and like anything else it comes down to your proficiency with the language.
WinRunner http://www.mercury.com/ = Gotta know C++, and you better not be looking at modern tech (.net, java, etc...) plus requires add ins
Quick Test Pro http://www.mercury.com/ = Expensive but deals with modern tech if you but the extention and know VB needs add ins
OpenSource = OK for light stuff
Rational http://www.rational.com/ = Very confusing
SmarteScript http://www.accordsqa.com/ = easy to use, but doesnt require programming experience, new to market, no need for add ins
compuware http://www.compuware.com/ = have to hire them as consultants
just a few views.
I've found that users who will eventually use the system are the best testers. This means deploying a full development environment, and mimicking live data. Try to get someone that has been using software in the particular context for a while. They'll tell you about the features they loved in the systems they used in the 1970's, as well as the features they've always wanted. Try to get a user who is just starting in the industry. They will tell you which parts of your app don't make sense. Those parts are the ones that are going to drive up training costs in the future. Outside of getting a solid product, there is an added advantage to this approach. The test users will recognize their suggestions in the finished product, which increases user buy-in in the end.
For any large scale system that needs testing in its whole, I would refer to a small beta group of users who originally asked for the system or the system's improvements. I find unless I write a testing grid of input and expected output, and adhere to it like a SysAdmin holding a full pot of coffee, I find that I may not be the best one to test my own code. That's why we have users. We develop because it is our art, our hobby, our zen. Any functionality that benefits a user base is purely accidental.
"Beauty is the ultimate defense against complexity." - David Gelernter
The first thing to keep in mind is that test is not QA. I am puzzled so many conflate the two and I suspect the reason is that the QA chief is not visible in the company. If anyone follow up to this posting of mine I would appreciate if you say wethe ror not you have seen/talked to the QA crew much.
OK, so let's start with test: I recommend a single sheet set of testing guidelines and a test process requirement set. test is done at several levels:
Note that test manager should review unit test documents and review possibility of automation. Resist automating GUI testing until system is getting stable, otherwise the overhead in setting up will eat up all benefits in running it.
A test manager needs of course thoroughness and a sense of details but also intuition. The latter part is more important them most people realise. I can from personal experiences tell you it is no fun to be tere with a customer at an acceptance test that has tanked.
Next is Quality Assurance. This is not about testing, it is about process, the process that includes testing, verification and validation.
Do consider CMM (Capability Maturity Model). many Indian companies are top rated and this is used as a lever to get outsourcing contracts. Sure, many cheat and we all know that there are consulatnts that can brief you on the "right answers" on an audit. No matter what, do try to live up to the intentions at least within the company. It does have a useful function, including the unofficial but very real ombudsman function. Thus this job requires tact and diplomacy as there is some authority associated with it. Suggestions:
I'd be interested in hearing from others with CMM/SQA experiences.
In my own Company, yes there are sladotters here actually start up startups, I am moving towards automating as much testing as possible..
:)
But even with moving automated tests towards the front of OOP object designed code I still have to manual acceptance testing...
But since Iam in j2me Gaming I really do not want acceptance tsting automated as then I woudl not get to play any neat games
Don't Tread on OpenSource
I did something like this for the resource manager product. we had performance and workload profile data from thousands of systems ... and we built a parameterized benchmarking/testing infrastructure. we predefined something like 1000 benchmarks that selected statistical samples from range of workloads and configurations. we then developed a workload and performance model implemented in apl.
... this was automated and turned loose. In all approximately 2000 benchmarks were run taking three months of elapsed time to calibrate and validate the operation of the resource manager before it shipped to customers. some benchmarking methodology references:. html#bench. html#fairshar e
after the first 1000 or so benchmarks, all the results were feed into the apl performance and workload model which got to pick the next set of workload and configuration parameters for the next benchmark
http://www.garlic.com/~lynn/subtopic
various performance, scheduling and resource manager references:
http://www.garlic.com/~lynn/subtopic
Our application is just about all Java, written in a J2EE application container, and almost all of our tests are written in some form of Java as well. We use Test Driven Design to drive our API design, and end up with unit tests written with JUnit. We run these unit tests continuously, after every batch of checkins, using Cruise Control which performs an automated build.
To perform functional tests, we use JWebUnit which sits on top of HTTPUnit. The combination emulates a web browser, performing clicks and form sets against the actual server. We automate the running of these tests against an installed system, using Tinderbox, and we do this on every platform that we support (combinations of application servers, databases, and operating systems). The tinderboxes run 4 times a day, using the Cruise Control results.
It takes a lot of hardware (the tinderboxes for each platform) but the infrastructure is easy to set up with free software. The HttpUnit tests are tedious to write, mostly because they take a long time to run. Because of this, we don't consider a feature (user story in XP) complete until it has an automated HttpUnit test.
As for process, we have a small QA team that design test documents for each user story. We use those test documents to design the HttpUnit tests. Since we do this during the development stage, we involve the QA team on a daily basis.
When a user story is considered finished by the development team, the tests are reviewed by QA and the functionality is reviewed by the product manager that designed the feature. This way, we have no big surprises when everything is done, everyone can see the product progress day by day.
I won't say we ship a 100% bug free product, but our quality is through the roof compared to what it used to be, and it's actually much higher than we were even expecting.
What's up with moderators today? This guy shows insight making an important distinction but is buried in the noise.
Right, so it is the colour scheme that stops moderation?
I guess this isn't exactly what you are looking for, but in an enterprise setting, the (high paid) developers should not be doing any testing. That ever so fun task of QA testing should be delegated to the (lower paid) testers.
Its a fundamental rule of business. Why pay developer cash for tester skill? Seperate it. The developers can get more done (faster deployment time, more features, better code, etc) by not 'wasting' their time on testing. Plus, it makes for a happier developer; a happy developer is a productive developer. And don't stop there; break it out into one more level - Software Engineers, Developers, Testers. IMHO the rule of three is the way to go.
Of course, this could backfire... The PHB might just come to the conclusion that we can loose some developers, hire testers, and cut costs. Silly PHB - ticks are for kids. (get it? bugs, ticks - kids,... oh never mind...)
Here are things that are less fun:
1. The database servers going down due to badly written stored procs performing extremely intensive queries against poorly designed schemas
2. Billing your customers redundantly or billing them a completely wrong amount causing small claims lawsuits and potentially class actions.
3. Revealing credit card info or other sensitive data by accident
4. Opening your entire infrastructure to your competitors and hackers
You tell me what's less fun. I for one welcome our new QA/Automated testing conquerers.
There's no way to avoid a lot of manual testing, except at /. where they can implement an awful color scheme with next-to-zero testing, apparently.
Write automation at all levels: UI, middle-tier and data-level. You can also write mock objects to simulate the other layers, e.g. the database layer, so you can test each layer in isolation. This takes some effort to design and implement properly. Just treat it as part of the developement effort from the start. That way, writing test automation becomes part of the fun. Don't wait to design and create tests for your code. That's a sure-fire reciepe for creating buggy software.
Mercury has the premier testing tools, for both funcational and performance testing. For specific products check out Load Runner and Quick Test Professional.
I am not saying that these tools will replace a solid QA staff but they will increase the test coverage dramaticly and permit you to measure and manage quality and performance of your applications. Once you have used them, you can never go back. It makes that kind of difference.
Or do you still do it the old-fashioned way? (manually operating the UI, going through the data to check every transaction, etc.)
Isn't that the point of the system in the first place?
A lot of this issue is resolved when you simply code your tests before you design your code. By having the tests coded you have a functional spec that is rather hard to squirm out of and since it is pre-coded you can run it automatically!
Seastead this.
Rational Robot
I've been in QA for a number of companies and have found one consistent problem... re-invention of the wheel...
I attribute this to two things
1) Commercial tools are over complicated and not very good
2) There's no tool that exists to do X Y Z
3) Our software can't be automatically tested !
4) A lot of time is taken up with reporting !
1).. Totally agree... have you ever seen Test Director ? It's a nightmare to use, takes ages to import testcases, and it's automation interface sucks... plus it costs a FORTUNE.
Consequently people end up storing testcases in excel sheets, word docs or spreadsheets...
2) In some cases thats true, but in most its not.. I've seen companies write there own test environments that address problems that have been addressed (often much better) millions of times before... I'm talking automating starting / stop processes and test scripts, reporting, logging, seeding with test data etc... ALL COMMON problems...
The answer to this is SIMPLE
Take a look at STAF (Software Test Automation Framework) by IBM. It's free and GPL. It gives a common framework for witing tests scripts and applications in java/perl/python/c and a few other languages. It consists of a STAF process running and a set of APIs and a command line interface. Services exist to start / stop processes on remote boxes, transfer files, global variables, semaphores, logging.... in fact EVERYTHING you would want to do in automation.
Take a look at
http://staf.sourceforge.net/
This tool saves many man-months of coding in a test environment and gives a consistent way of doing things. No more tester X has this cool script, tester Y has this, dept Z does this....
If you are a tester you need to look at this !
3) This argument normally doesn't hold water anymore... most things can be automatically tested. The question is IS IT PRACTICAL ? If you are only going to test a product once (think end customer doing acceptance testing on your product) then often it's not.
However if you are a software house, then there is no excuse for not doing automated testing, especially when you've got access to source code. You can also buy lots of expensive analysers and script generators for stimulating systems under test.
4) Reporting... yes management wants to know how you are getting on.... A lot of companies do this by email... Each QA person has to send an email every day detailing what they've tested, problems they've had, progress etc . etc.
This is UNNECESSARY. This is the perfect excuse for automation.
An analysts' time should not be taken up with this crap. They should enter this info into a centralised test management system and management should be able to query and manage the test cycle using this tool. Again Test Director tries to do this, but it's TOO complicated.
Now for the shameless Plug....
I have writen a GPL testcase management system called Testmaster which does most of the above and integrates with STAF, allowing test scripts access to the testcase database via STAF.
It provides web front end which holds all testcases and proceedures for multiple projects and departments, imports testcases from word docs, CSV files or directly into the DB and is the primary interface for everyone involved in the QA process.
So now you have your testscripts running, automatically marking tests as passed/failed etc, automated emails going off to management on progress, a web based system for testers and managers to use and the ability with STAF to stop/start/pause testing at the click of a button on multiple remote systems.
It's easy to use, free, under the GPL, eerything is held in a database and runs on almost any Unix system...
Take a look at
http://testmaster.sourceforge.net/
and see if it will make your job easier.
End shameless plug
Alanp
We do automated testing where we can, to make sure everything's working properly after every build.
At the same time, we also utilize full-time testers when we hit milestones, to make sure that items related to look, feel and IA are caught and dealt with.
So, to sum up:
Automated testing to sanity-check builds
Manual testing to ensure milestone quality
I don't understand how QA could happen in software. QA is a manufacturing spot-check to make sure you're not turning out duds. How does that translate to software? Is click-testing a website in a bunch of different browsers QA? Please explain.
Although not quite specifically what you asked for, I can attest that using valgrind definately helps find many bad errors which invariably would send your coders insane.
/previous/ versions so that when you make a new version you can validate the results against the old. What this does is to make sure that you at least are as reliable as your last version.
I run a small open/commercial hybrid development company here - valgrind has been one of the most impressive bug prevention tools we've ever added to our arsenal.
One very important area of testing/QA is regression testing. Always make sure you have a record of what worked in the
We run an automated testing setup (custom scripts) which checks the new software against about 45,000 testcases; the worst part is that it takes about 8 hours with valgrind to get through this (time to buy a bigger server!).
One last thing, if you're compiling C code, turn on -Werror, although some warnings may seem annoying/benign at the time - it's best to fix them all up lest they turn out to be your intermittent, confusing as hell bug later.
PLD.
The developer testing should never be the last step in testing. Developers shouldn't even build the test scenario's. These should be built seperately from customer input and spec's. Scenarios can be reviewed by developers prior to testing, but they shouldn't be doing the actual testing.
Developers during coding develope a mindset of how a user should interact with an application. The users invariably do it differently and the system breaks. So developer only tests for the way he/she thinks the user should work on the system. Then the user uses it and the system breaks.
You usually end up wasting time and money having developers test a system.
I have had many years of experience in QA departments over my career. My observation is that it is difficult to attract good talent to a QA depratment. Many developers and technically inclinded folks see QA as menial labor. This mentality misses the value add and complexities of a true QA department/function. Ideally, you would hire dedicated and technically experiences individuals that: + Can analyize requirements into test plans (not by following the programming logic, but by following the business logic) + Understand the application architecture and environments so they can design tests to get at those components and risks + Develop automation tools, test harnesses and tests data loaders. You really want the technical expertise in your QA department to think about certifying or trying to break your application from a different perspective. When developers guide test plan development too closely, QA can never really be sure they are getting the best test coverage. Do thses skills sound like a "college entry level job"? I think not. Companies that hire inexperienced QA analysts are missing the real benefits of an objective QA department. Depending on the size of the organization, it is helpful to have a QA department report into a centrallized org structure, like the PMO (Project Management Office), or have policies requiring "hard signoff" on quality from QA. This allows them a level of objectivity and ability to ensure the quality of the product. I challenge developers to think of QA differently than in the past. Look for talented independant technical professionals in your QA department, and you will truely assure quality.
with automated testing. Unfortunately it was really kinda silly. They got a new head of QA who went to one of the biggest party schools in California. He hired all of his old buddies from chico and together they chased out all of the good employees. Anyway, his buddy, the head of auotomation and tools decided we would spend a shitload of money to get a whole farm of machines and an enterprise SILK (segue) license. They thought that the QA team could write down each of their tests in baby steps that the automation team could then automate. They though that they could exhaustively test using automation. Worst of all, they thought that they could use automation to test for new bugs.
A year later, and I don't know how many dollars, the powers that be got rid of him (by promoting him to a harmless positions as an "architect" - whatever the fuck that is) and unfortunately kept his buddy as the head of the automation team, although they did roll automation back under the control of QA (thank god). Anyway, they also decided that automation is best for regression testing, as in testing things that used to work and making sure they still work. It's also great for load testing. It totally sucks for testing new features, but it's pretty good at testing legacy features. It's not a good idea to have "automation developers" who have no fucking idea how the product they are supposed to tetst works, and only know how to write silk scripts and 4test and all that other proprietary seque bullshit. Instead it's a much better idea to train the QA team to write automation tests themselves. THere are things QA people know and learn and live and love that make them good at QA testing, and you can't switch to automation only without losing that "instictual" bug finding ability. Finally, the idea that you can replace your QA team with machines is stupid and costs you money in the long run. Last I heard though, the idea that you can replace your QA team with Indians is still believed in over there.
One more thing, the best way to get automation in your testing is to build it into your product. There are some products wiht "unlimited undo" and a full blown command line interface to the engine. It will literally record what the program is doing, each function call, etc. When a bug happens, it spits out the history just like a stack trace, only QA testers, Beta testers, even high-value customers can take this and send it back in. You as a developer and launch the file and get your operating environment in the exact same situation, tracing the exact same crash. It's like time-shifting and station-shifting for bug traces. Of course, this is well known and implimented by Intuit, Microsoft and even mozilla in their little _ would you like to send information about this error to microsoft" feature - which may or may not be under patent pressure by MS, but was used in some forms long before ms, and should be used inhouse at the very least regardless of patent rights (imho). Anyway, there you have it. Automation is useful, is not a holy magic bullet, and QA is worth paying for.
Also, segue is way overpriced and could use some Open Source competition.
Finally, if anyone has applied automation tools towards their p2p application with success, please share with us what you used that worked. Thanks.
Fun is where you find it. All the good testers I know enjoy their work, they really get a kick out of investigating something new and being the one to find the interesting artifact.
Think about cavers, these people go to great lengths to explore underground caverns just for the sake of going somewhere no one else has been before and finding that interesting stalactite or other underground formation etc. I really enjoy software testing but would never go caving, it seems like a huge amount of effort for very little return but there you go.
Coders think they have seen it all because they are the original creators of a product but in an odd kind of way they haven't. Once any non-trivial program is put together and run in the real world all sorts of interesting and unexpected things happen that only a thorough, independent tester will see for the first time (hopefully before a customer does).
... any sort of central searchable repository of source code? Not complete apps, just raw code? something like a google for code? Or just a way to do it if not in a centralised location?
Just seems like an idea to make developing easier. I was thinking about this reading the difficulties in testing and the various tecnhiques. I keep thinking perhaps "better solutions" might be out there, but hidden in non related apps to what any developer looking for code might be working on, so he wouldn't even bother looking there. A searchable database might be a good tool. Say some aspect of an app is doing a particular thing that would be useful in some other totally unrelated project, so maybe the person trying to develop for it might not even be aware of it, but if he could search for it in some manner it might already exist.
It's like patent searching in a way, it should be easier to search, and it should be *there* to search.
No you have to test the actual ballot each elections. You are not trying to prove that touch screen selections can be done accurately in general.
and this needs to be done in every legilative districit. Not al districits will have the funds to do this properly.
i was the director of testing for a company for a long time. the formula i figured out was take the amount of time it took you to write it, triple it, and thats how long it will take you to adequatly test it. that worked until i stopped a release because i wouldn't sign off on it.
when i finally left, their system of testing was also called, SHIPPING.
and you think this i fun? have no fear, your still young.
*** I suffer from a colorful array of psychological problems
... the most automated testing facility we use is a system to press buttons to a script, which is used mainly for stress testing. All other testing is done "old-style" by doing "operations" on the system then checking that all the data for these is saved correctly, which usually means hand trawling through thousands of lines of XML.
I must admit that occasionally finding a bug makes it all worth it, almost.
What you need to do is:
First, make sure all your developers are "test infected". They should all be good at unit testing, and most importantly, test-first development (not exactly the same as test-*driven* development, more like a subset).
Tests should be written *before* the code. This does two things.. first of all it makes sure the tests are written, second of all it keeps the code *SIMPLE*. Everytime you add an exception handler, for instance, you *must* write a test for it first. Pretty soon you learn to leave the exception handling in higher, more-abstract layers where it belongs. Etc.
When it comes to GUI's you will soon learn to strip as much logic as possible out of the GUI and put it in testable controller classes. Basically your GUI does nothing but send and receive events from controllers, and the controllers are tested using the same hooks that the GUI uses.
This takes discipline but I've found in my own experience it is one of the few "achievable" things that require discipline. Once you get into the rhythm you can't give it up (that's what "test infected" means). Your guys most *love* testing. They must get a kick out of 100% test coverage.
Everytime a bug report comes in, write the test that exposes the bug, *then* fix the bug, and watch the test pass. Fun!
Next, you must AUTOMATE everything you can. I wholeheartedly recommend reading Pragmatic Project Automation by Mike Clark for some ideas.
Finally, you need professional testers if your app is big and complicated. They will sit down with your user docs and start testing your builds. This gets expensive of course.
Oh yeah, be sure to TRIPLE all your time estimates. You have to write good docs, and create good tests, and each of those takes almost as much time as writing the code.
Yes testing (especially the non-unit testing, functional, integration, whatever) is super-expensive. But once you start noticing that your bugs have slowed to a trickle, you will be impressed.
It also helps if the end-user of the software is enthusiastic and actually reports problems and guides the development.
If you can't get professional testers, focus on building as many AUTOMATED tests as you can!
http://cthtech.com/ has a product called Get IT Right that can be used to create scripts to do regression, QA, automation. The plus side of this company is that they actually listen to customers when they have good ideas http://cthtech.com/index.php?option=content&task=v iew&id=2&Itemid=25.
...regression testing.
Loading...
For a Web app, a link checker can act as an automatic 'smoke test'. While link checkers do not substitute for unit tests and coverage matrices, they offer a nice trade-off as a low-cost, automatically-expanding tool that can uncover problems quickly.
..."
Any link checker will do for well-linked 'public' pages which do not require a login. On linux, see checklinks for example.
With some additional code, a link checker can:
* log into a login-protected web app and carry the resulting cookie between queries
* spider from each of a list of start pages (ensuring each of these is hit, at minimum)
* parse resulting HTML for required common components (e.g. header/footer), as well as for human-readable error messages like the text "Cannot display
* limit redundant visits to same 'state' with different parameters (requires a URL scheme with identifiable finite state)
For an example of a FOSS link checker with these features, see morebot.
Programming done "properly" is both design and coding. Ideally, (and crucial on large projects) there is design of all functions including "in and out". Anyone whose coded in Assembly knows this is a must.
If properly speced, bounds checking and others tests are very simple to write, and either Q&A can do it, or a day can be set aside where you listen to your favorite music, book on tape, or movie, and test each module. Testing is simple, and many suites make it even easier.
Ultimately, the problem with testing is poor design. Poor design means functions and modules are haphazzardly designed, and being unspeced, it is hard to know the "in and out" of them. Thus, it cannot be tested, for it is now known what to test.
Even if it is not kept to, for whatever the reason, it is worthwhile to note that it *should* be 80% time design , 20% coding and debugging. Design, design, design. Find someone who like it, and it'll answer nearly all your problems. There no reason to feel in the dark, and complain that turning on the light removes all the excitement.
Have you read my journal today?
Being a tester for to many years to count. I say all testers should be required to test for 2 years before thay are allowed to get the CS degree.
...has shown me that most of the commercial tools available are of extremely limited use. If you're working in any cutting edge environment or pushing existing technology to its limits the tools will not be able tohelp you.
Home grown test suites built to suit the job at hand have always turned out to be the bets testing solution. Automation is the only way to keep up with the test effort required for a large project. Especially when QA is seen as an afterthough or a barrier to release. Automation will kepp the code from regressing and serves as a great check for problems duing large merges of patches.
Load test tools are invaluable for verifying the stability of a system. The quality of these tools vary greatly, but things like Apache Flood and httpload have done wonders for stressing web services.
FWIW YMMV
As seen on Wired: Get a free desktop PC
The example GUI application is a simple game, but the methodology could be used for any GUI application.
From the article:
The author goes on to discover, after searching with Google, that he had developed "a pattern known as Model-View-Presenter (MVP), a variant of Model-View-Controller (MVC)."
Martin Fowler has a good description of how the Model View Presenter works.
The example source code is available on the site. It utilizes an automated Java test suite called JUnit
Software Development Magazine is a magazine targeted at software developers, and there is no charge to subscribe to it.
The best defense against a client-side blowup is to have as many layers as possible helping out, because they all have their various strengths and vulnerabilities.
In development:
QA:
Support and IT
Binary geeks can count to 1,023 on their fingers
At least not if you wrote the thing.
It's your baby, you don't wanna hurt it. You also know what it likes and what it doesn't like. You're not fair, you're too nice when testing.
Get someone who wasn't directly involved in developing it. He'll find gazillion of bugs that you won't.
Free as in mason.
Me: "Just like we had the state of software back in 1956 when he wrote the book."
AC: "It was actually 1975. But your point still holds true."
That was a deliberate over-simplification. I should have known somebody here would call me on it. Full version:
The original edition of TMMM was copyright in 1975, yes. However, Brooks based his writings on his professional experience. He was working on projects relating to IBM's mainframe computer products in 1956. He became manager of the OS/360 project in 1964. He actually wrote most of the essays in TMMM from 1965 to 1974. I don't have exact dates for original publication, but I know some or all of the essays (including the "The Mythical Man-Month") were published separately, albeit in much less widely distributed media, prior to the book.
Then, of course, in the "20th Anniversary" Edition, we have a renewed copyright of 1995, an essay "No Silver Bullet" first published in 1986, and a retrospective chapter written in 1994.
I, of course, used the 1956 date because it sounds the most impressive. However, I honestly believe it is a valid "starting date" for the field of personal experience on which Brooks bases his essays.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
These things have proved true to me:
(o) Just like coding testing is engineering: you have limited resources, and have to allocate them to get the max. benefit.
(o) Some small enterprises cannot afford full-time software testers. Some have huge existing code bases, and small or non-existent QA teams. And that just has to do.
(o) Black-box testing for regressions is often where the max. bang for your buck is. Consider: your unit tests may cover 99% of code, and 100% of the units tests may pass. But when one high-level user interface bug creeps in close to release, and your units tests do not detect it, and it stops thousands of users doing the most common operation on your system, and that forces a recall of your software CDs, of what use were your unit tests? Yes, user-interface testing can be a pain, but it is far better to test the 20% of operations that comprise 99% of high-level actions of real users, then to try and code inane tests for every object.
(o) Developers can double as testers.
(o) Yes. Developers can write decent unit tests. Developers can also write good integration tests, black box tests and regression tests. This just needs developers committed to quality software, and a management that sees the benefits of this approach, and pushes for it. For eg: it considers test reviews part of the code review process. Yes, uncommitted developers can do a shoddy job with testing. But committed developers can do a great job. They can also use their inside knowledge of the code to write "grey-box" tests that are more likely to tease out bugs in their own code. For eg: feeding a string "bausch & lomb" to a component that they _know_ converts it to XML for internal use, then making sure that the non-XML output is NOT "bausch & lomb".
(o) Consider critically, the current XP-related fashion of unit-testing everything. It may be useful in some areas, but not in others.
(o) If you have to test web-based systems, consider Apache/Jakarta Jmeter: http://jakarta.apache.org/jmeter/ - a great web testing tool. It is unfortunately focussed on load testing, and hence lacking in some functional testing areas. I wrote a perl wrapper (not open-source) that automates black box integration and load testing with multiple test applications , JMeter being one of them. It has proven quite useful, regularly catching bugs in our system under test, and also on a different system that our system interfaces to.
(o) If you have to test GUIs , it is quite difficult. Consider some of the commercial tools.
(o) Load testing / stress testing is often as important as simple correctness or integration testing. Some bugs can only be "teased out" under load.
(o) Testers are human, just like developers. Testing is generally boring, repetitive work. No one likes boring, repetitive work. Automate testing as much of it as possible. People like being creative. The more you automate, the better will be the quality and quantity of newer tests.
I'm a qa engineer for 8 years now and each days , i feel i like it more than the previous... (except some bad ones of course , but remembering that i could work in the johnson and johnson company provides much help) You could compare QA with a detective job, trying all days to find the little bugs hidding in the code. And laughing with the devellopers when asking then how damn this f... soft works !! i love doing QA.. i think it's the same way that developer loves creating soft... Max.
Drop "it." from the URL to get rid of
Thanks for the tip. Whoever designed the new scheme probably has to have his wife pick out his clothes for him, so the colors will match.
In Soviet Russia we do it like this:
10 coders: 2 years - 4 full-time testers, unit tests (moderate coverage), functional testing, regression testing(love it) - c# + mssql project
3 developers: 2 months C Bulider / 9 months ASP(JScript)+mssql/6 months C Builder - 1 part time tester for each of 3 projects, peer reviews(low coverage)
1 developer: 1 part time tester for most of the projects that last for more than 2 months(mostly php + mysql).
Release often(I prefer weekly releases), it helps a lot. Perform usability testing: it rocks and it's free (use you office mom, for example).
OK, I've been thinking about this and doing a couple of tests, and I believe it's theoretically possible to design an automated testing suite concurrently with the objects themselves if the method of designing objects is similar to the method described in another Jaaksi-Aalto book, "Tried and True Object Development: An Industry-proven approach using UML".
The difficulty is certainly no harder than port-knocking, decryption or virus pattern searching, and would be much faster because you could use a table of valid I/O responses on each object interface, along with a table of test data for communicating with each object and each integrated iteration. Once the test app's framework is designed, it should be re-usable from project to project. After the object is designed your responsible programmer would add the valid input data and valid response data to the appropriate tables. After the object is developed, those I/O's would be used to test the Object for logic and arithmetic errors, at the least. Final testers would integrate the object testers into app testers and confirm the logic. It seems to me that the testing suite would be something on the order of a script that replaced the user interface for massive testing.
Sounds like a good project!
Mike
"The mind works quicker than you think!"
While I agree that it is essential to have indepdendent testing of whatever you're writing, I don't think that absolves you of the responsibility to test your work yourself. Too often, I see developers who just assume their logic works without even trying it once...
Only once a developer is fully confident that there are no problems should he or she send it off to the Testing Team. There should be no "Ooops" moments.
There's stuff on testing with Perl at http://qa.perl.org/, including slides from a couple of talks I've done on the subject.
HOWEVER, and I cannot stress this enough, you need the correct tool for the job. Some tools will work better with different tool sets, some not at all. Try each of the front-runners and make sure whatever choice you make integrates well with your application. This is a one-time decission that if made poorly will kill an automation project. You'll spend more time writing workarounds rather than tests to click on that strange widget your app uses.
One very useful paradigm we've employed in automation is to use it to *prep* the system under test.
Absolutely. The first things I automate are usually installation and configuration. My goal is to have tests running overnight, using all of those spare cycles. I usually plug directly into the build process to get the test running straight away.
I usually open up the tools and let developers use them to configure their systems for their own use. That way they appreciate and understand the automation, another important aspect. If you have developers changing things in the GUI without warning or reasonable cause (it happens, moving icons around "because"), then it's good to have something they use get broken by it. They soon learn!!