Slashdot Mirror
Testing Products for Web Applications?
"I've seen a lot of automated test suites advertised and I've always assumed that they were no substitute for careful testing by a human. However, as the number of web pages that we need to maintain grows, I've begun to wish that we had something that we could kick off at night, that would follow all links on our system and fill in values for the various forms it encountered, then when we arrived in the next morning there'd be some sort of report available detailing its findings. It could flag any pages that returned something obviously incorrect, such as a SQL error, a blank page or just the word 'error'.
Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?"
Mercury Interactive - www.mercuryinteractive.com has some products that will do this. I used them for a short while and they seemed pretty good.
With the right perl modules, or even perhaps in shellscript with things like CURL, you could roll your own http/html regression tester. Won't handle javascript of course, and won't notice browser-dependant problems. You might be able to find a generic javascript/DOM library for unix to do the JS thing from your regression tester though.
11*43+456^2
Web Application Stress Tool (freebie from M$)
http://webtool.rte.microsoft.com/
You should check out Apache Cactus http://jakarta.apache.org/cactus/.
http://jakarta.apache.org/commons/latka/index.html
it's a java XML solution to writing automated suites of functional tests. and it's free.
Well you hit upon one good way, you just forgot to post the link...of course if you did you'd be more worried about your server overloading than your web frontends not working correctly...
"I may be quite wrong." - Socrates
Well if you are working in Java, I've used Cactus before with success. It's based on junit, and allows you to do unit testing on servlets/jsp's in a nicely automated way. As long as you take the time to create good test cases, it can do quite a good job.
This sig has been temporarily disconnected or is no longer in service
I run a couple free web games ... and let me tell you ... if it has a security flaw, these people will find it! Hire a couple people that play my game! I'm sure they'll find any security flaws you may have!
... or end a SQL query in a text box and try to execute another query (or put in a sub query) ... and edit your query strings if you use GET (or build a query string and make sure that your program doesn't take a GET where it is looking for a POST) .... just a couple basics to try ... You might want to write a "validate_input" function for your forms as well ....
...
Seriously, I don't know of any software that does that, but if you find one, I'M INTERESTED!
I don't know if you're looking for advice or not, but try putting in negative numbers or things like #(-3+1000)
Hopefully that helps a little
HallmarkOrnaments.Com
By far the best tools out there, use Google to find them, but they cost quite a bit. All of these solutions require quite a bit of scripting and customization for full testing. For completely automated testing, well, try hiring someone. --ellis
GetTheJob.com : Nothing but Real Jobs.
don't put your applications on the web, somebody will steal them
keep them locked up in the server room instead where they will be secure
then buy a big doberman and name it "george", put it in the server room with a big bowl of raw goat meat
if somebody breaks in you will hear george going ROOF ROOF ROOF BOWWOWWOWROWROWROW and then gurgling and screaming as the thief is eaten
this is all i can do.
Both Cactus and HttpUnit allow you to do unit tests on web components. Both are extensions of JUnit. Cactus allows you to do unit tests of servlets and JSPs, while HttpUnit allows for unit tests of the resulting HTML code. (Cactus also integrates HttpUnit to a certain degree.)
Obviously, these tools are targeted at Java development. I have less experience with HttpUnit than with Cactus, but I imagine it could be used as a general test suite.
>Weve just had 9/11/02, and bush is attacking
>Iraq, and your talking about TESTING PRODUCTS
>FOR WEB APPLICATIONS? MY GOD PEOPLE GET SOME
>PRIORTIES!!!!
If the web is full of buggy applications, the terrorists have already won.
(my talking about testing products? what?)
Take a look on Web Site Test Tools and Site Management Tools page. And of course shameless plug: HTTP-WebTest. If you will check the latest make sure to try it's beta version.
--
Ilya Martynov (http://martynov.org/)
I used to work on such a system, and I did a lot of the testing for it as well. We found quite a few bugs, but those were overwhelmed by the number of interface design flaws we encountered. There were literally hundreds of unnecessary mouse-clicks per user per day in the original interface, simply because the programmers never had to use their own software for days at a time.
So I guess my point is, make sure you don't simply rely on automated testing. A bot won't get sick of clicking unnecessary buttons, and won't develop RSI injuries. Humans will, and you'll get great feedback because of it. At my old company, the programmers were very nice about fixing these flaws once I brought them to their attention, and grateful for our input.
Cheers,
Anaphilius
Quick lesson in automated testing.
The only automated testing tools you can find is for regression tests. Basically, you make "build 1". You use the tool to 'record' the tests you currently run, and have it check for successes and failures. You make "build 2", and run the tests, to ensure everything that once worked, still works. Now you test the new stuff, record these tests with the tool, make "build 3", etc...
There are three major companies with good automated regression tools. Mercury Interactive's WinRunner, Rational's Robot, and Compuware's QA Center. All of them are great tools (and you can get them packaged with load testing tools if you'd like).
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Even if he programmed this, he would be too lazy to test it.
I know what he is talking about though. I like programming when something is a challenge, and I'm not sure how I'm going to accomplish it. But as soon as it gets down to small details and testing, i find it very tedious. Oh well... Those are the breaks!
Several comes to mind--Test Perspective, LoadPro, ActiveTest, etc. You can also buy your own software to do this, or write something in a script language.
s s_works.html) has the ability to randomly fill in forms with lists of data you give it. It will figure out what form it is, select the right list of data, submit the form and go to the next one. It can validate that the form returns the correct data too.
I'm most familiar with LoadPro and Test Perspective..and of course scripting it.
With Test Perspective, you can record the way the web app works, then have them play it back for you with lots of variations with however many number of users you want.
LoadPro (http://www.keynote.com/solutions/html/keyreadine
Scripting it yourself is pretty easy too, but you want to make sure you use one that does http 1.1 (perl LWP doesn't) and you want to model your users accurately.
As for purchasing a tool, there is SILK Performer and Segue, both traditional functionality testing tools
Donald E. Foss
No Not Again! Its whats for dinner.
Just post the link to your website on /., if it doesn't crash from the load then it's probably pretty good. Hey maybe Taco should look into this! He could start offering it as a service :)
The Anti-Blog
Here at VictoriasSecret.com, we just rewrote all of our C++ code as J2EE-compliant java. It was a huge project with a very short timeline. Given the amount of money that we lose during even short outages, we brought in outside consultants who's only job was to test the new site. They used empirix to test our site. I didn't actually use the tool so I can't comment on ease of use or whatnot but I looked over their shoulders from time to time and talked with them about it. It looked pretty slick.
I've never used it, but it seems like it would probably be pretty helpful.
-NiS
If you can't figure this out, please stop programming.
Man, this is basic stuff anybody with 2 years experience should be able to handle.
This is suppose to be part of the design, coding, and implimentation. Maybe I should do an ask slashdot:
dear slashdot readers, I've been programming for a long time, and now I have to write an accounting package, can anybody show me how?"
sheesh.
The Kruger Dunning explains most post on
Actually all the information I've seen is that developers should not do *all* the code testing. You should do unit testing and you should write tests, but you should *not* depend on that - you need external testers also
There are many, and as others have rightly posted finding them on Google is easier than posting to /.
SilkTest from Segue is good at both scripted testing & stress testing.
--#voxlator
I recommend httperf and http_load for banging on lists of URLs really hard. At one place I worked, one of our developers rigged up some shell scripts that would play back log files through httperf and that worked pretty well.
If you want to record browser sessions for testing specific paths through the site, look at http-recorder or roboweb. There's also webchatpp, HTTP::WebTest, and HTTP::MonkeyWrench on CPAN. More info on this can be found on the mod_perl mailing list or on PerlMonks.
I agree with you but I believe the main point ( besides the obvious statement of laziness ) is that one change can affect many places and many things. e.g. A function that handles formatting may work now on one page but break another ( out of perhaps 70 possible pages )
The way we handle this at work ( in the eCommerce dpt ) is that when we use a function in a page we document it in a database. We can cross search what functions are used in a page or what pages use a function. This way when we make changes to a function that has a scope larger than the page we're working on we can test it in all of those scenarios
Check out Mercury Interactive for Astro Quick Test, Astra Load Test, WinRunner and LoadRunner (not the game). Then also check out Segue Software for Silt Test and Silk Performer. I have used all of these tools and they are all great. I tend to lean toward the Mercury ones, as the scripts use common languages like C and VB. Segues use a proprietary language, but they have a lot of nice functions built in to them that you would have to create manually with the Mercury tools. Training and support for both companies is comparable. They are expensive though.
OK, maybe I am a little biased, as I have been in QA for 8 years. :-) But my comments still stand.
That said, we are currently using Rational's products to test our application, which includes a web piece. Hint: Don't use javascript if you plan on using Rational. They have SiteLoad, which I believe is free, but rest assured the rest of their products are NOT. Their licensing scheme is nothing short of trying to balance the budget of a small country. If you are wanting to implement their products in a big project, to handle requirements (Requisite Pro), Bugs (ClearQuest) and test plans (Test Manager), then prepare yourself for headaches. If you just want to get Rational Robot to record/playback user actions for testing, it is pretty solid. Rational purchased all different components of their system, so they aren't the smoothest to integrate. I have spent many hours with their phone support people.
I have also worked with Mercury and SilkTest, but to a lesser degree.
Oh, and if you are constantly changing critical code, you need to worry more about your development practices and not your testing.
My beliefs do not require that you agree with them.
"professional html programmers"? ;)
Maybe they could work for "military intellitgence" too, since we're going for oxymorons...
wget and diff
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
contract-based programming and all relatives.
This in fact enables you dismiss all tests.
Remember: the key to successful programming is not to find all error but not to make in first place.
Owner of a Mensa membership card.
I'd highly recommend picking the book:
Java Tools for eXtreme Programming
This is a great reference for all of the tools being mentioned and shows you how to integrate them into the development cycle if your using Java. You should be able to write the functional tests if your app is not written in Java.
As an aside, if your not developing these apps in Java, you really should look at using Tomcat, XDoclet and Struts for simple DB frontends, and then move to EJBs with JBoss, Jetty or Tomcat, Struts and XDoclet. If your lazy and don't want to write a lot of code, you'll love these tools. Reuse is high in Java, and the code generation tools like XDoclet take away most of the pain of using frameworks like EJB and Struts. Besides JSP taglibs allow me to have good looking pages made pretty by people who care about the differences between browsers for CSS, DHTML and what not.
Good Luck.
Arrogance is Confidence which lacks integrity. -- me
Come on guys, this is no way to help a lady. :) Quit being kings of your particular castles and lend a helping hand.
"I'm a leaf on the wind. Watch how I soar."
-Hoban Washburn
For a limited time only, the newest Robust Web Application Testing Program is available to you free of charge*. For this week only you can have your very own Slashdot Web Application Analyzer!
To use this miracle of modern computing, simply submit a story link to your Web Application and the webmaster's e-mail at the bottom of the page! Not only will you be able to test your server bandwidth, but every know-it-all Slashdot Web Guru(tm) will e-mail you with exactly why your Application is not worth the electrons it's stored on!
For added bonus, have your site flame one of the following groups for extremely extensive testing: Any Goverment, Adobe, Microsoft, Intel, Creative Labs, CowboyNeal.
Call Now!
Operators are standing by!
Are you? Ever tried writing testing programs for bulky packages? Will "DEFINITELY" cost more that probably Mercury Interactive's WinRunner / LoadRunner and their great suite of Web Application Testing tools. O by the way ...just had a great session with 4 Merc. Int. people. .....
Try them
The problem is that the programmers who write the application know too much about the app and how the app is *supposed* to be used to be good testers. I believe that they will subconcsiously test the wrong things. You need "typical users" (less than typical, actually) to really get useful results. If the only real testing that's being done is being done by the programmers, watch out.
Clueless newbies and kids will find the problems first. The problem is that they don't report very well. What I want is testing software that tests like a ten year old, but reports like a senior programmer!
Just thought I'd note the obvious...
:)
By reviewing how you work with the actual code, you can avoid making a lot of the bugs in the first place. When making solutions where more than one module/frontend depend on various backend functions I find that I usually avoid most problems with the API changing if I simply carefully map out whats needed of the API and deciding on how I want to access it once and for all. Once that's decided upon you can change the code as much as you want as long as you leave the actual API alone.
One of the things made easier by OOP for example
I know I might be pointing out the obvious here, but experiece have shown me that thinking about how you design your actual development cycle is a topic which is too often overlooked, with painful results.
Terje Elde
If an application is written correctly, you should only have to test one copy of each object (one input of type text, one select, etc) because your code should be reused over and over. If you have to test each input control, you obviously are doing something wrong, especially if you have hundreds of pages. Use a methodology in your coding of only writing something once. When you reuse it, it should work every time.
of commercial tools that support extensive
script-driven testing of web applications.
SilkTest is the testing tool.
At my previous startup, we bought and used
these tools and developed extensive test
libraries for our product.
There are also companies that will test your
product for usability on many different platforms.
Look at http://www.otivo.com/ for one such.
It's fine for a simple application, sure. But the more complicated the app is, or if it is tailored to a customer's eBusiness requirements (for instance), you will never beat a set of human eyes aggressively finding the bugs. It can take as long to write the script or set up the robot as it does not just do it.
*Commercial endorsement is not intended.
I keep forgetting that Alzheimer's runs in my family.
Can you believe that? Prejudiced? Against Italians? In the year 2002?
WTF is wrong with me?
Some words of advice if you care to follow them.
First off, ignore anything with the words "stress" or "performance" in the titles or descriptions. They are not the tools you want, and are focused primarily on simulating multiple clients rather than simulating users.
Second off, seperate the kinds of testing you want to do. Simple form validation requirements will most likely mean you can get away with a tool that bypasses the browser interface (typically a unit testing tool). More complicated user simulation should be done by a tool that actually drives the browser, such as SilkTest or Rational.
Finally - Hire a dedicated resource just for this purpose. A QA Engineer with experience in automated testing, REAL experience, not just playback and record experience. (My resume is available on demand).
best web host ever
I bet you got to see a lot of hot nude chicks while programming for Victoria Secret!!
You might look up my friends over at F-Test. By focusing only on functionality testing, they're able to do it more efficiently than almost anyone. They can do it more thoroughly and cheaply than most companies can do themselves, even small shops like ours (and probably yours). They've done great work with our stuff, as well as for big corporate clients like Sony. Nothing beats a team of well-trained, experienced testers banging away at keyboards, but there aren't many people around focusing on just that. Look 'em up. They're in Los Angeles.
Check out WWW::Automate if you need to code a web application test in perl.
So, do you charge more per hour for your development time, or for your whoring time?
I have a java application I wrote using open source technology, such as HTTPClient, PerlTools, and the JavaMail API. It can be used for load testing as well. You can download it here: http://www2.netdoor.com/~delonad/WebTestSuite_1.02 .zip
It uses an XML config file. In the config file, you specify notification addresses, URLs, and regular expressions....variables can be assigned when regular expressions match as well. Basically, the program runs through the test cases, and if something doesn't match an email notification is sent with the HTML received as an attachment. This application is most useful for regression testing in my opinion.
have them "use it like it'll get used" for a while. Note all the problems, fix them. From there, you can create use cases that'll let sobody else do the testing.
It's CRITICAL, IMHO, that the people requesting the application get directly involved with how the front ends should work. If they don't, you're just asking for UI rework pain.
ceci n'est pas un sig.
Recently we started using maxq.
It has two modes.. One is a proxy server which you'd set your browser to. It records the post/get arguments you used for the page and records it into a jython file. The backend then uses HTTPUnit to fire off the pages.
It isn't a complete solution.. I had to create a perl filter work with mime-multi-part data, or indeed any form-data that has carriage returns.
But since it's a simple mixture of python and java, it was relatively easy to apply statistics to the processes and search for all sorts of possible error-types.
The problem with simple non-human web crawlers is two fold. First there are pages that require valid form-data. Secondly, a "nightly sanity test" is going to be operating on production data.. You'll need to carefully manage such data.
-Michael
Mercury Interactive makes tools that will do this. We have used LoadRunner and WinRunner to test our SAP environment (including the ITS web environment), and it seems to work well. The trouble is that it is very expensive.
It isn't bad programming. It isn't lazy programmers. It is called REAL LIFE in a REAL PRODUCTION EVIRONMENT. If there is a programmer today who has followed book instructed design methods to the letter then they are not programmers. They are college professors or working for some governmental instituion.
One advantage of web applications is so changes can be implemented QUICKLY and CHEAPLY. If I used an includes to build drop down boxes and I changed the core include then EVERYWHERE I included that code I have to test to make sure it won't screw anything up. In the real world your not able to test everywhere when they need the change NOW.
As well the errors might not be 'true errors' in programming but simply making workflow harder or impossible to do with the new changes.
If you have a set of screens designed to do A. Individuals start using the set of screens to do B through F with and then the set of screens is modified to do A+1 but that stops B from being used through that set of screens while C through F is several hampered. THAT is what happens with not just web applications but all. THAT is still considered an error by the end users.
There is never a true replacement of humans from the testing of an application. Data and Workflow issues takes humans to determine the problem
I'll echo the comments about a good Q/A person. You find a good one (which seems rare), they'll easily pay for themselves in terms of saved programming and development time (as well as a lot less embarassing "we'll fix that immediately Mr./Ms. Client, sorry" phone calls).
As for tools, we've used RadView (http://www.radview.com) for a lot of the testing automation, and its done well. Does OK regression testing, but the big thing is that it can do excellent load testing, which seems to be something that is often forgotten with web apps.
My $.02 worth.
You can use ab (Apache Benchmark) for load testing. For HTML pages use Mozilla and PreFill form option which will fill the last entered values for a html page. This is not completly automated but its useful for retesting after modification.
Or else check Rational Rose tools.
Programming = Testing If you guys love programming then you better love testing, because testing is just as vital if not more so to the finished product than the code itself (white box testing).
Your proirity in testing shouldn't rely on automation necessarily because what you are bound to find is that the application works perfectly, when it's following the script you've programmed. When somebody on my team brings me code/functionality to review, the first thing I try to do is to "do the wrong thing" (eg. letters in a field to be interpreted as numeric). Thorough testing requires "unbridaled" human ingenugity.
Frankly, what you need are probably consistent programming methods (because your front-ends are probably being written by liberal arts majors who taught themselves --insert language here--), through error handling, documentation, a consistent testing mothodology, and much more upfront requirements analysis.
This stuff ain't cheap and you need to factor it into your pricing. I'd say that 10% to 20% of your budget should be QA and testing and you should insist that the budget be used for that. Too often QA time is used for actual development, leaving no QA.
sarchasm: The gulf between the author of sarcastic wit and the person who doesn't get it.
I've used automated testing products such as Silk Performer, and they are very useful but only in a very limited scope. For most practical situations it is more time and cost effective to simply hire a QA team then to purchase all these tools. Even after purchasing a full suite of automated testing software, which can range in the tens of thousands of dollars, a QA team is usually still needed. As far as automated goes, I've only found it useful for three things:
1.) Link checking.
2.) Server stress testing.
3.) Data entry.
I have just written a perl module that automates IE by using active perls Win32::OLE module. It grabs the IE com object and from there gets the DOM for every frame in the download. It pretty fast and you've got everything you need right under the hood. It's on sourceforge under samie. The project has not been "released" yet. But the basic SAM.pm file is there, and I'm constantly updating it. Check it out and drop me a note if you have any questions...
Let me guess...Hooked on Fonix Werked fer yew? "military intellitgence"
I had very good results using the Java based Pureload when we used it in our qa/testing labs.
http://www.minq.se/products/pureload/index.html
Well if it were just load based -- their are hundreds of programs that will automate and simulate till your hearts desire. That being said, I believe the question was more geared around how to test that when I hit the submit button -- does everything work like it should?
:) Everyone wants to claim QA in place, ISO Whatever in place...etc... The reality is, those were the first things to go.
The best thing to do is to ensure your testers are familiare enough with the back end and the transaction processes to be able to run cross checks on the Database -- to ensure everything is working as it should. Common things like missing where clauses on deletes, in statements like 'a,b,c' rather than 'a','b','c'. Just simple things that automated tools could never catch. The bad part is that things like this take time and bodies. Atr least were I am sitting -- not near as many of those around here these days
(+1 Funny) only if I laugh out loud.
Both from Mercury Interactive. Expensive, but excellent. LoadRunner in particular does an great job of load testing against any major RDBMS.
Ah, the spelling flame. the hallmark of the mindless. The calling card of the clueless. Truly an american icon!
btw, YHBT, thanks for playing. Now go back to your tinker toi linus plz. mkthnx.
check out samie on sourceforge...
If you design your code in the right ways testing is a straight forward process. I design my web applications using an object for each task the site needs to do. Then I can just write a test function that will run that object through a typical scenario to make sure everything works that should work and nothing works that shouldn't work (security tests). This is a fairly reasonable way to check for obvious problems with your site and is good to make sure you don't shoot yourself in the foot but you still need to test everything by hand now and then too. Just don't treat your web applications any different than any other application and you can test using the same methods any programmer uses.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
LOL!!! This needs to get modded up!
All my money went to Nigeria and all I got was this lousy sig. . .
That IS what you hired them for, eh?
Really picky on which Browser/OS/hardware it runs on, but very usefull once you get it going.
I think we'd still miss a number of problems, 'cos there's just so many different screens
Yes, a number of problems including spelling mistakes and bad grammar.
I'll have something intelligent to add one of these days...
(Jakarta) Apache's JMeter will do load testing
of not only web applications but
any HTTP service.
Taco:
AC: Taco, is it true what the wise say?
Taco:
AC: Please Taco, I must know.
Taco: Speak, boy.
AC: Is it true Taco? Dulce et decorum est pro patria mori.
Taco(whilst dropping his pants and bending over): Look at this... here will you find your answer
AC(now understanding The Truth): I see...
Taco(chanting):
AC: I'm understanding, Mr. Taco, I thought it was The Old Lie
Taco: No, my son, it surely is sweet and honorable, come boy, cum here
I hope you guys don't slaughter me for saying that Microsoft did a decent job, but check out:
WAST
and
WCAT
They both seem to work really well and are freely available if you agree to the license. It's been a while since I've used them but I think they'll work fine with testing an apache or any other web server.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
a big document on application testing can be found at the addy below.u reWebApplicationsAndWebServices-V1.0.pdf
http://www.cgisecurity.net/owasp/OWASPBuildingSec
My company's solution is pretty simple, get a full time guy to test everything continuously and log bugs :)
Memories become legend, Legend fades to myth, and even myth is forgotten by the time that age comes again.-Robert Jordan
Another approach is to design out the source of the problems you mention, by never ever using dynamic binding for things that static binding solves better (See my rant on this very topic at Gamelon/Earthweb, Nov 2001).
See http://virtualschool.edu/jwaa (Java Web Application Architecture) for an open source solution based on the above. In particular, note using static type-checking for reporting invalid links, and the use of centrally-mained Field libraries for getting input validation under control.
Finally, use junit for unit and integration testing.
For industrial age goods there were checks and credit cards. For everything else there is a href="http://virtualschool
Write your own. You really should anyway.
You really sound like you THINK your problem centers around testing when it really is more of a teamwork issue that is revealing a testing deficiency now that your projects are getting larger.
For example, your statement: Now, while we enjoy programming, we're pretty lazy when it comes to testing.
Ouch, bro. Testing IS part of programming. If you'd already incorporated testing into your process, building unit tests for planned iterations of code, then you would A) perceive the tests to be indispensable in your code design phase, and B), you'd see that you've already, almost assuredly (unless your development tasks are pretty near trivial) created far more work for yourself over time by being lazy about testing. The fact is, it's really pretty hard to work on large projects on larger teams while ignoring unit tests, no matter what team paradigm you're using.
But a better team paradigm might actually be your ticket. You may want to check out XP which makes testing, test frameworks, refactoring, and other team coding techniques integral to the overall process. But whatever you decide, the result of adopting any good team paradigm that solidly implements a flexible, necessary testing process is, in the long run, vastly increased efficiency, better code reuse, and less dependence on specific individuals.
Now that your code base is getting larger it sounds like what you're really seeking in testing solutions is a better overall team plan. And after all, if you really like the coding phase, well, the unit tests need to be coded as well. That's really what a Tools Developer does, anyway.
Peace.
http://home.attbi.com/~sept11/
It's real easy to think as a test tool as a panacea -- that you'll no longer require any testing effort. It's not true; do a very careful cost analysis before purchasing the tool.
I used to be the automated testing manager for my company, so I speak from experience. If your front end changes often, or is extremely data driven (especially if you have a large, shared, rules-driven database setup), you will spend significantly more time setting up the test suite than you will ever save running it.
Don't get me wrong -- automated testing tools are terrific in certain cases. In particular, for testing individual forms, deterministic behavior, and mathematical tests that your testers may find pedestrian or boring. But all processes that are even slightly complex or could have varying behavior depending on many parameters, are not appropriate for automated testing.
Also be sure that management is aware of the costs involved. The standard rule of thumb is that it takes about ten times as long to automate tests as it does to run them once. From experience, this number can move between five and fifty, but the point is this: don't try to automate everything, just those tests where you stand to run them repeatedly.
Given this, I hope you have a better sense of what's involved. Given your examples it seems like you're a case where automated testing can truly make you more efficient. Just remember to at least start with the tests that meet the criteria I've enumerated above, and you should be fine.
XMSGuardian's feature list includes:
- Crawl your site testing every component on every page
- Give you accurate metrics related to performance and errors
- Show you the related impact of error conditions
- Auto-complete forms dynamically to test server side functionality
- Execute pre-recorded paths through your application.
- Tons more...
I would invite anyone who is in need of quality, relative test results for your web applications to look into XMSGuardian at http://www.sentiat.com/.Have you used Mexicans?
I have been working on a number of applications. The one thing to prevent errors is design. I spet 2 months one time designing an application before I got aournd to writeing it, and no errors.
There are a large number of products in the market that are available to do automated testing.
Companies such as RSW Software, Segue, Rational and a million others out there, offer products that will do exactly what you want. The market is vast, and the offerings vary greatly (in features and price), so you'll have to pick you own.
As someone who spend from 96 to 2000 testing web apps, (I was actually QAing websites and early web applications in 96), I disagree with your comment (paraphrased) that a tool cant replicate careful manual testing.
Personally, I believe on very large and dynamic sites, manual testing simply cant give you the same coverage as a well designed automated test. Humans just cant move fast enough.
However, the underlieing issue is that testing, manual and automatic, is only as good as the test plans they are based around. Without someone writing good quality test plans, with (some) review from the developers themselves, then it doesn't matter how much you test, you're still going to miss bugs.
Good luck.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
...is Empirix. They have a whole portfolio of webtest products available. Nice stuff.
Your entire question, well, sucks. If you think you can test at the end of a product cycle, you're smoking the kind of crack cocain that leads to things like this.
When you write a function for your program, you need to write a test unit that is in the debug project. How it will work is that you write some tests in which you take an input, perform the operation, and test the output versus a contstant answer. Have one of these for each case that it handles in the unit. That way, you can always compile the test unit and examine its output versus the constant known-good value. That's good software engineering practice.
What you're asking, well, is a joke. Nothing's going to save your project if you've been just adding functionality without QAing at each step to verify correctness.
hellbunnia asks "I work with a team of developers who spend most of their time adding functionality to code. While we enjoy just cramming more code onto a source tree, we really never test anything. But even if we tested it, I think we'd miss a lot of bugs because we have no design policy. It's a lot to be tested, and it's all interrelated! So my question is, does anyone have a quick and easy solution that will save us from rewritting things with a proper design?"
"I've read a lot of freshmeat listings for testing, but I've always assumed that they were merely 'Hello, World' programs because nothing beats real testing by real humans. However, as the amount of code grows, I've begun to wish that we wrote a carefully designed set of unit tests as we added functionality, rather than trying to magically make it all work 2 weeks before our shipping deadline. I'm hoping we have some magic QA program which will do everything for us, except actually fix our squirrely code.
Does such a thing exist, or should I start updating my resume? How fucked am I?"
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
This is a subject that I am very interested in.
As a testing team lead for Web applications, I often have the debate as to why we don't automate more of our testing.
Automated testing is good for Regression testing ONLY!
Once your site is stable enough to use automated testing tool, you don't have to worry about it because you are out of a job.
While you are developing the site you should use validation tools.
There are many that are readily available, such as: Bobby for accessability testing; W3C for validation of HTML, CSS, XML...; Coast for link checking; Webglimpse for change control.
Log in to a site such QAForums for various opinion and reccomendations.
Do not rely on automation totally. Only a human with all their failings and experiences will be able to do what no automated tool can, the "stupid user stuff".
IMHO, developers should never be the testers of their own products. They have a mindset already about how the product should work and have already coded it to do what they'd expect.
Granted, they will find some bugs that were typos or minor mistakes made when coding, but they will not find a large percentage of the bugs.
Instead, get a seperate test group who knows the requirements (you do have requirements, don't you?) and can write proper bug reports.
Let developers be developers and testers be testers. The software will benefit.
-DCookie
My SIG is a SG-552 Commando
We've used Segue's Silk Test at our company (ArtSelect.com) and found it to be expensive and buggy. Not only that, when we get a new release to fix some of the bugs in the old one, there have been serious incompatibility issues (like we have to re-write many of our tests!) and/or new, more pernicious bugs. We're sorry we got the product -- it has had more bugs and problems than Windows!
Both Winrunner and Loadrunner are excellent for this type of testing. You can have them run through the site as an actual user would. You can also test your app's load handling ability (which is also based on the hardware). It has a learning curve, but any good web guy can figure out the interface with a little playing.
1+2+1+1 || 1+2+2+1
The nice thing about this approach is that the testing is done through the browser that the end user will most likely be using (like it or not). Phantom plays back keystrokes and mouse clicks and gets the timing right even when the page load timing varies tremendously (now that we have the IE helper). With the script language its easy to set timeouts and test for specific screen content. If the content doesn't match what is expected within a reasonable time, you can take a screen snapshot and save it away for later review.
We've got a set of parameterized scripts that we glue together (the script language has include libraries and procedures) to produce test suites for each release.
Aint it funny how y'all think your all so smart acting when you can't spell a word and can't form a good sentence when its just so easy to know English and it's syntax too?
Strict obedience to the law is the key to liberty.
SilkTest from Segue Solutions is very good too. $$$ but worth it.
www.segue.com
It is a good idea to think about how you are going to test the application at design time. There are dozens of good automated testing tools, but they all rely on repeating the same set of tests the same way. If the links are different or the data is different then your automated testing tool will be confused easily and provide you with very limited value.
Ask yourself:
Are we displaying dynamic data which is always changing?
If so, will the links be different every time we run the test tool?
How can we capture a dataset that we can use repeatedly for testing?
How will we change this dataset while the test tool is running to simulate various use cases?
Can we do anything to our application to make it play nicely with the testing tool?
Sometimes addressing these issues is harder than writing the application itself. If your application is interacting with physical real world systems that produce huge amounts of data then you have a big job ahead of you building a simulator that you can test against. Addressing these at design time can save alot of headaches later.
Last year I worked at a startup that was writing an instant messaging app consisting of a bunch of web pages and .jsp's. They created their own automated testing application in C++. I didn't examine the code closely, but it was essentially a screen scraper that navigated through the pages using a WebBrowser control and manipulating the DOM programatically -- entering text into input boxes, clicking checkboxes and buttons, checking results against expected results and writing out a log file. The test sequences were stored in a database, which they had a full-time person updating as the app changed.
Of course, any decent regression test suite ensures the negative cases fail as often as the positive cases pass.
You can't test simply one subset of the API.
Rod Taylor
We use jcookie to simplify the browser simulation within Java and Cactus.
Apache/Jakarta seems to have two tools for automated functional and load testing of web sites.
JMeter is more of a load testing tool, though it also can verify results. Latka seems to be more aimed at verifying the correctness of the HTTP response, rather than load testing. They are both extensible, open source, and Java. I like them over proprietary, closed, os-specific tools.
JMeter has a nice GUI, can graph the response time and show the http response for each request. It can also be run in batch mode and in server/client mode for distributed load testing.
They both have support for testing real-world web sites (keeps cookies, allows user authentication).
Seapine Software produces a product called QA Wizard that is a fully scriptable testing tool for web applications using Internet Explorer. Netscape/Java support is coming soon. A Windows application testing tool should be available by the end of the year, as well as a load testing tool.
--- igiveup ---
... you'll find it saves time and impresses your peers. (N.B. I can't guarantee it'll get you laid but the odds would shift in your favour)
Mercury Interactive has a good suite of tools for automating testing. Their web support is good for regression and volume testing
I had the same problem and only found one product that works for under $50,000. Try iOpus's Internet Macros (http://www.iopus.com/iim.htm). I bought the Scripting Edition ($320) and have been very pleased. Cheaper versions are available.
Try the demo. The demo's only limitation is that it can only record 10 steps.
rob2d2@softhome.net
Since that article was posted, I was asked by my company to do some load and scalability testing and I've had great success with OpenSTA. Give it a chance. It's awkward at once but once you get a feel for the HTTP/S (http scripting) language, you can do some very complicated scripting with it.
For example I wrote a script which interacts with one of our web products and navigates through several pages, submitting queries, retrieving 'wait' pages, and continuing on when the results are ready. Can't do that with wget... heh. And it gives excellent feeback on timing and can remotely monitor CPU and memory usage.
As far as I know it is only available on windows, though it is open source.
My previous comment posted to the Ask /. about Website Load Testing tools.
While the implementation is Smalltalk specific, Niall Ross and Andrew McQuiggin present an XP oriented approach to testing that may provide inspiration to slashdottes who can't make smalltalk. XML certainly is a possible approach for a non-Smalltalk implementation. In addition, John Metsker's Building Parsers with Java includes a flexible approach to testing that goes well beyond traditional JUnit. In particular it addresses the issue of generating ad hoc tests that exercise unexpected input.
This may be a little off the mark, but no matter how you test the actual transaction processing PLEASE make sure the output conforms to W3C specifications. I'm on a big standards kick (have been a Mozilla User since it could be distributed on a floppy), and I have seen too many dynamic web apps which totally break HTML and XHTML rules. One site I visit regularly actually declares itself to be written in XHTML STRICT 1.0 in the DOCTYPE when it is clearly not (looks more like HTML 4.01, but broken).
Yeah, I know "Web Standards" are like "Nobel Laurate-Supermodels". Still, I dream of a better tomorrow where I can use a standards-compliant browser without pages breaking, and have my cancer cured by someone who invented instantaneous travel while looking FAAAAABULOUS.
Check out the W3C HTML Validation Service to test your code for standards compliance. I love seeing those little "W3C: HTML 4.01" images all over the place!
</ENDRANT>
Plenty of test tools exist to automate testing of a Web application. I really like the idea of having an automated test system that would tell you in the morning what it found wrong on your site during a nightly check. I have built several such systems and they provide a big benefit back to the company in decreased down-time and improved user satisfaction. You will find details on how these test automation systems work in my upcoming book Testing Web Services. Try http://www.pushtotest.com/ptt/thebook.html to download the chapters. It's free and I would appreciate your feedback.
I would advise you to not take a decision to implement an automated test system lightly. Your decision commits your business to maintain the system and that can be expensive and complicated. All of the commercial test tools require an engineer to instrument all of the Web pages to be tested. They give you GUI tools to click through a Web site and the tool writes a test script that the test system can run. Eventually you wind up with a library of test scripts that need to be kept up-to-date as the Web site changes.
Additionally, these tools are reading Web pages to build scripts. One of HTML's shortcomings is that it mixes presentation data (font sizes, paragraph locations, etc.) with the actual content. HTML is very loosely formatted so test tools often fail to automate the script-writing process.
I've been building and testing complex interoperable systems for the past 15 years. In my experience the best way to build an automated test system is to give your software developers a test tool that lets them build tests while they are coding. The same tests may then be brought out of the developer's lab and used to check the service in production for scalability, performance and functionality.
One other thing to point out: there is little difference in functionality between the commercial test tools (which cost $20,000 to $50,000) and the free open-source test tools. I recommend you look at my open-source TestMaker project (http://www.pushtotest.com/ptt) and JMeter (http://www.apache.org.)
TestMaker comes with a graphic environment, script language, library of test objects (TOOL), sample test agents and a LOT of documentation. Plus my company PushToTest is the "go to" company for enterprises that need to test systems in Web environments. We're here to add functions needed by our customers, to run tests and to train your team in how to use the tool for their own needs.
Hope this helps. Feel free to drop me a line (fcohen@pushtotest.com) if you need additional help.
-Frank
Free open-source test automation tools and techniques at http://www.PushToTest.com
Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?
holy crap what an amazing idea! automated software testing of web applications! you're the first to come with this, so pat yourself on the back and run to the USPTO to patent this excellent idea!!
next time how about browsing the web (google helps!) for 5 mins before posting this kind of Ask Slashdot question...
E V E R Y T H I N G I W R I T E I S F A L S E
...is the tool we use here. I was involved in the search for a tool. I like it because it can parse the returned DOM and tell you things about it (like the value of the 3rd option in the select named "Blargh" in the form named "Foo") and make random picks in dropdowns in returned pages. Doing that with some of the other tools (including Mercury) required matching the entire page with regular expressions. Yuck. (I.e., you get the raw page text, a substring() function and a regexp matcher, if you were lucky.)
And the tests are developed in Javascript (Ecmascript), not some proprietary Pascal-like language, or something that required integration with custom C libs if you wanted custom functionality.
But, developing tests that don't trip over the slightest change in the UI (or underlying data) is definitely programmer-intensive.
John.
I'm building a web-application testing application and I'd like to know if there are any test applications I can use to test it.
who is this company? I've never been given a testing time and/or budget.....
DO NOT DISTURB THE SE
Whenever I write a web app I begin by creating a script just for that app that uses the browser object. As I add features, I add routines to the script that check that the features work. When I change anything, all I have to do is run the test script.
I don't have the Browser object on CPAN yet, but if you email me at miko at idocs dot com, I'll be happy to send you the package. Put WWW::Browser in the subject line.
Miko O'Sullivan
We used to deploy a product called Linkbot, who could crawl a site and fill in form and stuff, and make us a report, this was a very well thought product: the company who was doing it was called Watchfire but it seems their product range has changed.
We use Canoo WebTest, which is a very nice extension to HTTPUnit. We used to use Cruise Control for automated builds, but replaced it with Gump from Apache Jakarta as it seems to do a better job. We have been having a lot of trouble, however, trying to get the Canoo web tests integrated as part of the automated Gump builds, as it is very difficult to get all of our servers to shut down reliably every time the tests are done.
automated testing is a tricky thing. At the onset, sounds great. But in realty, there's a lot of work that goes into it. For some projects, automated testing is the "right thing" but for the majority of projects, it is not.
**Writing and maintaining automated test scripts takes lot time.** Someone else posted a metric of 10-1, which I believe is quite fair. You really need to treat those scripts as its own mini-development project. You need to map out scenarios for each script and what goal each should accomplish. Coding (yes, even for those record/playback tools... you need to spend quite a bit of time tweaking it). And testing. Testing test scripts? Absolutely. If your test scripts are wrong, you could end up masking real bugs and creating false confidence.
Now the questions you need to ask yourself along these lines are: What is the lifexpectancy of my application? How often do release new code to production? The relevance of these two questions are of a cost/benefit ratio. If I'm going to spend x amount of man-weeks (yes, weeks) to create an automated test suite, am I going to get the cost savings back when I know v2.0 is 8 months away? Maybe. What if I only do two releases in those 8 months? Most likely not. (if you're releasing code to a production system on a per fix basis... well that's another slashdot topic)
In lieu of automated testing, I do have a few suggestions for improving testing.
1) incorporate "impact analysis" as part of your design/code reviews. If someone is planning on touching function y in module x, your architect / tech lead / rest of developers should be able to identify what other areas are going to be affected. When it comes time to test, you know exactly what areas you need to really focus on and which areas can do with a spot check.
2) come up with a sensible schedule for bundling multiple code fixes into incremental releases. Every time you touch production, there's an inherent testing overhead. Bundle a multiple fixes together and that overhead is better distributed.
3) hire dedicated testers. Having someone full time on QA (or part time, split across multiple projects) does wonders. The good ones bring both a great deal of experience for finding "common errors" as well as a fresh perspective to the table to see things that the developers overlook because they're too deep in the trenches. Now of course, dedicated testers may not fit into the budget. Even if you can afford them, developers should always be on the hook for testing. Which brings me to my next point...
4) tell your developers that they better learn to test or fire them. sounds harsh, but testings part of the game. I don't want anyone who doesn't understand the value of testing -- and isn't willing to put in the effort to test -- on my team.
my 2 cents and then some...
Mercury Winrunner is a tool I use every day. If you have the Java Addin and Webtest Modules (both supplied by Mercury), you can easily test these front ends you've mentioned.
If you need help setting it up in the future, or if you're wondering how to do something in particular with winrunner, feel free to e-mail me.
I can be reached at ben@iqmail.net (starting monday).
There is absolutely nothing that'll find a bug as well as a good QA person who thinks "how can I break this?" However, that QA person should have recorded the sequence of events that breaks the code for two reasons...
- Reproducing the problem to show the prorgrammer.
- Regression testing to test that bugfix+1000 doesn't re-introduce the bug.
To me, reason 2 is why automated tools are valuable. Give me a QA person who can break my code in novel ways and knows how to run regressions and I'll crank solid code 3 times faster. Productivity and code quality around here dove when they laid off our single QA person.Here is a link to something along the same lines of your questioning, or rather, some of these tools will work in both ways (link check and load testing): slashdot.org
"...I'll need guns" --Chow Yun-Fat in 'Replacement Killers'
Hire an intern!
If you happen to read this post then I hope this bit of advice proves helpful. I work for one of the test tool companies mentioned above as a Sales Engineer. Essentially what I do are demos of these types of products and then Proofs of Concept. That is what I was going to advise...no matter what tools you look at or what the cost make sure that you either trial the product and play with it yourself or ask for a POC (Proof of Concept). The main difference is that with a POC someone with my job comes out to your site, installs the product, and then makes it work on your apps over a two day period. The advantage here is that you get an expert in the tool to do all the work for you and you get to watch and see what it takes to work with your application without burning up anywhere from a week to a month of your time.
Another thing to watch out for are whiz-bang demos. The demos are setup to show the strengths of the tools and to impress...the POC is where you are going to see the real proof (no pun intended). Basically take what you see with a grain of salt until you see it on your app...that is the most important thing...how does it work for you?
Finallly, if you do purchase a tool suite look at the pricing structure and how things break out very closely. Pay attention to the long term maintenance costs as well as the up front costs especially if there is a significant disount. Both are equally important.
Good luck with your search and feel free to email me if you like. Like I said I work for one of the companies selling these tools, so I do this on a daily basis. Make sure you do a POC and a proper evaluation of any tools you are considering. It may take some time, but there is a significant payoff when after the purchase you have the confidence that you bought what worked for you.
I think it was exactly what he was asking for. A tool to test websites. Someone mod me back on topic please.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
"Monkey is a real DA that has been around practically forever. The very
first low-memory global (0x100) is called MonkeyLives; it is a flag to let
apps know that Monkey is running so they can refuse to do destructive
things like replacing the Finder with a MacPaint document (which is the
sort of thing Monkey tries to do surprisingly often)."
"I can personally attest that in a user-interface-intensive program
Monkey finds bugs like nothing else I know, precisely because of its
randomness. As IM says, it just spits random keys and mouse clicks into
the app and waits for it to crash, so it gets into situations that no sane
human, tester or otherwise, would ever try--it just wouldn't occur to them."
I agree.
Jerk moderators modding me offtopic but why?
It's the same information (I missed this post earlier.) plus an additional link to WCAT which is not easily found.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
this chick is hot. if I answer the question do you think she'll have sex with me?
I've worked with JUnit and HTTPUnit and putting HTTPUnit tests that run over a wide variety of circumstances. Using HTTPUnit with JUnit makes for easy automation of testing (JUnit comes with an application that allows you to run the various test suites that you create--HTTPUnit is simply an extension of JUnit, so you can run HTTPUnit test suites in JUnit)
Of course, you still have to deal with browser issues. Just make sure those pages validate! (Not that it makes that much difference anyway.)
What I want is testing software that tests like a ten year old, but reports like a senior programmer!
What you need is a senile programmer, preferably with Alzheimers! Find some COBOL drone in his 70s, and let him file his bug reports on punched cards.
I am in the process of using HTTPUnit to retrofit a testing framework onto our JSP and Servelet powered site and boy is it fun. I'm not using it inside JUnit and it's great in a stand-alone app. The modeling of web pages and their components is very complete and easy to use. I am very easily able to follow from one page to the next.
I do have two big problems with it. The first is that it only accepts perfect HTML. If you are using tables to format a form, then HTTPUnit doesn't pick up all of the form elements if they are in different tables or even a nested table. It does allow you to add elements by hand, but then you aren't really doing what the user would. Also, as other people have mentioned, there is no JavaScript support, which sometimes makes for interesting work arounds, say if developers like to use JavaScript to submit forms. But once again, you have to know what you are looking for and are doing some thing that the user would not.
Good Luck.
I work on Web-based applications at Watchfire. One of our products is WebQA Interaction, which is a record-and-playback web client. While it is not intended primarily as a web application test tool, it does the job. And we do eat our own dogfood here. The focus is more on database-driven web sites.
Sorry, it runs only on Windows and is proprietary. Single seat licenses are $1495 US in the WebQA package. Free (as in beer) evaluation download available, registration required. You register and get the download location and keycode emailed to you.
Recently I have investigated this very issue with regards to automated web application testing. One of the most important things that came up in this venture was the requirement for managing service levels of a web application. To solve this problem a solution that could automatically perform web transactions and baseline the performance of the responses over time was necessary. During this time we looked at numerous web application testing tools, but none fully provided the flexibility and functionality that we needed. Mercury's software was close but still often failed creating a test case for complex web apps - beyond this the GUI and reporting interface was severely lacking. Anyway to make a long story short we've begun developing an application using many common Perl modules that allows to obtain accurate web transaction times and integrate those values into a network performance reporting tool. The abilities provided by Perl and the associated modules provided an amazing framework for obtaining the service level statistics we required.
Jeff
As featured on IBM's devWorks site ...Puffin Automation Framwork
What is it? see for your self. :)
The advantage to using an open source tool for doing this, rather than something sold by Rational, is that if you don't have a completely "normal" application, you can modify SPIKE Proxy in a few seconds to support whatever weird syntax you use. And by sending me patches and making SPIKE Proxy better (keep in mind this is the second python script I ever wrote) you help the whole community.
These have a product called Monitor Master which seems pretty flexible and will do at least part of what you want.
It will be on the front page of http://www.argogroup.com
In the hardware world, the great product testing standard is called "24 hour dock testing". That means that you put the damn thing out on the shipping dock, and if it's still there the next day, you ship it.
What's the equivalent to that in the web app world?
All pass beyond reach of medicine. None pass beyond the reach of love.
Too bad it's so damn expensive.
.vbs file you can easily modify the script and add dynamic variables such as a making a list of 1000 keywords, picking a random number and testing your search box [as one example] Not good for stress testing, but good for finding memory leaks and generating psuedo-traffic.
/., but since they run ads [perhaps unknowingly through doubleclick] for VS.NET I thought it would be approriate. Not to mention, you can run ACT against any web application; it doesn't matter because it just simulates clicking around.
If you have a copy of VS.NET, you can use the included ACT to test applications. You basically click on the links you want to test as it records and then you can run the scripts, simulating number of users and for how long. Since it outputs to a
I wouldn't have mentioned this on
Live web cams
The issue is not that things are changing....the issue is: WHY are they changing?
Requirements still matter. Always have. Always will. Denying that and pretending that "radical" codind techniques work is a dangerous ledge to stand on.
...or QA Partner (haven't heard much about QA Partner in years). We used WinRunner, Load Runner, and Test Director in house. Expen$ive $tuff.
OpenSTA
OpenSTA is primarily designed to be a pluggable test rig that has a lot of plugins designed for stress testing. It has served us very well and with a bit of scripting it can be adopted to do functional regression tests too.
I urge everyone to give OpenSTA a try especially if you're after a load testing solution. It's just a tool that's really powerful and well respected in the industry. And the best part is that it's Free as in OpenSource :).
Your pizza just the way you ought to have it.
I've used a few. I strongly recomend you invest in one. However you need to beware of the limitations of these tools. They only test what you tell them to test to make sure it works the same as last time. You will have trouble with dynamic data. (even Dates. The tool can be told to ignore things, but then it is ignoring data, so make sure it is ignoring the right thing)
These tools do NOT substitute for the first time through testing. You will still need a QA person to examine all known changes and verifty it they work right, and then tell the tool how to test for the new change.
It is a daily job (Often full time) to update the tool. In fact you should not let the tool guy go on vacation until he has a (several?) replacements who will do the job while he is away. In little time, enough changes that by the time you catch up you are often better off starting over from scratch. Do not let your updates slide, no matter what, or you will regret it.
The tool is not a substitute for first time testing. In fact if you want something that will only test your pages the first time you write them, you are better off doing it by hand, part of teaching the tool how to test a page is to test it while the tool watches. However once you have tested the page once, the tool has no problem testing it every day to make sure nobody accidenly changed something on it. Fortunatly this latter testing is the boring part nobody wants to do. Just make sure that everyone takes the time to write the test for each change. (or at least has the tools guy write the test, depending on your process)
We found that it was as much effort to write the test automation as to do the test for each version change (this was software not web pages), but once the test for each version was written you would press the button and run the test each time a patch was released, and everything would be tested. Once in a while bugs were found, but not very often. Many of the "bugs" found were not bugs, but changes in the way the product worked and we needed to change the script.
Finially the pay off, if there is one, will take more then a year. Warn your management right now about that. Somehow you need to keep metrics (and I'm not convinced any reasonable metrics exists to take) to compare the before and after case. Not everyone who has done test automation is convinced it was worth it. If you think it will take away a lot of the work you are doing now, then no it is not. If you want it to find a lot of bugs you are finding much later, then yes it is.
Overall, test automation is MORE work than you are doing now (just a guess, but likely), but it will catch more bugs faster. Try it, but remember a fair trial is a lot of work and it will take some time for the pay out.
The best bug finding mechanism around is to release the software as open source software. Many, many eyes will then be looking at the code and using it.. Other coders will email you improvements, patches, and all kinds of good helpful stuff..
It's the darn'dest thing ever, open source software, you should try it!
You should code the "core" or "engine" of the code before you add the GUI. Then problems like this wouldn't happen.
OK, so this is not always practical because requirements change. So how do you ensure that problems like this don't crop up?
Simple. Use unit tests. When the "function that handles formatting" is first written and used on web page X the coder writes a unit test for it - this easy, because there is no UI involved, call the function a few times and test the returned results.
That way, when another coder comes along to code web page Y, and decides he needs to change the "function that handles formatting", he knows he's mucked up because the unit tests start failing.
Catalytic Test by Catalytic Architects. An automated testing tool developed for creating robust simulations and usage scenarios, borne out of the needs of a team of web integrators to deliver well tested solutions to its clients.
http://www.catarc.com
www.segue.com
Look it up, its really expensive but really good. I was introduced to it because a previous employer had already purchased the licence before I was hired. It is a very easy "language" to learn and it can do anything that a person sitting in front of a GUI can. Great for regression tests.
They also have a load testing suite but I never used it so I cant really comment.
If you're using ASP.NET, you can use NUnitAsp to unit test your web pages. NUnitAsp is an extension to NUnit, a tool for Extreme Programming-style test-driven development with .NET.
Once you have a comprehensive automated suite of tests, you'll never go back. It gives you incredible confidence in your code. That confidence allows you to code much faster, because you can make risky changes secure in the knowledge that your tests will catch any foul-ups.
ASP.NET is for unit tests only. It's meant for programmers, not QA teams, and it's not very good for QA-style acceptance tests. It only tests server-side logic. JavaScript and other client-side code is ignored. But if you're using ASP.NET, it's an essential part of your toolset.
Jim Little
Slashdotting. Don't worry. It was probably just a little Slashdotting. Works fine now.
Another topic -- The U.S. government, Microsoft: Before you support the U.S. government in invading Iraq, you should know that the U.S. government has been (mostly secretly) causing violence in numerous countries. See What should be the response to violence? . (The article takes a long time to load, and is badly in need of updating.)
My research indicates that the U.S. government support for violence and Microsoft's inability to treat its customers well are related. They are both are part of a social breakdown caused by a kind of low-level mental disturbance in which people become progressively insensitive to themselves and others. See Windows XP Shows the Direction Microsoft is Going.
For testing the HTML itself:
Amazingly great software finds HTML errors, and edits HTML:
HTML Tidy (Win 32 version) finds HTML errors and corrects them automatically if possible. See the configuration options for HTML Tidy at HTML Tidy Quick Reference
HTML Tidy works best as a plug-in to HTML Kit. (The command-line software is used as the plugin.) HTML kit positions the editor at each line with an HTML error when you click on the error.
Truly awesome free software!
Just an idea if you want to roll your own solution:
save your web application state in a XML document. Have a database full of inputs and expected outputs (all as XML), run a script to go through them all and make sure they match up.
Microsoft's Application Center Test (ACT) is a freakin kick ass web-script testing tool / environment. It comes with the Enterprise Architect version of Visual Studio .Net, but is language / server / client agnostic. Simply record your actions using a normal web browser, save the scripts and re-run them infinately simulating x-number or connected users. It returns nicely formatted performance, and error reports.
i should have known better to follow a link to "somethingawful.com"
the image in the parent post is not very pleasant, a very close up of a penis.
There are 3 or 4 major companies providing Functional Test tools. Mercury and Segue both providing the two "favourite" tools.
Read through the forums at http://www.qaforums.com get a feel for what is available and if you wish ask some questions there (no offense to /. but you'll get a much higher signal:noise ratio with details on the pros and cons of each tool).
BetaSoft (owner of QAForums) also have a poll showing some stats for the preferance of the different products.
Good luck.
Yet another one: tclwebtest. Haven't gotten around to announce it on freshmeat yet since it's too unfinished in many regards, but it proved useful in a bunch of projects already.
... and it will also transform documents into XHTML... http://tidy.sourceforge.net/
test(this);