Testing Products for Web Applications?

hellbunnie asks: "I work with a team of developers many of whom spend much of their time writing web-based front-ends for DBs or other applications. Now, while we enjoy programming, we're pretty lazy when it comes to testing. Even if we weren't so lazy, I think we'd still miss a number of problems, 'cos there's just so many different screens that use any particular method/function that you might change. That means there's a lot to be tested after each change. So, my question is has anyone any experience with automated systems for testing web applications?"

"I've seen a lot of automated test suites advertised and I've always assumed that they were no substitute for careful testing by a human. However, as the number of web pages that we need to maintain grows, I've begun to wish that we had something that we could kick off at night, that would follow all links on our system and fill in values for the various forms it encountered, then when we arrived in the next morning there'd be some sort of report available detailing its findings. It could flag any pages that returned something obviously incorrect, such as a SQL error, a blank page or just the word 'error'.

Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?"

Comments

mercury

[psychopenguin]

Mercury Interactive - www.mercuryinteractive.com has some products that will do this. I used them for a short while and they seemed pretty good.

Re:mercury

[Anonymous Coward]

I've used Mercury loadrunner and winrunner for a couple of years and they are very good. The scripts created can be manipulated with C. They also have a very good customer support with people actually knowing their s***.

Downside: Expensive (but it is all relative to the savings to be made.)

Re:mercury

[SpeakerOfTruth]

My company was using Mercury Interactive's WinRunner for a while, but has since switched to Seapine's QA Wizard. We found that with WinRunner, it took a long time to create the scripts - which sometimes took longer because it semi-frequently crashes. We ended up purchasing the advanced training class that WinRunner offers, but the training received only seemed rudimentary. It also requires that your QA people have the ability to program - which I am not sure is a fair requirement. After about 12-14 months of WinRunner, we gave up on it. At the moment, we are using QA Wizard. We haven't been using it for very long, but it does seem to be much easier for the QA people to create their test scripts and test the web pages.

If you didn't already know about it...

[Boss, Pointy Haired]

Web Application Stress Tool (freebie from M$)

http://webtool.rte.microsoft.com/

Cactus

[DevilM]

You should check out Apache Cactus http://jakarta.apache.org/cactus/.

try a latka

[Anonymous Coward]

http://jakarta.apache.org/commons/latka/index.html

it's a java XML solution to writing automated suites of functional tests. and it's free.

Cactus

[sterno]

Well if you are working in Java, I've used Cactus before with success. It's based on junit, and allows you to do unit testing on servlets/jsp's in a nicely automated way. As long as you take the time to create good test cases, it can do quite a good job.

Cactus or HTTPUnit

[revscat]

Both Cactus and HttpUnit allow you to do unit tests on web components. Both are extensions of JUnit. Cactus allows you to do unit tests of servlets and JSPs, while HttpUnit allows for unit tests of the resulting HTML code. (Cactus also integrates HttpUnit to a certain degree.)

Obviously, these tools are targeted at Java development. I have less experience with HttpUnit than with Cactus, but I imagine it could be used as a general test suite.

Re:Cactus or HTTPUnit

[slagdogg]

IIRC, HttpUnit interacts with the site it is testing entirely at the HTTP
protocol level. There is an object model for parsing elements of web pages
to check values, set form elements, etc. While the tests themselves are
written in Java (not to mention the tool), I believe the tool is capable of
testing sites created using other tools and frameworks. I've not used it
myself, but it does seem pretty capable from the docs and things I've heard
from folks who have.

Web Site Test Tools

[m_ilya]

Take a look on Web Site Test Tools and Site Management Tools page. And of course shameless plug: HTTP-WebTest. If you will check the latest make sure to try it's beta version.

Automated testing.

[FortKnox]

Quick lesson in automated testing.
The only automated testing tools you can find is for regression tests. Basically, you make "build 1". You use the tool to 'record' the tests you currently run, and have it check for successes and failures. You make "build 2", and run the tests, to ensure everything that once worked, still works. Now you test the new stuff, record these tests with the tool, make "build 3", etc...

There are three major companies with good automated regression tools. Mercury Interactive's WinRunner, Rational's Robot, and Compuware's QA Center. All of them are great tools (and you can get them packaged with load testing tools if you'd like).

There are services that can do this for you

[DFossmeister]

Several comes to mind--Test Perspective, LoadPro, ActiveTest, etc. You can also buy your own software to do this, or write something in a script language.

I'm most familiar with LoadPro and Test Perspective..and of course scripting it.

With Test Perspective, you can record the way the web app works, then have them play it back for you with lots of variations with however many number of users you want.

LoadPro (http://www.keynote.com/solutions/html/keyreadines s_works.html) has the ability to randomly fill in forms with lists of data you give it. It will figure out what form it is, select the right list of data, submit the form and go to the next one. It can validate that the form returns the correct data too.

Scripting it yourself is pretty easy too, but you want to make sure you use one that does http 1.1 (perl LWP doesn't) and you want to model your users accurately.

As for purchasing a tool, there is SILK Performer and Segue, both traditional functionality testing tools

Donald E. Foss

One of many...

[voxlator]

There are many, and as others have rightly posted finding them on Google is easier than posting to /.

SilkTest from Segue is good at both scripted testing & stress testing.

--#voxlator

many open source test frameworks available

[consumer]

My experience with commercial load-testing apps is that they are outrageously expensive, a pain to program, don't really scale all that well, and mostly have to run on Windows with someone sitting at the mouse. There are some that work better than others, but the free stuff in this areas is quite good.

I recommend httperf and http_load for banging on lists of URLs really hard. At one place I worked, one of our developers rigged up some shell scripts that would play back log files through httperf and that worked pretty well.

If you want to record browser sessions for testing specific paths through the site, look at http-recorder or roboweb. There's also webchatpp, HTTP::WebTest, and HTTP::MonkeyWrench on CPAN. More info on this can be found on the mod_perl mailing list or on PerlMonks.

Get a good QA person

[gosand]

If you are a developer, do what you do best. If you want a tester, go out and find a good one. They are worth their weight in gold.

OK, maybe I am a little biased, as I have been in QA for 8 years. :-) But my comments still stand.

That said, we are currently using Rational's products to test our application, which includes a web piece. Hint: Don't use javascript if you plan on using Rational. They have SiteLoad, which I believe is free, but rest assured the rest of their products are NOT. Their licensing scheme is nothing short of trying to balance the budget of a small country. If you are wanting to implement their products in a big project, to handle requirements (Requisite Pro), Bugs (ClearQuest) and test plans (Test Manager), then prepare yourself for headaches. If you just want to get Rational Robot to record/playback user actions for testing, it is pretty solid. Rational purchased all different components of their system, so they aren't the smoothest to integrate. I have spent many hours with their phone support people.

I have also worked with Mercury and SilkTest, but to a lesser degree.

Oh, and if you are constantly changing critical code, you need to worry more about your development practices and not your testing.

Java Tools

[GOD_ALMIGHTY]

A couple of people have already mentioned the Jakarta project and cactus in particular.
I'd highly recommend picking the book:
Java Tools for eXtreme Programming

This is a great reference for all of the tools being mentioned and shows you how to integrate them into the development cycle if your using Java. You should be able to write the functional tests if your app is not written in Java.

As an aside, if your not developing these apps in Java, you really should look at using Tomcat, XDoclet and Struts for simple DB frontends, and then move to EJBs with JBoss, Jetty or Tomcat, Struts and XDoclet. If your lazy and don't want to write a lot of code, you'll love these tools. Reuse is high in Java, and the code generation tools like XDoclet take away most of the pain of using frameworks like EJB and Struts. Besides JSP taglibs allow me to have good looking pages made pretty by people who care about the differences between browsers for CSS, DHTML and what not.

Good Luck.

Silk

[thvv]

Segue (http://www.segue.com/) makes a set
of commercial tools that support extensive
script-driven testing of web applications.
SilkTest is the testing tool.

At my previous startup, we bought and used
these tools and developed extensive test
libraries for our product.

There are also companies that will test your
product for usability on many different platforms.
Look at http://www.otivo.com/ for one such.

I might get murdered, but Microsoft WAST does well

[Mustang Matt]

I hope you guys don't slaughter me for saying that Microsoft did a decent job, but check out:
WAST

and

WCAT

They both seem to work really well and are freely available if you agree to the license. It's been a while since I've used them but I think they'll work fine with testing an apache or any other web server.

There's a better solution than Mercury!!!!!

[codeking24]

Our company, Sentiat Technologies, Inc., has developed a better solution to having several humans bangings on keyboards or to using very expensive products from Mercury or Tonic. We have a new product that automates the testing process for web based applications called XMSGuardian.

XMSGuardian's feature list includes:

• Crawl your site testing every component on every page
• Give you accurate metrics related to performance and errors
• Show you the related impact of error conditions
• Auto-complete forms dynamically to test server side functionality
• Execute pre-recorded paths through your application.
• Tons more...

I would invite anyone who is in need of quality, relative test results for your web applications to look into XMSGuardian at http://www.sentiat.com/.

QA Wizard

[igiveup]

Seapine Software produces a product called QA Wizard that is a fully scriptable testing tool for web applications using Internet Explorer. Netscape/Java support is coming soon. A Windows application testing tool should be available by the end of the year, as well as a load testing tool.

PushToTest and TestMaker

[fcohen]

Plenty of test tools exist to automate testing of a Web application. I really like the idea of having an automated test system that would tell you in the morning what it found wrong on your site during a nightly check. I have built several such systems and they provide a big benefit back to the company in decreased down-time and improved user satisfaction. You will find details on how these test automation systems work in my upcoming book Testing Web Services. Try http://www.pushtotest.com/ptt/thebook.html to download the chapters. It's free and I would appreciate your feedback.

I would advise you to not take a decision to implement an automated test system lightly. Your decision commits your business to maintain the system and that can be expensive and complicated. All of the commercial test tools require an engineer to instrument all of the Web pages to be tested. They give you GUI tools to click through a Web site and the tool writes a test script that the test system can run. Eventually you wind up with a library of test scripts that need to be kept up-to-date as the Web site changes.

Additionally, these tools are reading Web pages to build scripts. One of HTML's shortcomings is that it mixes presentation data (font sizes, paragraph locations, etc.) with the actual content. HTML is very loosely formatted so test tools often fail to automate the script-writing process.

I've been building and testing complex interoperable systems for the past 15 years. In my experience the best way to build an automated test system is to give your software developers a test tool that lets them build tests while they are coding. The same tests may then be brought out of the developer's lab and used to check the service in production for scalability, performance and functionality.

One other thing to point out: there is little difference in functionality between the commercial test tools (which cost $20,000 to $50,000) and the free open-source test tools. I recommend you look at my open-source TestMaker project (http://www.pushtotest.com/ptt) and JMeter (http://www.apache.org.)

TestMaker comes with a graphic environment, script language, library of test objects (TOOL), sample test agents and a LOT of documentation. Plus my company PushToTest is the "go to" company for enterprises that need to test systems in Web environments. We're here to add functions needed by our customers, to run tests and to train your team in how to use the tool for their own needs.

Hope this helps. Feel free to drop me a line (fcohen@pushtotest.com) if you need additional help.

-Frank

Use the Puffin Web Testing Framework

[dismayed]

As featured on IBM's devWorks site ...Puffin Automation Framwork

What is it? see for your self. :)

As it's name implies, Puffin allows you to automate "actions." An action, in terms of Puffin, is any "high level" execution item that may require inputs, may produce outputs, and whose results may be validated for success or failure. For example, an action may involve making an HTTP request to a dynamic web page. It may involve grabbing a file's contents or even retrieving a specific email based on a keyword in the subject line. All of these are actions in the sense that they can be automated by Puffin. Puffin will manage all inputs, outputs, and validation for these actions.

Re:WinRunner and LoadRunner

[3am]

Having used both, I honestly believe that Segue's Silk Performer is a superior load testing tool. I haven't used the Mercury tool in about a year, so it may have improved dramatically since then...

I guess it really depends on what you're testing - for finer grain control, I would choose LoadRunner, the moderately constrained C variant scripting language allow for some need tricks (of course, you can shoot yourself in the foot...always nice to have a memory leak in a test script, but it is nice to easily be able to call your own custom dlls and existing C code).

Silk Performer has some very nice playback and verification features, and their tool is much better for scripting at a higher level (IE, if your pages have a lot of javascript that dynamically builds links, handles form inputs...). The BDL is a Pascal-esque bastard language, and the script editor is awful.

So, LoadRunner: can generate tons of load not doing complex requests or workflows, SilkPerformer: can generate a lot of load and do a good job with complex workflows and funky scripting.

My expirence

[bluGill]

I've used a few. I strongly recomend you invest in one. However you need to beware of the limitations of these tools. They only test what you tell them to test to make sure it works the same as last time. You will have trouble with dynamic data. (even Dates. The tool can be told to ignore things, but then it is ignoring data, so make sure it is ignoring the right thing)

These tools do NOT substitute for the first time through testing. You will still need a QA person to examine all known changes and verifty it they work right, and then tell the tool how to test for the new change.

It is a daily job (Often full time) to update the tool. In fact you should not let the tool guy go on vacation until he has a (several?) replacements who will do the job while he is away. In little time, enough changes that by the time you catch up you are often better off starting over from scratch. Do not let your updates slide, no matter what, or you will regret it.

The tool is not a substitute for first time testing. In fact if you want something that will only test your pages the first time you write them, you are better off doing it by hand, part of teaching the tool how to test a page is to test it while the tool watches. However once you have tested the page once, the tool has no problem testing it every day to make sure nobody accidenly changed something on it. Fortunatly this latter testing is the boring part nobody wants to do. Just make sure that everyone takes the time to write the test for each change. (or at least has the tools guy write the test, depending on your process)

We found that it was as much effort to write the test automation as to do the test for each version change (this was software not web pages), but once the test for each version was written you would press the button and run the test each time a patch was released, and everything would be tested. Once in a while bugs were found, but not very often. Many of the "bugs" found were not bugs, but changes in the way the product worked and we needed to change the script.

Finially the pay off, if there is one, will take more then a year. Warn your management right now about that. Somehow you need to keep metrics (and I'm not convinced any reasonable metrics exists to take) to compare the before and after case. Not everyone who has done test automation is convinced it was worth it. If you think it will take away a lot of the work you are doing now, then no it is not. If you want it to find a lot of bugs you are finding much later, then yes it is.

Overall, test automation is MORE work than you are doing now (just a guess, but likely), but it will catch more bugs faster. Try it, but remember a fair trial is a lot of work and it will take some time for the pay out.