Slashdot Mirror
Testing Products for Web Applications?
"I've seen a lot of automated test suites advertised and I've always assumed that they were no substitute for careful testing by a human. However, as the number of web pages that we need to maintain grows, I've begun to wish that we had something that we could kick off at night, that would follow all links on our system and fill in values for the various forms it encountered, then when we arrived in the next morning there'd be some sort of report available detailing its findings. It could flag any pages that returned something obviously incorrect, such as a SQL error, a blank page or just the word 'error'.
Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?"
Mercury Interactive - www.mercuryinteractive.com has some products that will do this. I used them for a short while and they seemed pretty good.
Web Application Stress Tool (freebie from M$)
http://webtool.rte.microsoft.com/
You should check out Apache Cactus http://jakarta.apache.org/cactus/.
Well you hit upon one good way, you just forgot to post the link...of course if you did you'd be more worried about your server overloading than your web frontends not working correctly...
"I may be quite wrong." - Socrates
Well if you are working in Java, I've used Cactus before with success. It's based on junit, and allows you to do unit testing on servlets/jsp's in a nicely automated way. As long as you take the time to create good test cases, it can do quite a good job.
This sig has been temporarily disconnected or is no longer in service
I run a couple free web games ... and let me tell you ... if it has a security flaw, these people will find it! Hire a couple people that play my game! I'm sure they'll find any security flaws you may have!
... or end a SQL query in a text box and try to execute another query (or put in a sub query) ... and edit your query strings if you use GET (or build a query string and make sure that your program doesn't take a GET where it is looking for a POST) .... just a couple basics to try ... You might want to write a "validate_input" function for your forms as well ....
...
Seriously, I don't know of any software that does that, but if you find one, I'M INTERESTED!
I don't know if you're looking for advice or not, but try putting in negative numbers or things like #(-3+1000)
Hopefully that helps a little
HallmarkOrnaments.Com
Both Cactus and HttpUnit allow you to do unit tests on web components. Both are extensions of JUnit. Cactus allows you to do unit tests of servlets and JSPs, while HttpUnit allows for unit tests of the resulting HTML code. (Cactus also integrates HttpUnit to a certain degree.)
Obviously, these tools are targeted at Java development. I have less experience with HttpUnit than with Cactus, but I imagine it could be used as a general test suite.
>Weve just had 9/11/02, and bush is attacking
>Iraq, and your talking about TESTING PRODUCTS
>FOR WEB APPLICATIONS? MY GOD PEOPLE GET SOME
>PRIORTIES!!!!
If the web is full of buggy applications, the terrorists have already won.
(my talking about testing products? what?)
Take a look on Web Site Test Tools and Site Management Tools page. And of course shameless plug: HTTP-WebTest. If you will check the latest make sure to try it's beta version.
--
Ilya Martynov (http://martynov.org/)
I used to work on such a system, and I did a lot of the testing for it as well. We found quite a few bugs, but those were overwhelmed by the number of interface design flaws we encountered. There were literally hundreds of unnecessary mouse-clicks per user per day in the original interface, simply because the programmers never had to use their own software for days at a time.
So I guess my point is, make sure you don't simply rely on automated testing. A bot won't get sick of clicking unnecessary buttons, and won't develop RSI injuries. Humans will, and you'll get great feedback because of it. At my old company, the programmers were very nice about fixing these flaws once I brought them to their attention, and grateful for our input.
Cheers,
Anaphilius
Quick lesson in automated testing.
The only automated testing tools you can find is for regression tests. Basically, you make "build 1". You use the tool to 'record' the tests you currently run, and have it check for successes and failures. You make "build 2", and run the tests, to ensure everything that once worked, still works. Now you test the new stuff, record these tests with the tool, make "build 3", etc...
There are three major companies with good automated regression tools. Mercury Interactive's WinRunner, Rational's Robot, and Compuware's QA Center. All of them are great tools (and you can get them packaged with load testing tools if you'd like).
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Just post the link to your website on /., if it doesn't crash from the load then it's probably pretty good. Hey maybe Taco should look into this! He could start offering it as a service :)
The Anti-Blog
I recommend httperf and http_load for banging on lists of URLs really hard. At one place I worked, one of our developers rigged up some shell scripts that would play back log files through httperf and that worked pretty well.
If you want to record browser sessions for testing specific paths through the site, look at http-recorder or roboweb. There's also webchatpp, HTTP::WebTest, and HTTP::MonkeyWrench on CPAN. More info on this can be found on the mod_perl mailing list or on PerlMonks.
OK, maybe I am a little biased, as I have been in QA for 8 years. :-) But my comments still stand.
That said, we are currently using Rational's products to test our application, which includes a web piece. Hint: Don't use javascript if you plan on using Rational. They have SiteLoad, which I believe is free, but rest assured the rest of their products are NOT. Their licensing scheme is nothing short of trying to balance the budget of a small country. If you are wanting to implement their products in a big project, to handle requirements (Requisite Pro), Bugs (ClearQuest) and test plans (Test Manager), then prepare yourself for headaches. If you just want to get Rational Robot to record/playback user actions for testing, it is pretty solid. Rational purchased all different components of their system, so they aren't the smoothest to integrate. I have spent many hours with their phone support people.
I have also worked with Mercury and SilkTest, but to a lesser degree.
Oh, and if you are constantly changing critical code, you need to worry more about your development practices and not your testing.
My beliefs do not require that you agree with them.
I'd highly recommend picking the book:
Java Tools for eXtreme Programming
This is a great reference for all of the tools being mentioned and shows you how to integrate them into the development cycle if your using Java. You should be able to write the functional tests if your app is not written in Java.
As an aside, if your not developing these apps in Java, you really should look at using Tomcat, XDoclet and Struts for simple DB frontends, and then move to EJBs with JBoss, Jetty or Tomcat, Struts and XDoclet. If your lazy and don't want to write a lot of code, you'll love these tools. Reuse is high in Java, and the code generation tools like XDoclet take away most of the pain of using frameworks like EJB and Struts. Besides JSP taglibs allow me to have good looking pages made pretty by people who care about the differences between browsers for CSS, DHTML and what not.
Good Luck.
Arrogance is Confidence which lacks integrity. -- me
Your proirity in testing shouldn't rely on automation necessarily because what you are bound to find is that the application works perfectly, when it's following the script you've programmed. When somebody on my team brings me code/functionality to review, the first thing I try to do is to "do the wrong thing" (eg. letters in a field to be interpreted as numeric). Thorough testing requires "unbridaled" human ingenugity.
Frankly, what you need are probably consistent programming methods (because your front-ends are probably being written by liberal arts majors who taught themselves --insert language here--), through error handling, documentation, a consistent testing mothodology, and much more upfront requirements analysis.
This stuff ain't cheap and you need to factor it into your pricing. I'd say that 10% to 20% of your budget should be QA and testing and you should insist that the budget be used for that. Too often QA time is used for actual development, leaving no QA.
sarchasm: The gulf between the author of sarcastic wit and the person who doesn't get it.
XMSGuardian's feature list includes:
- Crawl your site testing every component on every page
- Give you accurate metrics related to performance and errors
- Show you the related impact of error conditions
- Auto-complete forms dynamically to test server side functionality
- Execute pre-recorded paths through your application.
- Tons more...
I would invite anyone who is in need of quality, relative test results for your web applications to look into XMSGuardian at http://www.sentiat.com/.Your entire question, well, sucks. If you think you can test at the end of a product cycle, you're smoking the kind of crack cocain that leads to things like this.
When you write a function for your program, you need to write a test unit that is in the debug project. How it will work is that you write some tests in which you take an input, perform the operation, and test the output versus a contstant answer. Have one of these for each case that it handles in the unit. That way, you can always compile the test unit and examine its output versus the constant known-good value. That's good software engineering practice.
What you're asking, well, is a joke. Nothing's going to save your project if you've been just adding functionality without QAing at each step to verify correctness.
hellbunnia asks "I work with a team of developers who spend most of their time adding functionality to code. While we enjoy just cramming more code onto a source tree, we really never test anything. But even if we tested it, I think we'd miss a lot of bugs because we have no design policy. It's a lot to be tested, and it's all interrelated! So my question is, does anyone have a quick and easy solution that will save us from rewritting things with a proper design?"
"I've read a lot of freshmeat listings for testing, but I've always assumed that they were merely 'Hello, World' programs because nothing beats real testing by real humans. However, as the amount of code grows, I've begun to wish that we wrote a carefully designed set of unit tests as we added functionality, rather than trying to magically make it all work 2 weeks before our shipping deadline. I'm hoping we have some magic QA program which will do everything for us, except actually fix our squirrely code.
Does such a thing exist, or should I start updating my resume? How fucked am I?"
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
-
WWW::Mechanize for the automated fetching.
- for the HTML validation
both of which I just happen to maintain.Seapine Software produces a product called QA Wizard that is a fully scriptable testing tool for web applications using Internet Explorer. Netscape/Java support is coming soon. A Windows application testing tool should be available by the end of the year, as well as a load testing tool.
--- igiveup ---
I've used a few. I strongly recomend you invest in one. However you need to beware of the limitations of these tools. They only test what you tell them to test to make sure it works the same as last time. You will have trouble with dynamic data. (even Dates. The tool can be told to ignore things, but then it is ignoring data, so make sure it is ignoring the right thing)
These tools do NOT substitute for the first time through testing. You will still need a QA person to examine all known changes and verifty it they work right, and then tell the tool how to test for the new change.
It is a daily job (Often full time) to update the tool. In fact you should not let the tool guy go on vacation until he has a (several?) replacements who will do the job while he is away. In little time, enough changes that by the time you catch up you are often better off starting over from scratch. Do not let your updates slide, no matter what, or you will regret it.
The tool is not a substitute for first time testing. In fact if you want something that will only test your pages the first time you write them, you are better off doing it by hand, part of teaching the tool how to test a page is to test it while the tool watches. However once you have tested the page once, the tool has no problem testing it every day to make sure nobody accidenly changed something on it. Fortunatly this latter testing is the boring part nobody wants to do. Just make sure that everyone takes the time to write the test for each change. (or at least has the tools guy write the test, depending on your process)
We found that it was as much effort to write the test automation as to do the test for each version change (this was software not web pages), but once the test for each version was written you would press the button and run the test each time a patch was released, and everything would be tested. Once in a while bugs were found, but not very often. Many of the "bugs" found were not bugs, but changes in the way the product worked and we needed to change the script.
Finially the pay off, if there is one, will take more then a year. Warn your management right now about that. Somehow you need to keep metrics (and I'm not convinced any reasonable metrics exists to take) to compare the before and after case. Not everyone who has done test automation is convinced it was worth it. If you think it will take away a lot of the work you are doing now, then no it is not. If you want it to find a lot of bugs you are finding much later, then yes it is.
Overall, test automation is MORE work than you are doing now (just a guess, but likely), but it will catch more bugs faster. Try it, but remember a fair trial is a lot of work and it will take some time for the pay out.