Slashdot Mirror
Testing Products for Web Applications?
"I've seen a lot of automated test suites advertised and I've always assumed that they were no substitute for careful testing by a human. However, as the number of web pages that we need to maintain grows, I've begun to wish that we had something that we could kick off at night, that would follow all links on our system and fill in values for the various forms it encountered, then when we arrived in the next morning there'd be some sort of report available detailing its findings. It could flag any pages that returned something obviously incorrect, such as a SQL error, a blank page or just the word 'error'.
Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?"
Mercury Interactive - www.mercuryinteractive.com has some products that will do this. I used them for a short while and they seemed pretty good.
With the right perl modules, or even perhaps in shellscript with things like CURL, you could roll your own http/html regression tester. Won't handle javascript of course, and won't notice browser-dependant problems. You might be able to find a generic javascript/DOM library for unix to do the JS thing from your regression tester though.
11*43+456^2
Web Application Stress Tool (freebie from M$)
http://webtool.rte.microsoft.com/
You should check out Apache Cactus http://jakarta.apache.org/cactus/.
http://jakarta.apache.org/commons/latka/index.html
it's a java XML solution to writing automated suites of functional tests. and it's free.
Well you hit upon one good way, you just forgot to post the link...of course if you did you'd be more worried about your server overloading than your web frontends not working correctly...
"I may be quite wrong." - Socrates
Well if you are working in Java, I've used Cactus before with success. It's based on junit, and allows you to do unit testing on servlets/jsp's in a nicely automated way. As long as you take the time to create good test cases, it can do quite a good job.
This sig has been temporarily disconnected or is no longer in service
I run a couple free web games ... and let me tell you ... if it has a security flaw, these people will find it! Hire a couple people that play my game! I'm sure they'll find any security flaws you may have!
... or end a SQL query in a text box and try to execute another query (or put in a sub query) ... and edit your query strings if you use GET (or build a query string and make sure that your program doesn't take a GET where it is looking for a POST) .... just a couple basics to try ... You might want to write a "validate_input" function for your forms as well ....
...
Seriously, I don't know of any software that does that, but if you find one, I'M INTERESTED!
I don't know if you're looking for advice or not, but try putting in negative numbers or things like #(-3+1000)
Hopefully that helps a little
HallmarkOrnaments.Com
Both Cactus and HttpUnit allow you to do unit tests on web components. Both are extensions of JUnit. Cactus allows you to do unit tests of servlets and JSPs, while HttpUnit allows for unit tests of the resulting HTML code. (Cactus also integrates HttpUnit to a certain degree.)
Obviously, these tools are targeted at Java development. I have less experience with HttpUnit than with Cactus, but I imagine it could be used as a general test suite.
>Weve just had 9/11/02, and bush is attacking
>Iraq, and your talking about TESTING PRODUCTS
>FOR WEB APPLICATIONS? MY GOD PEOPLE GET SOME
>PRIORTIES!!!!
If the web is full of buggy applications, the terrorists have already won.
(my talking about testing products? what?)
Take a look on Web Site Test Tools and Site Management Tools page. And of course shameless plug: HTTP-WebTest. If you will check the latest make sure to try it's beta version.
--
Ilya Martynov (http://martynov.org/)
I used to work on such a system, and I did a lot of the testing for it as well. We found quite a few bugs, but those were overwhelmed by the number of interface design flaws we encountered. There were literally hundreds of unnecessary mouse-clicks per user per day in the original interface, simply because the programmers never had to use their own software for days at a time.
So I guess my point is, make sure you don't simply rely on automated testing. A bot won't get sick of clicking unnecessary buttons, and won't develop RSI injuries. Humans will, and you'll get great feedback because of it. At my old company, the programmers were very nice about fixing these flaws once I brought them to their attention, and grateful for our input.
Cheers,
Anaphilius
Quick lesson in automated testing.
The only automated testing tools you can find is for regression tests. Basically, you make "build 1". You use the tool to 'record' the tests you currently run, and have it check for successes and failures. You make "build 2", and run the tests, to ensure everything that once worked, still works. Now you test the new stuff, record these tests with the tool, make "build 3", etc...
There are three major companies with good automated regression tools. Mercury Interactive's WinRunner, Rational's Robot, and Compuware's QA Center. All of them are great tools (and you can get them packaged with load testing tools if you'd like).
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Even if he programmed this, he would be too lazy to test it.
I know what he is talking about though. I like programming when something is a challenge, and I'm not sure how I'm going to accomplish it. But as soon as it gets down to small details and testing, i find it very tedious. Oh well... Those are the breaks!
Several comes to mind--Test Perspective, LoadPro, ActiveTest, etc. You can also buy your own software to do this, or write something in a script language.
s s_works.html) has the ability to randomly fill in forms with lists of data you give it. It will figure out what form it is, select the right list of data, submit the form and go to the next one. It can validate that the form returns the correct data too.
I'm most familiar with LoadPro and Test Perspective..and of course scripting it.
With Test Perspective, you can record the way the web app works, then have them play it back for you with lots of variations with however many number of users you want.
LoadPro (http://www.keynote.com/solutions/html/keyreadine
Scripting it yourself is pretty easy too, but you want to make sure you use one that does http 1.1 (perl LWP doesn't) and you want to model your users accurately.
As for purchasing a tool, there is SILK Performer and Segue, both traditional functionality testing tools
Donald E. Foss
No Not Again! Its whats for dinner.
Just post the link to your website on /., if it doesn't crash from the load then it's probably pretty good. Hey maybe Taco should look into this! He could start offering it as a service :)
The Anti-Blog
Actually all the information I've seen is that developers should not do *all* the code testing. You should do unit testing and you should write tests, but you should *not* depend on that - you need external testers also
There are many, and as others have rightly posted finding them on Google is easier than posting to /.
SilkTest from Segue is good at both scripted testing & stress testing.
--#voxlator
I recommend httperf and http_load for banging on lists of URLs really hard. At one place I worked, one of our developers rigged up some shell scripts that would play back log files through httperf and that worked pretty well.
If you want to record browser sessions for testing specific paths through the site, look at http-recorder or roboweb. There's also webchatpp, HTTP::WebTest, and HTTP::MonkeyWrench on CPAN. More info on this can be found on the mod_perl mailing list or on PerlMonks.
I agree with you but I believe the main point ( besides the obvious statement of laziness ) is that one change can affect many places and many things. e.g. A function that handles formatting may work now on one page but break another ( out of perhaps 70 possible pages )
The way we handle this at work ( in the eCommerce dpt ) is that when we use a function in a page we document it in a database. We can cross search what functions are used in a page or what pages use a function. This way when we make changes to a function that has a scope larger than the page we're working on we can test it in all of those scenarios
OK, maybe I am a little biased, as I have been in QA for 8 years. :-) But my comments still stand.
That said, we are currently using Rational's products to test our application, which includes a web piece. Hint: Don't use javascript if you plan on using Rational. They have SiteLoad, which I believe is free, but rest assured the rest of their products are NOT. Their licensing scheme is nothing short of trying to balance the budget of a small country. If you are wanting to implement their products in a big project, to handle requirements (Requisite Pro), Bugs (ClearQuest) and test plans (Test Manager), then prepare yourself for headaches. If you just want to get Rational Robot to record/playback user actions for testing, it is pretty solid. Rational purchased all different components of their system, so they aren't the smoothest to integrate. I have spent many hours with their phone support people.
I have also worked with Mercury and SilkTest, but to a lesser degree.
Oh, and if you are constantly changing critical code, you need to worry more about your development practices and not your testing.
My beliefs do not require that you agree with them.
wget and diff
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
I'd highly recommend picking the book:
Java Tools for eXtreme Programming
This is a great reference for all of the tools being mentioned and shows you how to integrate them into the development cycle if your using Java. You should be able to write the functional tests if your app is not written in Java.
As an aside, if your not developing these apps in Java, you really should look at using Tomcat, XDoclet and Struts for simple DB frontends, and then move to EJBs with JBoss, Jetty or Tomcat, Struts and XDoclet. If your lazy and don't want to write a lot of code, you'll love these tools. Reuse is high in Java, and the code generation tools like XDoclet take away most of the pain of using frameworks like EJB and Struts. Besides JSP taglibs allow me to have good looking pages made pretty by people who care about the differences between browsers for CSS, DHTML and what not.
Good Luck.
Arrogance is Confidence which lacks integrity. -- me
For a limited time only, the newest Robust Web Application Testing Program is available to you free of charge*. For this week only you can have your very own Slashdot Web Application Analyzer!
To use this miracle of modern computing, simply submit a story link to your Web Application and the webmaster's e-mail at the bottom of the page! Not only will you be able to test your server bandwidth, but every know-it-all Slashdot Web Guru(tm) will e-mail you with exactly why your Application is not worth the electrons it's stored on!
For added bonus, have your site flame one of the following groups for extremely extensive testing: Any Goverment, Adobe, Microsoft, Intel, Creative Labs, CowboyNeal.
Call Now!
Operators are standing by!
The problem is that the programmers who write the application know too much about the app and how the app is *supposed* to be used to be good testers. I believe that they will subconcsiously test the wrong things. You need "typical users" (less than typical, actually) to really get useful results. If the only real testing that's being done is being done by the programmers, watch out.
Clueless newbies and kids will find the problems first. The problem is that they don't report very well. What I want is testing software that tests like a ten year old, but reports like a senior programmer!
of commercial tools that support extensive
script-driven testing of web applications.
SilkTest is the testing tool.
At my previous startup, we bought and used
these tools and developed extensive test
libraries for our product.
There are also companies that will test your
product for usability on many different platforms.
Look at http://www.otivo.com/ for one such.
Some words of advice if you care to follow them.
First off, ignore anything with the words "stress" or "performance" in the titles or descriptions. They are not the tools you want, and are focused primarily on simulating multiple clients rather than simulating users.
Second off, seperate the kinds of testing you want to do. Simple form validation requirements will most likely mean you can get away with a tool that bypasses the browser interface (typically a unit testing tool). More complicated user simulation should be done by a tool that actually drives the browser, such as SilkTest or Rational.
Finally - Hire a dedicated resource just for this purpose. A QA Engineer with experience in automated testing, REAL experience, not just playback and record experience. (My resume is available on demand).
best web host ever
You might look up my friends over at F-Test. By focusing only on functionality testing, they're able to do it more efficiently than almost anyone. They can do it more thoroughly and cheaply than most companies can do themselves, even small shops like ours (and probably yours). They've done great work with our stuff, as well as for big corporate clients like Sony. Nothing beats a team of well-trained, experienced testers banging away at keyboards, but there aren't many people around focusing on just that. Look 'em up. They're in Los Angeles.
have them "use it like it'll get used" for a while. Note all the problems, fix them. From there, you can create use cases that'll let sobody else do the testing.
It's CRITICAL, IMHO, that the people requesting the application get directly involved with how the front ends should work. If they don't, you're just asking for UI rework pain.
ceci n'est pas un sig.
Your proirity in testing shouldn't rely on automation necessarily because what you are bound to find is that the application works perfectly, when it's following the script you've programmed. When somebody on my team brings me code/functionality to review, the first thing I try to do is to "do the wrong thing" (eg. letters in a field to be interpreted as numeric). Thorough testing requires "unbridaled" human ingenugity.
Frankly, what you need are probably consistent programming methods (because your front-ends are probably being written by liberal arts majors who taught themselves --insert language here--), through error handling, documentation, a consistent testing mothodology, and much more upfront requirements analysis.
This stuff ain't cheap and you need to factor it into your pricing. I'd say that 10% to 20% of your budget should be QA and testing and you should insist that the budget be used for that. Too often QA time is used for actual development, leaving no QA.
sarchasm: The gulf between the author of sarcastic wit and the person who doesn't get it.
Something tells me the photoshoots aren't happening in the server room or on developers' desks in cubicleville.
"I'm a leaf on the wind. Watch how I soar."
-Hoban Washburn
Well if it were just load based -- their are hundreds of programs that will automate and simulate till your hearts desire. That being said, I believe the question was more geared around how to test that when I hit the submit button -- does everything work like it should?
:) Everyone wants to claim QA in place, ISO Whatever in place...etc... The reality is, those were the first things to go.
The best thing to do is to ensure your testers are familiare enough with the back end and the transaction processes to be able to run cross checks on the Database -- to ensure everything is working as it should. Common things like missing where clauses on deletes, in statements like 'a,b,c' rather than 'a','b','c'. Just simple things that automated tools could never catch. The bad part is that things like this take time and bodies. Atr least were I am sitting -- not near as many of those around here these days
(+1 Funny) only if I laugh out loud.
If you design your code in the right ways testing is a straight forward process. I design my web applications using an object for each task the site needs to do. Then I can just write a test function that will run that object through a typical scenario to make sure everything works that should work and nothing works that shouldn't work (security tests). This is a fairly reasonable way to check for obvious problems with your site and is good to make sure you don't shoot yourself in the foot but you still need to test everything by hand now and then too. Just don't treat your web applications any different than any other application and you can test using the same methods any programmer uses.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
I hope you guys don't slaughter me for saying that Microsoft did a decent job, but check out:
WAST
and
WCAT
They both seem to work really well and are freely available if you agree to the license. It's been a while since I've used them but I think they'll work fine with testing an apache or any other web server.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
XMSGuardian's feature list includes:
- Crawl your site testing every component on every page
- Give you accurate metrics related to performance and errors
- Show you the related impact of error conditions
- Auto-complete forms dynamically to test server side functionality
- Execute pre-recorded paths through your application.
- Tons more...
I would invite anyone who is in need of quality, relative test results for your web applications to look into XMSGuardian at http://www.sentiat.com/.Your entire question, well, sucks. If you think you can test at the end of a product cycle, you're smoking the kind of crack cocain that leads to things like this.
When you write a function for your program, you need to write a test unit that is in the debug project. How it will work is that you write some tests in which you take an input, perform the operation, and test the output versus a contstant answer. Have one of these for each case that it handles in the unit. That way, you can always compile the test unit and examine its output versus the constant known-good value. That's good software engineering practice.
What you're asking, well, is a joke. Nothing's going to save your project if you've been just adding functionality without QAing at each step to verify correctness.
hellbunnia asks "I work with a team of developers who spend most of their time adding functionality to code. While we enjoy just cramming more code onto a source tree, we really never test anything. But even if we tested it, I think we'd miss a lot of bugs because we have no design policy. It's a lot to be tested, and it's all interrelated! So my question is, does anyone have a quick and easy solution that will save us from rewritting things with a proper design?"
"I've read a lot of freshmeat listings for testing, but I've always assumed that they were merely 'Hello, World' programs because nothing beats real testing by real humans. However, as the amount of code grows, I've begun to wish that we wrote a carefully designed set of unit tests as we added functionality, rather than trying to magically make it all work 2 weeks before our shipping deadline. I'm hoping we have some magic QA program which will do everything for us, except actually fix our squirrely code.
Does such a thing exist, or should I start updating my resume? How fucked am I?"
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
Last year I worked at a startup that was writing an instant messaging app consisting of a bunch of web pages and .jsp's. They created their own automated testing application in C++. I didn't examine the code closely, but it was essentially a screen scraper that navigated through the pages using a WebBrowser control and manipulating the DOM programatically -- entering text into input boxes, clicking checkboxes and buttons, checking results against expected results and writing out a log file. The test sequences were stored in a database, which they had a full-time person updating as the app changed.
Of course, any decent regression test suite ensures the negative cases fail as often as the positive cases pass.
You can't test simply one subset of the API.
Rod Taylor
Seapine Software produces a product called QA Wizard that is a fully scriptable testing tool for web applications using Internet Explorer. Netscape/Java support is coming soon. A Windows application testing tool should be available by the end of the year, as well as a load testing tool.
--- igiveup ---
Since that article was posted, I was asked by my company to do some load and scalability testing and I've had great success with OpenSTA. Give it a chance. It's awkward at once but once you get a feel for the HTTP/S (http scripting) language, you can do some very complicated scripting with it.
For example I wrote a script which interacts with one of our web products and navigates through several pages, submitting queries, retrieving 'wait' pages, and continuing on when the results are ready. Can't do that with wget... heh. And it gives excellent feeback on timing and can remotely monitor CPU and memory usage.
As far as I know it is only available on windows, though it is open source.
My previous comment posted to the Ask /. about Website Load Testing tools.
Plenty of test tools exist to automate testing of a Web application. I really like the idea of having an automated test system that would tell you in the morning what it found wrong on your site during a nightly check. I have built several such systems and they provide a big benefit back to the company in decreased down-time and improved user satisfaction. You will find details on how these test automation systems work in my upcoming book Testing Web Services. Try http://www.pushtotest.com/ptt/thebook.html to download the chapters. It's free and I would appreciate your feedback.
I would advise you to not take a decision to implement an automated test system lightly. Your decision commits your business to maintain the system and that can be expensive and complicated. All of the commercial test tools require an engineer to instrument all of the Web pages to be tested. They give you GUI tools to click through a Web site and the tool writes a test script that the test system can run. Eventually you wind up with a library of test scripts that need to be kept up-to-date as the Web site changes.
Additionally, these tools are reading Web pages to build scripts. One of HTML's shortcomings is that it mixes presentation data (font sizes, paragraph locations, etc.) with the actual content. HTML is very loosely formatted so test tools often fail to automate the script-writing process.
I've been building and testing complex interoperable systems for the past 15 years. In my experience the best way to build an automated test system is to give your software developers a test tool that lets them build tests while they are coding. The same tests may then be brought out of the developer's lab and used to check the service in production for scalability, performance and functionality.
One other thing to point out: there is little difference in functionality between the commercial test tools (which cost $20,000 to $50,000) and the free open-source test tools. I recommend you look at my open-source TestMaker project (http://www.pushtotest.com/ptt) and JMeter (http://www.apache.org.)
TestMaker comes with a graphic environment, script language, library of test objects (TOOL), sample test agents and a LOT of documentation. Plus my company PushToTest is the "go to" company for enterprises that need to test systems in Web environments. We're here to add functions needed by our customers, to run tests and to train your team in how to use the tool for their own needs.
Hope this helps. Feel free to drop me a line (fcohen@pushtotest.com) if you need additional help.
-Frank
Free open-source test automation tools and techniques at http://www.PushToTest.com
who is this company? I've never been given a testing time and/or budget.....
DO NOT DISTURB THE SE
Whenever I write a web app I begin by creating a script just for that app that uses the browser object. As I add features, I add routines to the script that check that the features work. When I change anything, all I have to do is run the test script.
I don't have the Browser object on CPAN yet, but if you email me at miko at idocs dot com, I'll be happy to send you the package. Put WWW::Browser in the subject line.
Miko O'Sullivan
automated testing is a tricky thing. At the onset, sounds great. But in realty, there's a lot of work that goes into it. For some projects, automated testing is the "right thing" but for the majority of projects, it is not.
**Writing and maintaining automated test scripts takes lot time.** Someone else posted a metric of 10-1, which I believe is quite fair. You really need to treat those scripts as its own mini-development project. You need to map out scenarios for each script and what goal each should accomplish. Coding (yes, even for those record/playback tools... you need to spend quite a bit of time tweaking it). And testing. Testing test scripts? Absolutely. If your test scripts are wrong, you could end up masking real bugs and creating false confidence.
Now the questions you need to ask yourself along these lines are: What is the lifexpectancy of my application? How often do release new code to production? The relevance of these two questions are of a cost/benefit ratio. If I'm going to spend x amount of man-weeks (yes, weeks) to create an automated test suite, am I going to get the cost savings back when I know v2.0 is 8 months away? Maybe. What if I only do two releases in those 8 months? Most likely not. (if you're releasing code to a production system on a per fix basis... well that's another slashdot topic)
In lieu of automated testing, I do have a few suggestions for improving testing.
1) incorporate "impact analysis" as part of your design/code reviews. If someone is planning on touching function y in module x, your architect / tech lead / rest of developers should be able to identify what other areas are going to be affected. When it comes time to test, you know exactly what areas you need to really focus on and which areas can do with a spot check.
2) come up with a sensible schedule for bundling multiple code fixes into incremental releases. Every time you touch production, there's an inherent testing overhead. Bundle a multiple fixes together and that overhead is better distributed.
3) hire dedicated testers. Having someone full time on QA (or part time, split across multiple projects) does wonders. The good ones bring both a great deal of experience for finding "common errors" as well as a fresh perspective to the table to see things that the developers overlook because they're too deep in the trenches. Now of course, dedicated testers may not fit into the budget. Even if you can afford them, developers should always be on the hook for testing. Which brings me to my next point...
4) tell your developers that they better learn to test or fire them. sounds harsh, but testings part of the game. I don't want anyone who doesn't understand the value of testing -- and isn't willing to put in the effort to test -- on my team.
my 2 cents and then some...
There is absolutely nothing that'll find a bug as well as a good QA person who thinks "how can I break this?" However, that QA person should have recorded the sequence of events that breaks the code for two reasons...
- Reproducing the problem to show the prorgrammer.
- Regression testing to test that bugfix+1000 doesn't re-introduce the bug.
To me, reason 2 is why automated tools are valuable. Give me a QA person who can break my code in novel ways and knows how to run regressions and I'll crank solid code 3 times faster. Productivity and code quality around here dove when they laid off our single QA person.I think it was exactly what he was asking for. A tool to test websites. Someone mod me back on topic please.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
Jerk moderators modding me offtopic but why?
It's the same information (I missed this post earlier.) plus an additional link to WCAT which is not easily found.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
As featured on IBM's devWorks site ...Puffin Automation Framwork
What is it? see for your self. :)
Too bad it's so damn expensive.
.vbs file you can easily modify the script and add dynamic variables such as a making a list of 1000 keywords, picking a random number and testing your search box [as one example] Not good for stress testing, but good for finding memory leaks and generating psuedo-traffic.
/., but since they run ads [perhaps unknowingly through doubleclick] for VS.NET I thought it would be approriate. Not to mention, you can run ACT against any web application; it doesn't matter because it just simulates clicking around.
If you have a copy of VS.NET, you can use the included ACT to test applications. You basically click on the links you want to test as it records and then you can run the scripts, simulating number of users and for how long. Since it outputs to a
I wouldn't have mentioned this on
Live web cams
Having used both, I honestly believe that Segue's Silk Performer is a superior load testing tool. I haven't used the Mercury tool in about a year, so it may have improved dramatically since then...
I guess it really depends on what you're testing - for finer grain control, I would choose LoadRunner, the moderately constrained C variant scripting language allow for some need tricks (of course, you can shoot yourself in the foot...always nice to have a memory leak in a test script, but it is nice to easily be able to call your own custom dlls and existing C code).
Silk Performer has some very nice playback and verification features, and their tool is much better for scripting at a higher level (IE, if your pages have a lot of javascript that dynamically builds links, handles form inputs...). The BDL is a Pascal-esque bastard language, and the script editor is awful.
So, LoadRunner: can generate tons of load not doing complex requests or workflows, SilkPerformer: can generate a lot of load and do a good job with complex workflows and funky scripting.
A: None. The Universe spins the bulb, and the Zen master merely stays out of the way.
WHY are requirements changing? Because they do, and they always will. No matter how hard you try to hold back the waterfall, it will drown you. Deal with the fact that you are lucky to get 80% of the most important requirements up front. Deal with the fact that some of those will change anyway, as the users get more comfortable with the solutions. You must be flexible and able to handle change or you'll bust a blood vessel in frustration at "wishy-washy" users.
Read, for example, Wicked Problems, Righteous Solutions , written in 1998. Not exactly radical bleeding edge stuff.
OpenSTA
OpenSTA is primarily designed to be a pluggable test rig that has a lot of plugins designed for stress testing. It has served us very well and with a bit of scripting it can be adopted to do functional regression tests too.
I urge everyone to give OpenSTA a try especially if you're after a load testing solution. It's just a tool that's really powerful and well respected in the industry. And the best part is that it's Free as in OpenSource :).
Your pizza just the way you ought to have it.
I've used a few. I strongly recomend you invest in one. However you need to beware of the limitations of these tools. They only test what you tell them to test to make sure it works the same as last time. You will have trouble with dynamic data. (even Dates. The tool can be told to ignore things, but then it is ignoring data, so make sure it is ignoring the right thing)
These tools do NOT substitute for the first time through testing. You will still need a QA person to examine all known changes and verifty it they work right, and then tell the tool how to test for the new change.
It is a daily job (Often full time) to update the tool. In fact you should not let the tool guy go on vacation until he has a (several?) replacements who will do the job while he is away. In little time, enough changes that by the time you catch up you are often better off starting over from scratch. Do not let your updates slide, no matter what, or you will regret it.
The tool is not a substitute for first time testing. In fact if you want something that will only test your pages the first time you write them, you are better off doing it by hand, part of teaching the tool how to test a page is to test it while the tool watches. However once you have tested the page once, the tool has no problem testing it every day to make sure nobody accidenly changed something on it. Fortunatly this latter testing is the boring part nobody wants to do. Just make sure that everyone takes the time to write the test for each change. (or at least has the tools guy write the test, depending on your process)
We found that it was as much effort to write the test automation as to do the test for each version change (this was software not web pages), but once the test for each version was written you would press the button and run the test each time a patch was released, and everything would be tested. Once in a while bugs were found, but not very often. Many of the "bugs" found were not bugs, but changes in the way the product worked and we needed to change the script.
Finially the pay off, if there is one, will take more then a year. Warn your management right now about that. Somehow you need to keep metrics (and I'm not convinced any reasonable metrics exists to take) to compare the before and after case. Not everyone who has done test automation is convinced it was worth it. If you think it will take away a lot of the work you are doing now, then no it is not. If you want it to find a lot of bugs you are finding much later, then yes it is.
Overall, test automation is MORE work than you are doing now (just a guess, but likely), but it will catch more bugs faster. Try it, but remember a fair trial is a lot of work and it will take some time for the pay out.
are you sure about the lack of javascript support? I am running the latest version and every test it complains that rhino.jar isnt found, so scripting is off. I wonder if someone hasn't just added it...
-a good trick with httpunit is to run it under ant's then the results in a summary page -this never fails to impress.
I agree that it is hard to automate UI Testing; you cant automate usability tests. But it is good for basic regression testing: making sure links are still there, etc. just think how much better the web would be if every site, every night, ran a regression test to verify that their local site worked and the external links werent 404-ing
Yes, 1990. I was quoting from amazon.com since my copy was at home (sitting in a place where I do a lot of reading, hehe). I have it in front of me now: Copyright 1990, Prentice-Hall 2nd printing paperback. Even more classic. It should be required reading.
I agree that it is hard to automate UI Testing; you cant automate usability tests.
I think that's true generally, but I've been thinking we can make some progress in that direction. There's an automated software design analysis tool called Small Worlds that offers opinions on OO design. It's pretty good.
I suspect that similar metrics could be calculated for web UIs so that we could help UI designers focus on dangerous areas, and so that they could be alerted if areas get worse. Even basic things like "links per page", "fields per form", and "percentage of page below the fold" would give you reminders to find pages that were unusually complicated for your site.
Slashdotting. Don't worry. It was probably just a little Slashdotting. Works fine now.
Another topic -- The U.S. government, Microsoft: Before you support the U.S. government in invading Iraq, you should know that the U.S. government has been (mostly secretly) causing violence in numerous countries. See What should be the response to violence? . (The article takes a long time to load, and is badly in need of updating.)
My research indicates that the U.S. government support for violence and Microsoft's inability to treat its customers well are related. They are both are part of a social breakdown caused by a kind of low-level mental disturbance in which people become progressively insensitive to themselves and others. See Windows XP Shows the Direction Microsoft is Going.
For testing the HTML itself:
Amazingly great software finds HTML errors, and edits HTML:
HTML Tidy (Win 32 version) finds HTML errors and corrects them automatically if possible. See the configuration options for HTML Tidy at HTML Tidy Quick Reference
HTML Tidy works best as a plug-in to HTML Kit. (The command-line software is used as the plugin.) HTML kit positions the editor at each line with an HTML error when you click on the error.
Truly awesome free software!