Slashdot Mirror
Testing Products for Web Applications?
"I've seen a lot of automated test suites advertised and I've always assumed that they were no substitute for careful testing by a human. However, as the number of web pages that we need to maintain grows, I've begun to wish that we had something that we could kick off at night, that would follow all links on our system and fill in values for the various forms it encountered, then when we arrived in the next morning there'd be some sort of report available detailing its findings. It could flag any pages that returned something obviously incorrect, such as a SQL error, a blank page or just the word 'error'.
Does such a thing exist or am I just engaging in wishful thinking to imagine that there might be something flexible enough to do the job? What do other people do to test their software?"
I run a couple free web games ... and let me tell you ... if it has a security flaw, these people will find it! Hire a couple people that play my game! I'm sure they'll find any security flaws you may have!
... or end a SQL query in a text box and try to execute another query (or put in a sub query) ... and edit your query strings if you use GET (or build a query string and make sure that your program doesn't take a GET where it is looking for a POST) .... just a couple basics to try ... You might want to write a "validate_input" function for your forms as well ....
...
Seriously, I don't know of any software that does that, but if you find one, I'M INTERESTED!
I don't know if you're looking for advice or not, but try putting in negative numbers or things like #(-3+1000)
Hopefully that helps a little
HallmarkOrnaments.Com
By far the best tools out there, use Google to find them, but they cost quite a bit. All of these solutions require quite a bit of scripting and customization for full testing. For completely automated testing, well, try hiring someone. --ellis
GetTheJob.com : Nothing but Real Jobs.
I used to work on such a system, and I did a lot of the testing for it as well. We found quite a few bugs, but those were overwhelmed by the number of interface design flaws we encountered. There were literally hundreds of unnecessary mouse-clicks per user per day in the original interface, simply because the programmers never had to use their own software for days at a time.
So I guess my point is, make sure you don't simply rely on automated testing. A bot won't get sick of clicking unnecessary buttons, and won't develop RSI injuries. Humans will, and you'll get great feedback because of it. At my old company, the programmers were very nice about fixing these flaws once I brought them to their attention, and grateful for our input.
Cheers,
Anaphilius
Actually all the information I've seen is that developers should not do *all* the code testing. You should do unit testing and you should write tests, but you should *not* depend on that - you need external testers also
The problem is that the programmers who write the application know too much about the app and how the app is *supposed* to be used to be good testers. I believe that they will subconcsiously test the wrong things. You need "typical users" (less than typical, actually) to really get useful results. If the only real testing that's being done is being done by the programmers, watch out.
Clueless newbies and kids will find the problems first. The problem is that they don't report very well. What I want is testing software that tests like a ten year old, but reports like a senior programmer!
if you are constantly changing critical code, you need to worry more about your development practices and not your testing.
Not true in many developement shops. With short iterations, refactoring, rigorous unit testing, collective code ownership, and continuous integration, code can be constantly changing but stable. Take for example the Mozilla Tinderbox. Development proceeds on many components and the builds and tests are run continuously. There are daily build smoketests (download a daily build and you'll see the smoketest menuitem), and sometimes things are broken for an hour or a day, but overall things just get better.
Embrace Change.
Your proirity in testing shouldn't rely on automation necessarily because what you are bound to find is that the application works perfectly, when it's following the script you've programmed. When somebody on my team brings me code/functionality to review, the first thing I try to do is to "do the wrong thing" (eg. letters in a field to be interpreted as numeric). Thorough testing requires "unbridaled" human ingenugity.
Frankly, what you need are probably consistent programming methods (because your front-ends are probably being written by liberal arts majors who taught themselves --insert language here--), through error handling, documentation, a consistent testing mothodology, and much more upfront requirements analysis.
This stuff ain't cheap and you need to factor it into your pricing. I'd say that 10% to 20% of your budget should be QA and testing and you should insist that the budget be used for that. Too often QA time is used for actual development, leaving no QA.
sarchasm: The gulf between the author of sarcastic wit and the person who doesn't get it.
LOL!!! This needs to get modded up!
All my money went to Nigeria and all I got was this lousy sig. . .
Your entire question, well, sucks. If you think you can test at the end of a product cycle, you're smoking the kind of crack cocain that leads to things like this.
When you write a function for your program, you need to write a test unit that is in the debug project. How it will work is that you write some tests in which you take an input, perform the operation, and test the output versus a contstant answer. Have one of these for each case that it handles in the unit. That way, you can always compile the test unit and examine its output versus the constant known-good value. That's good software engineering practice.
What you're asking, well, is a joke. Nothing's going to save your project if you've been just adding functionality without QAing at each step to verify correctness.
hellbunnia asks "I work with a team of developers who spend most of their time adding functionality to code. While we enjoy just cramming more code onto a source tree, we really never test anything. But even if we tested it, I think we'd miss a lot of bugs because we have no design policy. It's a lot to be tested, and it's all interrelated! So my question is, does anyone have a quick and easy solution that will save us from rewritting things with a proper design?"
"I've read a lot of freshmeat listings for testing, but I've always assumed that they were merely 'Hello, World' programs because nothing beats real testing by real humans. However, as the amount of code grows, I've begun to wish that we wrote a carefully designed set of unit tests as we added functionality, rather than trying to magically make it all work 2 weeks before our shipping deadline. I'm hoping we have some magic QA program which will do everything for us, except actually fix our squirrely code.
Does such a thing exist, or should I start updating my resume? How fucked am I?"
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
-
WWW::Mechanize for the automated fetching.
- for the HTML validation
both of which I just happen to maintain.automated testing is a tricky thing. At the onset, sounds great. But in realty, there's a lot of work that goes into it. For some projects, automated testing is the "right thing" but for the majority of projects, it is not.
**Writing and maintaining automated test scripts takes lot time.** Someone else posted a metric of 10-1, which I believe is quite fair. You really need to treat those scripts as its own mini-development project. You need to map out scenarios for each script and what goal each should accomplish. Coding (yes, even for those record/playback tools... you need to spend quite a bit of time tweaking it). And testing. Testing test scripts? Absolutely. If your test scripts are wrong, you could end up masking real bugs and creating false confidence.
Now the questions you need to ask yourself along these lines are: What is the lifexpectancy of my application? How often do release new code to production? The relevance of these two questions are of a cost/benefit ratio. If I'm going to spend x amount of man-weeks (yes, weeks) to create an automated test suite, am I going to get the cost savings back when I know v2.0 is 8 months away? Maybe. What if I only do two releases in those 8 months? Most likely not. (if you're releasing code to a production system on a per fix basis... well that's another slashdot topic)
In lieu of automated testing, I do have a few suggestions for improving testing.
1) incorporate "impact analysis" as part of your design/code reviews. If someone is planning on touching function y in module x, your architect / tech lead / rest of developers should be able to identify what other areas are going to be affected. When it comes time to test, you know exactly what areas you need to really focus on and which areas can do with a spot check.
2) come up with a sensible schedule for bundling multiple code fixes into incremental releases. Every time you touch production, there's an inherent testing overhead. Bundle a multiple fixes together and that overhead is better distributed.
3) hire dedicated testers. Having someone full time on QA (or part time, split across multiple projects) does wonders. The good ones bring both a great deal of experience for finding "common errors" as well as a fresh perspective to the table to see things that the developers overlook because they're too deep in the trenches. Now of course, dedicated testers may not fit into the budget. Even if you can afford them, developers should always be on the hook for testing. Which brings me to my next point...
4) tell your developers that they better learn to test or fire them. sounds harsh, but testings part of the game. I don't want anyone who doesn't understand the value of testing -- and isn't willing to put in the effort to test -- on my team.
my 2 cents and then some...