Slashdot Mirror
Web Performance and QA Tools?
perf_monkey asks: "I'm part of a large Web Infrastructure Quality Assurance (QA) team at a large financial institution. Currently, we use Mercury Interactive's LoadRunner like a bunch of trained monkeys. We also use QA Load and SilkPerformer, but for smaller (non-J2EE) projects. As one of the technical folks, I've been trying to expand our horizons and our budget. I can no longer believe that large companies are willing to pay a QUARTER OF A MILLION dollars for the privilege of an additional 2000 Mercury VUsers. I'm looking at both commercial and open source alternatives. I've been tinkering with The Grinder and have had pleasing results. While not a full-blown QA tool, it is an excellent 'programmer's' load test tool. I was hoping that there are other tools like this and was hoping for the community's opinion. What web performance tools do you use and what do you think of them?"
I'm looking for the same kind of tool but for Windows applications. Any guidance is appreciated.
More than enough BS
Well I have to say I don't know anything about this type of application but typically what type of requirements do you have for a system like this.
How does lode runner test systems now?
for a major software company, and we have licenses for Mercury WinRunner, LoadRunner, QuickTest, Rational TestSuite, and even a product we wrote ourselves. This comes out to millions of dollars in software and maintanence licenses.
Now, I'm not saying the spending isn't wasteful. I think it is, but it's not correct to say that large companies are paying for expensive SQA software licenses anymore.
Vonal Declosion
Perl and Java are your friends...
I've used Mercury products for unit testing, full out scaling tests, monitoring, and defect tracking for over three years. The problem was the total cost of a Mercury load test bed. It gets expensive rather quickly unless you have planned in advance or do a lot of Mercury work. If you don't it makes more sense to hire a consultant that has an open license arrangement. That way you only pay for when you use it. YMMV.
So, I looked into some other alternatives and ended up going to unit test applications from the Apache group and home grown benchmarks to approximate load scenarios.
Eventually, I ended up looking into OpenSTA OpenSTA and was fortunate enough to work with some real experts that did a lot of load testing work with OpenSTA. There are some limitations to the OpenSTA *cough* Windows only *cough* but overall it is an excellent tool for driving load for a complex web application. Much like Mercury or any load testing tool the test is only as good as the planning and analysis you perform to isolate performance issues.
My most recent job called for load testing but in that time Mercury changed their license program again and we really wanted to use something that would run on Linux. So, OpenSTA was not really in the running. Also, we needed to simulate multiple IP addresses not just hundreds of virtual users from a single source address. (fine for simulations of a access from a corporate firewall). So, I had to find something that would work and that wouldn't blow our budget.
I checked back with a company called WebPerformance Inc. Now, I looked into a few years ago to see if they supported IP spoofing for virtual users and SSL. They had done SSL but the IP spoofing wasn't done.
As it turns out, they put this feature into their 2.6 release. We use it to run our large tests for burn in and acceptance for revisions to our network hardware that provides web interface. Like Mercury, you can really ramp up serious traffic. Couple this with basic network load generation and you can create a sound simulation of network throughput and application access from multiple network addresses with mulitple users and multiple business cases. Oh, and the fact that you can get a price list that is straight forward is very nice.
So, I'd say each is right for a certain type of test and a certain type of shop. For our shop, Mercury is cost prohibitive, OpenSTA lacks a key feature (IP Spoofing), and WebPerformance, Inc -- while commercial -- satifies our feature requirements near perfectly.
http://fudge.org
Have you looked at OpenSTA? Our development team in NH uses it and they enjoy it quite well.
"He wrested the world's whereabouts from the heavens And locked the secret in a pocketwatch." - Dava Sobel
The fact that it is windows only is not great. Otherwise, I agree. It is the tool I use in windows environments.
I generally don't appreciate proprietary software, but I've actually liked Web Performance Inc's tool.
http://www.webperformanceinc.com/
They've been quick to respond to our concerns, and their price was significantly cheaper than any of their competitors.
While hardly comprehensive test tools, there are some nice load generation tools that have some pretty remarkably features for their price (free). Check out WebBench (www.webbench.com) which does a nice job of harnessing a ton of machines to generate as much load as you can dream up. Then there is WAST (Web Application Stress Tool) from Microsoft. WAST has some nice features as far as scriptability goes, you just need to be warm and fuzzy with WSH to do it. On the Unix side you can use things like http_load, however it doesn't offer nearly the tweakability that WebBench offers.
On the commercial side, check out the Spirent Caw box. I've used that a far amount and it does an excellent job at capturing data and generating a lot of load from a single system. Great for when rack space is at a premium and you want an appliance-like solution.
I was a VP of IT for a startup, developing a large content web site. Initially the site was hosted on 2 Pentium based IBM servers, running Linux. The server were no where near full capacity, but we wanted to check how much load these machines could take before requiring additional servers.
Solution such as LoadRunner were too expensive, but even renting LoadRunner users for a few days was too much. We figured that for the money we saved by not checking the load we could buy and host two additional servers for two years.
Think how much serving capacity you could for a QUARTER OF A MILLION dollars.
Even a better idea: put the $250K in the bank, and buy a new server every month on the interest alone!
I wrote a cywin script using curl to access the server. I signed on 100 users and tracked each response. I then did a few different paths down an application chain and then did a few major image loads. I could get response times for each screen and using 10 PC's around the place running 10 instances I had a simple load test. (25 bash scripts appeared to blow a P4 windows, but cruised on a PII350 pure Linux)
OK this is not comprehensive and slick but it worked sufficiently to prove we could handle 100 concurrent users with concurrent image retreives under 5 seconds which was our required benchmark.
The interesting thing was that a few users, say 5, performed worse than 50 users. Some serious cache management in there that gets in the road of small sites.
I used to be the development manager for a content-based website, and we had significant performance issues.
We had a limited budget, and buying even tens of thousands of dollars worth of licenses was - in our view - a bad use of that budget. Instead, we concentrated on building a test infrastructure with known performance characteristics, and extrapolated (sorry, I meant "guessed") that this performance profile would scale by a factor of X on the live infrastructure. By using a relatively small and flexible test infrastructure, we could rapidly cycle through load test scenarios and find whether any particular change moved us in the right direction. We determined a set of key performance criteria for the test infrastructure, and found out quickly what the overall trend was because we could run several fullscale loadtest scenarios per day.
We used Loadrunner - but only using a limited number of concurrent users (100, I seem to recall), and since then I have used apache.org's JMeter product with similar levels of success.
As far as I know, if you absolutely have to be able to simulate thousands of concurrent users, the Mercury/Rational style commercial tools are the only ones available. I guess there are circumstances where this is appropriate - but the sheer effort of organising and running a test on that scale is a huge pain in the backside. If you can find a way to scale down your testing requirements - by running on lower spec infrastructure with optional throttling on key components - you reduce your licensing costs and improve the flexibility of your test process - if you can run several load tests a day without dimming the lights throughout the neighbourhood, load testing becomes "just another test", rather than a big deal.
It's all very well in practice, but it will never work in theory.
Just post us a link whenever you feel like a good slashdotting :).
Here is a website that maintains a list of open source testing tools. Some of the tools mentioned in some of the comments here are in that list.
Interesting timing on this question. Just recently I attended the Pacific Northwest Software Quality Conference and heard a talk by Bret Pettichord on just this very subject. His presentation, Home Brew Test Automation covered this subject with some terrific lessions.
Here's a trick I stole from agile methods like Extreme Programming: build the test suite up at the same time you write the main code.
The main reason testing existing applications is so hard is that they aren't designed to be easily tested, so you have to jump through a lot of hoops to do proper testing. However, if the developers don't get to check anything off until their components are covered by unit tests and their features are covered by acceptance tests, that gives the developers the incentive to make the features easy to test.
On a recent Java web project, we used HttpUnit, a handy framework that simulates a web browser, including even a limited JavaScript processor. As we developed features, we built end-to-end tests that walked the web site and did all the things that users were supposed to do (or not supposed to do, like visiting admin-only pages). Sometimes it was the same person who built the feature, and sometimes it was somebody else on the team.
This doesn't mean that QA people are unneeded; a good QA person will think of many more edge cases than your average developer will, and they'll also think a lot more about keeping the site consistent as a whole. Instead, developers get a lot more respect for QA, as they discover that good testing is hard to do right.
Anyhow, if you've built feature tests as you built the main code, load tests are easy; you just run portions of the feature test suite from multiple boxes.
My big tip with this approach is to make sure to treat the test code with the same respect as the main code. Unless you refactor aggressively, it's easy to let it get messy. Once it's messy, people stop maintaining it. And once you stop maintaining it, developers can easily break features without anybody noticing.
Get your website into a story on Slashdot. A case beer and an XL Pizza should be enough bribing for a real geek. Get out your stopwatch, measure the seconds before you smell something burning.
Repeat and tweak as necessary.
I just wanted to say thanks for everyone's suggestions. I consider myself a pretty experienced perf expert and I learned about some good OS tools. I'd like to give a bit more of a background of my setup.
My PTE (Performance Testing Environment) is made up of an AWEFUL lot of REALLY expensive Sun boxes. That is where I use Mercury, and will continue to use Mercury. I have another setup for the DIT, SIT, and DEV environments. The load generators that I use in the aforementioned environments are 5 beefy x86 Red Hat boxes. I have one beefy windows box to use as a controller. I have a strong predisposition to using the new Red Hat Enterprise 3.0 Linux because it has been optimized for a buttload of threads. It is on these boxes that I will be using all sorts of other QA tools.
I also want to say that I'm not Mercury Bashing, I'm just being an economic realist. I've worked with Mercury to sovle a lot of bugs in their software and know their hooks into OpenSSL almost as well as their developers in Israel (who I've spoken to and are pretty helpful guys). I've actually spoken to their CSO before and I have to say that the service I received was pretty damn good.
Thanks again Slashdot!