Web Performance and QA Tools?

perf_monkey asks: "I'm part of a large Web Infrastructure Quality Assurance (QA) team at a large financial institution. Currently, we use Mercury Interactive's LoadRunner like a bunch of trained monkeys. We also use QA Load and SilkPerformer, but for smaller (non-J2EE) projects. As one of the technical folks, I've been trying to expand our horizons and our budget. I can no longer believe that large companies are willing to pay a QUARTER OF A MILLION dollars for the privilege of an additional 2000 Mercury VUsers. I'm looking at both commercial and open source alternatives. I've been tinkering with The Grinder and have had pleasing results. While not a full-blown QA tool, it is an excellent 'programmer's' load test tool. I was hoping that there are other tools like this and was hoping for the community's opinion. What web performance tools do you use and what do you think of them?"

Comments

Web Performance Trainer vs. OpenSTA vs. Mercury

[Fudge.Org]

Disclaimer: The company I work for is a very happy WebPerformance Inc. customer.

I've used Mercury products for unit testing, full out scaling tests, monitoring, and defect tracking for over three years. The problem was the total cost of a Mercury load test bed. It gets expensive rather quickly unless you have planned in advance or do a lot of Mercury work. If you don't it makes more sense to hire a consultant that has an open license arrangement. That way you only pay for when you use it. YMMV.

So, I looked into some other alternatives and ended up going to unit test applications from the Apache group and home grown benchmarks to approximate load scenarios.

Eventually, I ended up looking into OpenSTA OpenSTA and was fortunate enough to work with some real experts that did a lot of load testing work with OpenSTA. There are some limitations to the OpenSTA *cough* Windows only *cough* but overall it is an excellent tool for driving load for a complex web application. Much like Mercury or any load testing tool the test is only as good as the planning and analysis you perform to isolate performance issues.

My most recent job called for load testing but in that time Mercury changed their license program again and we really wanted to use something that would run on Linux. So, OpenSTA was not really in the running. Also, we needed to simulate multiple IP addresses not just hundreds of virtual users from a single source address. (fine for simulations of a access from a corporate firewall). So, I had to find something that would work and that wouldn't blow our budget.

I checked back with a company called WebPerformance Inc. Now, I looked into a few years ago to see if they supported IP spoofing for virtual users and SSL. They had done SSL but the IP spoofing wasn't done.

As it turns out, they put this feature into their 2.6 release. We use it to run our large tests for burn in and acceptance for revisions to our network hardware that provides web interface. Like Mercury, you can really ramp up serious traffic. Couple this with basic network load generation and you can create a sound simulation of network throughput and application access from multiple network addresses with mulitple users and multiple business cases. Oh, and the fact that you can get a price list that is straight forward is very nice.

So, I'd say each is right for a certain type of test and a certain type of shop. For our shop, Mercury is cost prohibitive, OpenSTA lacks a key feature (IP Spoofing), and WebPerformance, Inc -- while commercial -- satifies our feature requirements near perfectly.