Slashdot Mirror
Website Load Testing Tools?
bLanark asks: "I'm in the process of converting one of my clients from IIS/CGI to Apache/mod_perl. I need a (free) web site stress/load test tool to prove that performance will be increased.
Using this page as my starting point, I can see that there are quite a few tools to investigate. Has anyone used any of these (or any others) and what are they like. I need HTTP GETS, form POSTing, and clever stuff like simulation of caching of images would be useful too, I guess." The previous story didn't get much of a response, but that was about a year ago, but the submittor has shared a fairly impressive list with us, impressions about any piece of software on it would be greatly appreciated.
Post it on the front page of slashdot...
Make even shorter URLs - 8LN.org
I would also think that it would have to be distributed (at least across several computers) to ensure that bandwidth and cpu/memory of the testing computer are not the limiting factors.
How hard can it be to write something like that? The hardest part would probably be parsing the log files. I would not be suprised if there are several testing tools that would do an adequate job.
"By upgrading from IIS to Apache/mod_perl, we were able to increase our load capacity from 300 millislashdots to a full 1.2 slashdots while cutting costs by nearly two-thirds...."
"Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
For the simpel testing that i've done, i've always used http://webtool.rte.microsoft.com/ . It's free, easy, quick (nothing like 100 hit/s to bring a site down) and does what you want. It has lots of fancy features only a point and a click away (yes, it can be scripted to), but I've newer had the need for it. I only use the stress tool the see *if* there is a problem, you then need a profiler to see where the problem is.
Checkout Apache Test page. Apache comes with Flood, a profile-driven web server tester.
In a previous job we needed to do some benchmarking and testing. The first pass was to script some stuff using wget. Once everything looked fine (we could really slam the server) we turned it loose on the public.
Oops. The site collapsed nearly instantly. The problem was that people with slow modem connections kept connections active for a couple orders of magnitude longer than happened on our internal network and the server ran out of resources.
Microsoft's free tool can simulate a mix of connection speeds and I believe you can find similar functionality in many of the web test tools you will find in a freshmeat.net search.
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
-j
I forget what 8 was for.
The problems are never what you expect. After hypertuning the web server you may find that your SQL backend is causing a bottleneck. Or that something in an external, offline, transaction- finishing process is flaky/unreliable/slow. Or that one particular area used by one particular class of users is making their perceived response horribly slow, while the rest of the world notices no problems at all. One thing that has helped on an aging site I work with is to track percentage of "completed" actions - for example, the user should click pages 1/2/3/4/5, in that order. If 80% make it to #3 and 15% are getting to #4, that may reveal bottlenecks that ps and ab and top won't show you.
I have had good luck with the neat little program called siege. It can stress a single URL, multiple URLs, follow links from a root URL (simulating an actual user), and have many multiple concurrent connections active. At the end, Siege can tell you all about the server performance, latency, etc.
I really like one of the other poster's idea about having a load tester read actual log files from Apache, then simulate real user activity. The only problem I can see with this method, is if you changed the layout of your site, all the program would get is a bunch of 404s. However, if one were so motivated, one could hack up such a thing relatively easy, I think. analog can parse Apache/httpd log files, could'nt be all that hard. Siege works well for me, though, so I'll stick with it.
Constitutional rights may be respected, repealed, or modified; but they must never be ignored.
though it seems to run out of resources at a few hundred concurrent users. You can find it at jakarta.apache.org. It's not very sophisticated, but it can certainly perform basic load testing.
I've also used (Mercury's) LoadRunner, a fairly expensive and complex beast; it comes with a built in proxy which allows you to capture a browsing session as the basis for a test scenario - this is a particularly useful feature. The scenarios can be scripted, and there's a bunch of random timers you can use to represent "real-world" behaviour.
I'd suggest creating 2 types of test scenarios : 1 type to represent your best guess at "real world" usage, i.e. representing users browsing, spending time reading a page, clicking all over the shop etc. - this should give you a good idea of how your application is going to perform.
The second type of test should provide you with metrics you can use when making improvements once you spot bottlenecks. They don't necessarily represent "real-world" activity, but allow you to measure whether code changes have made a difference. For example, you might have a test where all users log into the application simultaneously. In version 0.1 you can support 10 concurrent users; in version 0.5 100 concurrent users, and in version 1.0 250 users. These numbers don't represent "real" users - how likely are they to log in all at the same time ? It's a lot easier to manage your performance improvement work using these metrics.
It's all very well in practice, but it will never work in theory.
The Grinder (http://grinder.sourceforge.net) is an open source Java based load generating tool. It is multi-threaded, distributed, scriptable and has a central statistics reporting mechanism (console). It was used extensively by the authors of J2EE Performance Testing with BEA WebLogic Server and that book has excellent instructions and recommendations about its use.
I suggest you check it out.
Solaristrum: One who has spent way too long staring at the Sun
It does not take to long to write your own tool to stress test your webapp, espically if your already experienced in perl. I wrote one for the app design when mysql started to fall over on a weekly basis, it just could not handle the db size and transaction load. We ported the code base in a quick and dirty fashion so it would run on postgres, ms sql, and interbase -- ran my stress testing tool for 1 week and the only db which stood up was ms-sql.
Microsoft Web Application Stress Tool
I've used both JMeter and Siege and liked the results (fairly accurate, easy to use, scriptable). I've discussed some of the high cost solutions such as offerings from Mercury Interactive[mecuryinteractive.com] with developers I trust, and the concensus seems to be that if you use several of the testing tools avialable, grab some humans at the same time to check the URL, and take all of the results with a grain of salt, you'll be better off than spending beaucoup bucks on Mercury Interactive, or similar tools. (Note: I have never worked for Mercury Interactive, or used their tools) Also, I have tried the Micro$oft tool, I thought it sucked, but I'm an open source biased individual... :)
Good luck!
"...I'll need guns" --Chow Yun-Fat in 'Replacement Killers'
Like a previous poster, I recommend The Grinder. I looked all around for a load testing tool. People usually recommend the Microsoft tool, but I don't do Microsoft. I tried JMeter and a bunch of other tools, but they never feel quite right. Most tools fall in two categories: either completly unscriptable or so scriptable that using a general-purpose script language would be faster.
Among (many) other things, The Grinder has a built-in proxy that allows it to record a browsing session and play it back later. Basically, you start the proxy, set your browser to use this proxy, browse your website, and get back a log of your actions complete with timings and POST values. One other cool features is that it let you define your own datasources so you can fill POSTs and GETs with custom data. Last but not least the author of the tool will personally answer your questions, albeit slowly.
Nobox: Only simple products.
[joke]
Check out the WAST. It's free, and IMHO, better, and easier to use than ab (apache benchmark)... although, it's comparing tangerines to oranges.
http://webtool.rte.microsoft.com/
S
If you want your tool to be free as in beer, then I can't help you, but if you want to know what one of the best tools for the job is: mercury interactive's loadrunner. I work at a test department which has a sub department specialized solely in load and stress tests (and our testresults often show painfully enough that loadtests experts are very much needed) and they wont work with anything else. It is easy to build a small tool that will generate load, but if you also want to easy analyse the results, reuse test cases, integrate your graphs with other measurements (webserver memory, cpu, I/O etc)and want to breakdown the results in server time, network time etc then you'd be very happy to use a professional tool. Only caveat: it's expensive.
I intend to live forever, so far so good.
We tried Homer. It was OK, but we very quickly came to the ends of its capabilities. It doesn't simulate an actual user; rather, it acts as a proxy when you record your script, watching all your HTTP requests. It then plays that back, and you can parameterize some of the values.
What it doesn't let you do is make a hop from HTTP to HTTPS, nor does it simulate caching of images, nor many other things that an actual user would do. JavaScript is right out. For small, simple apps, it worked OK, but start getting complex and Home just can't handle it. D'OH!