Slashdot Mirror
Stress and Volume Testing - Your Experiences?
Tarohann asks: "I work for a large software services company and I'm planning to write an article on Enterprise application Stress and Volume Testing (SVT) based on a couple of experiences I've had.
For the most part I feel that most Enterprise SVT initiatives performed by companies at the enterprise level are flawed as they are done on ideal/non-production environments. I was wondering if anybody here has had any good/bad experiences or pointers (websites, books) I can refer to on this subject. Full credits will be given. Thanks, Slashdot!"
I was wondering if anybody here has had any good/bad experiences or pointers (websites, books) I can refer to on this subject.
Wonder no more. There are people here who have had some good/bad experiences or pointers that you can refer to on this subject.
-Adam
Gather 20 of your closest friends and relatives, give them each a different OS cd, and a different used computer (some laptops, some desktops, different brands, etc.) and have them each install the OS to their liking. After that, let them swap and repeat. For better results, image those systems (with ghost, etc.), and install the images on other systems with different hardware, and do whatever it takes to get them going on the new system. Finally, take a couple of the systems, have someone who didn't work on it go over to it and remove the crap that the other person put on it, and install it to their taste.
After this, you will finally have an environment that reflects what really happens in most computer networks. Image again, and save one image per machine, and you should be all set. (Unless you want to swap some hardware around, like NICs, or modems)
I work for a company that writes systems management software for windows and *nix systems, shameless plug, and each of the QA machines have a different OS, and a mix of languages for color. We've got solaris boxen, macs, linux in many different flavors, and every x86 Microsoft OS released since September '95 (except Windows ME, 'cause no one really counts that as an OS ;) )
We've got used computers from many different vendors, servers, desktops, etc., and let me tell you that we encounter many more issues in captivity because of it...
If you can afford it, get help from Sun.
The Sun application server software team
has people with very deep backgrounds
and the right experience and equipment.
I was truly impressed working with them.
They do massive testing projects
and have all the necessary contacts
with other enterprise app companies.
Cheers, Joel
Can we do your kid's homework, too?
My deviantArt site
The best thing I can say to you is that if you want to 'sell' the test results to your management, etc., you are going to have to do both database/application server capacity and load testing and user-side 'performance' testing if you have a user-side to your application. For some reason, PHBs on up like to see the little application animating its brains out. I'd suggest Mercury Interactive's WinRunner and LoadRunner, but I was too lazy to RTFQ and beside that, I've been coding in WinRunner all day, and I now need beer to soothe my nerves.
I like how P&CP (performance and capacity planning) becomes S&VT when you are talking "Enterprise" applications. It's funny how far people will go to sell a new approach, or to pad their resume with more buzzwords that nobody understands.
Heute die Welt, morgen das Sonnensystem!
My boss used to give me stress, and a very high volume of it. My experience was I didn't like it one bit, so I found another job.
i've had trouble testing my volume, and it's stressful.
Last time I was involved in that kind of testing, we found out that our code worked as designed (yay!). Unfortunately, we also found out that the application server engine we'd written it for had some critical bugs buried very deeply, that had never shown up because no one had ever tried to do anything quite that stressful with it. :(
Oh well.
Sometimes ideal environments offer higher throughput and stress on a system. Consider a webapplication on a busy network with one interface. Very typical and much lower stress than you might expect. Consider the stress-volume environement: You could have 3 nicks on a quiet switched segemnt and HAMMER the server.
Obviously, you want to profile your testing environment while testing to find a bottle neck and work it into your analysis. During one testing experience we found that the IDE drive started working very very hard and limited our throughput!
Just some thoughts. Home you find it interesting.
Sam
That will teach me to proof read my posts more closely.
Sam
You might look into using something like the Pareto distribution or Zipf's Law for estimating the probability of low probability, high-magnitude loads on the system. A little math like this in conjunction with a limited sample size of observed input load levels will help you guesstimate the 99.9999...% level of loading.
Two wrongs don't make a right, but three lefts do.
Jakarta has a couple of articles related to stress testing java web applications. There are also articles on IBM about using jmeter and other load testing tools. Personally, I run a variety of tests to measure the performance under average, peak, n x peak and /-overload. Then I write up a report explaining what it means in the context of how the business plans to grow. Raw number by themselves may get you into trouble. Explaining what those numbers mean is just as important as running the tests.
Would that be pints or quarts or liters or fifths or ... ?
Infuriate left and right
I'd suggest asking your question at Sticky Minds (forums require registration - they haven't spammed me yat after a lot of years), or QA Forums.
The best is the enemy of the good
I'd strongly recommend a section on this company. They have a number of load generation centres distributed around the world, and can hit your live site with a huge load (practically a denial of service)
Their engineers are very experienced with load testing and their product Load Runner is the defacto standard for stress testing.
I dont' work for them, but I have worked with them and been very impressed.
Asmo
See OpenSTA, an Open Source load testing application for web applications, with some more links in the WebLink section on portal.opensta.org