Slashdot Mirror
Testing Pre-Production Servers Accurately?
An anonymous reader asks: "Having been granted a 90-day demo enclosure of new blade servers from a major vendor the question I find myself asking is: On a limited budget, how does one simulate 1000+ attached clients and the activity of those clients? We're a K-12 school district and our current servers don't keep up with all the roaming-profile abuse from our Windows workstations. Are there tools or tricks available to simulate load on Netware/Linux servers? The user groups around here usually answer this question with 'Get some workstations for a test lab!', there's got to be a less expensive option, right? Can we leverage our existing client populous to achieve our goal, without interrupting or changing the quality of service at the desktop, substantially?"
Talk to the Mercury test people and others about load generation. They will sell you some nice programs to unit and load test systems...
Store some MP3's and some porn, especially mpegs, on it and then connect it to the network. You don't even need to announce it but, within hours every workstation on your network will be hitting the servers and if it fails or is removed at a later date, no one will have the nerve to complain.
Just add a program to the login scripts that runs some tests against the server in the background.
Then when everyone logs in in the morning, it just overloads the thing...
And simulates the load spikes you see. I don't know a program or script that would do this, but I'm sure someone out there does.
JC
On Arrakis: early worm gets the bird. Magister mundi sum!
We're a K-12 school district ... Can we leverage our existing client populous [sic] to achieve our goal, without interrupting or changing the quality of service at the desktop, substantially?"
You're gonna hate the answer, but this will give you a better test than anything else. Plug in your test system and get a bunch of the kids to help you out on a weekend. Have them do logins, logouts, play games, surf, write and save papers, etc. on throwaway accounts that go to the test server.
Write out a test plan -- how many clients, how many local, how many remote, how many do you start with, what is the step size (e.g., start with 5 clients, then 10, then 15, then 20, then 30, etc.). Profile your existing systems so that you know what's really creating the load on them. Is it really the roaming profiles or is it web site caching or is it something else? Good luck with it.
--Paul
Perhaps its a reference to the fact that windows *copies* your profile to each machine when you log in and so if users have hundreds of megs in their docs etc.. then the network quickly becomes busy.
Bush and Blair ate my sig!
So why not just point each user's "My Documents" folder to a network share? This will keep the profile much smaller. I guess I just took it for granted that this was a standard practice. I've done it for years.
-paul
Pistol caliber is like religion: everyone has their favourite, and theirs is the only right choice.
So why not just point each user's "My Documents" folder to a network share?
Because most admins, > 60%, don't RTFM and therefore are unaware that the My Documents folder can be redirected through the use of Group Policy. Even more admins, > 80%, don't know that the IE cache directory should also be redirected to prevent multimeg profiles due to browser cache.
Of course, even those admins, that go to the trouble of doing these things and many more to minimized the profile size, are torpedoed by the users that save their downloads and documents to their desktops. I even saw an admin with a Windows 2000 service pack(>120MB) saved to his desktop while using roaming profiles.
The fact is that the roaming profile implementation is crap! Nothing should be copied over the network at login. Everything should be stored on a network drive and accessed directly from there. This is how Unix does it with remotely mounted Home directories. That way files don't cross the network if you don't access them. If you want to implement caching for slow links or disconnected access as Windows does, that's fine, but copying everything over the wire everytime someone logs in is just stupid.
This is very do-able, but there are a few gotchas. Your plight made be think about how I would do it if I were in your shoes...here's what I would do:
1. Come up with a (hopefully short) list of items that need to be observed under load. For example, authenticating users and allowing logins, serving up files).
2. For each testing item identified, find a repeatable, scriptable method of testing it. Perl will be your best friend here. For questions consult the great folks at perlmonks.com. You can find very nice modules that will allow you to create scripts that can authenticate to your test server, and report back a time. If you dump all your responses into a csv format, it will make the reporting of the data even easier.
3. Write a separate script to track system performance during the testing period. I couldn't actually tell from your question if the servers you are testing are windows or *nix , but whatever the os, you'll probably want a snapshot of overall CPU utilization and Run Queue size, swap use, disk access times, and memory consumption.
4. Be sure you are testing what you think you're testing! You're testing results will be invalid once you hit a limiting factor. So if you're network link gets overloaded by your test script, then your file transfer test is not a good one.
5. Design your testing scenarios carefully. It's probably best to have as short a test list as possible, and then make them as iterative as possible. For example, first test with having 1 users logon and transfer a file. Second test has 10 simultaneous users logon and transfer a file. Third test has 100 users involved, etc.
The key to all of this stuff, is you want tests that are repeatable, valid, and indicative of production uses. The nice thing about developing these test scripts is that you can use them to test hardware from several diferent vendors. If it seems daunting, just remember that it's better to spend a week of late nights developing the scripts - then to spend the next several months wondering if you're implementing hardware that will hold up to your production workload!