Slashdot Mirror
Testing Pre-Production Servers Accurately?
An anonymous reader asks: "Having been granted a 90-day demo enclosure of new blade servers from a major vendor the question I find myself asking is: On a limited budget, how does one simulate 1000+ attached clients and the activity of those clients? We're a K-12 school district and our current servers don't keep up with all the roaming-profile abuse from our Windows workstations. Are there tools or tricks available to simulate load on Netware/Linux servers? The user groups around here usually answer this question with 'Get some workstations for a test lab!', there's got to be a less expensive option, right? Can we leverage our existing client populous to achieve our goal, without interrupting or changing the quality of service at the desktop, substantially?"
Its extremely difficult to create useful benchmarks of high loads. Leave it to professionals. Find or buy an evaluation of the type of system that you are interested in. End-user reviews of systems that they have purchased would be great, but too many manufacturers will try to quash negative comments or inject positive comments, so its hard to find unbiased advice.
Intron: the portion of DNA which expresses nothing useful.
Talk to the Mercury test people and others about load generation. They will sell you some nice programs to unit and load test systems...
Setup a site and see how the server can handle a slashdotting.
Store some MP3's and some porn, especially mpegs, on it and then connect it to the network. You don't even need to announce it but, within hours every workstation on your network will be hitting the servers and if it fails or is removed at a later date, no one will have the nerve to complain.
Just add a program to the login scripts that runs some tests against the server in the background.
Then when everyone logs in in the morning, it just overloads the thing...
And simulates the load spikes you see. I don't know a program or script that would do this, but I'm sure someone out there does.
JC
On Arrakis: early worm gets the bird. Magister mundi sum!
Just put them on the internet and publish the address on Slashdot. That will be a real stress-test of those machines.
We're a K-12 school district ... Can we leverage our existing client populous [sic] to achieve our goal, without interrupting or changing the quality of service at the desktop, substantially?"
You're gonna hate the answer, but this will give you a better test than anything else. Plug in your test system and get a bunch of the kids to help you out on a weekend. Have them do logins, logouts, play games, surf, write and save papers, etc. on throwaway accounts that go to the test server.
Write out a test plan -- how many clients, how many local, how many remote, how many do you start with, what is the step size (e.g., start with 5 clients, then 10, then 15, then 20, then 30, etc.). Profile your existing systems so that you know what's really creating the load on them. Is it really the roaming profiles or is it web site caching or is it something else? Good luck with it.
--Paul
What's "roaming-profile abuse?"
Well just to start i'll tell a bit about my self, Im a grade 12 student at Bishop O Byrne high school.
If you have 90 days get the system setup to do everything on the network as fast as you can (you only have 90 days) set it up as if you where going to replace your current computers and then do just that, put it where it is suppose to go if you where to buy one. See the load it gets and log everything.
Your not part of a multi billion dollar company teachers and students can make it without the internet (its true!) tell them its a network upgrade.
In my school district there is quite a few day a year when the network and computers don't work, guess what we survived!
I can nearly guarantee that whatever company is providing you the blade servers has already done a ton of quantitative testing [or had someone else do it for them] for the sort of thing you're looking for. Just ask them for the data.
Talk to the AP computer science teachers about getting their students to develop a load generator solution. It should make an interesting extra credit assignment for them.
If nothing else, put it on the network and invite them to use it as much as possible.
It's good to use your head, but not as a battering ram.
-paul
Pistol caliber is like religion: everyone has their favourite, and theirs is the only right choice.
While not exactly the same, the fifth post in this thread is similar enough to yours that yours qualifies as -1 Redundant.
-peter
-mary
I really could go on and on with problems they cause. They have but one sole purpose which is to save user settings. While this in itself is a very strong reason, all of the negative side-effects of using them outweighs their benefit. Over time you will find yourself cutting more and more out of the user profiles in order to fix the problems they cause. For example, to reduce the size of the profiles, you can have certain subdirectories ignored. This works great, but by the time you eliminate the problem directories to reduce the number of files in the profile, you realize that much of the value of roaming profiles is being eliminated just to make them work decently. In the end it's just better to avoid them.
This even applies to situations where each user typically uses only a single PC, an environment which is very different from a typical school. In those environments, what purpose does a roaming profile serve? If the users don't roam, you might as well just use local profiles stored on the workstations themselves.
This is very do-able, but there are a few gotchas. Your plight made be think about how I would do it if I were in your shoes...here's what I would do:
1. Come up with a (hopefully short) list of items that need to be observed under load. For example, authenticating users and allowing logins, serving up files).
2. For each testing item identified, find a repeatable, scriptable method of testing it. Perl will be your best friend here. For questions consult the great folks at perlmonks.com. You can find very nice modules that will allow you to create scripts that can authenticate to your test server, and report back a time. If you dump all your responses into a csv format, it will make the reporting of the data even easier.
3. Write a separate script to track system performance during the testing period. I couldn't actually tell from your question if the servers you are testing are windows or *nix , but whatever the os, you'll probably want a snapshot of overall CPU utilization and Run Queue size, swap use, disk access times, and memory consumption.
4. Be sure you are testing what you think you're testing! You're testing results will be invalid once you hit a limiting factor. So if you're network link gets overloaded by your test script, then your file transfer test is not a good one.
5. Design your testing scenarios carefully. It's probably best to have as short a test list as possible, and then make them as iterative as possible. For example, first test with having 1 users logon and transfer a file. Second test has 10 simultaneous users logon and transfer a file. Third test has 100 users involved, etc.
The key to all of this stuff, is you want tests that are repeatable, valid, and indicative of production uses. The nice thing about developing these test scripts is that you can use them to test hardware from several diferent vendors. If it seems daunting, just remember that it's better to spend a week of late nights developing the scripts - then to spend the next several months wondering if you're implementing hardware that will hold up to your production workload!
Come on.
You buy blades to serve 1000 users on Linux?
For most purposes like openldap logins, simple database queries, pam logins, mail serving, simple firewall etc, you can get away with a simple Athlon64 server for upto 10,000 users. Remote profiles in windows is REAAALY heavy, think of the delay when youre logging in on someone else's workstation... Linux's authentication isnt that heavy at all.
Just get an xSeries 206 ($500) and be done with it.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
There is lots of test data out there. You could use the serverbench test suite from PC Magazine - it's free. It uses a bunch of client PCs to generate huge amounts of network file serving load (read/write/bigfile/small files).
Do you have enough network bandwidth? 1000 computers will easily saturate fast ethernet connections. You need gigabit ethernet or better.
And frankly, a bunch of blade servers isn't a good way to solve a file serving problem. You want either a bunch of NAS boxes (network attach storage) or regular file servers with big scsi raid arrays.
It may be overkill, or it may be playing The Budget Game. Sure, you could take the cheap and simple route and just slap a $500 server in there, but then poof goes the money you didn't spend. "You didn't spend it," the bureacrats will say, "so obviously you don't need it in the budget."
And then there goes your IT budget when you need to do something important like roll out a new OS or launch a network application or buy Fembot parts for the computer programming students to use in the science fair.
you could be considered a 'genius' for saving the it dept oodles of $, and get a raise..and get to play with linux everyday.. and..and.. :)
I will gladly loose all of life's battles.. in order to win the war..
after all, a 'good' win geek will become a uni geek someday ;)
I will gladly loose all of life's battles.. in order to win the war..