IBM Sets SPECweb Record

the frogspotter writes "IBM has an article on their site talking about their new RS/6000 that can supposedly do 40,161 http ops./sec. as measured by SPECweb96. Jeez...that'd be 3469910400 http ops./day. " And it only took 12 (out of 24) processors (on an unreleased server) and a hacked kernel! It seems as if modifying the kernel for benchmarking is becoming quite a popular tactic these days...

Comments

SPECweb96 is a bit outdated

[ChrisRijk]

It's also not that good. (think of the recent NT vs Linux benchmarks where the webserver only servers static pages...)

However, the "new and improved (tm)" SPECweb99 is almost ready: See the SPECwww99 home page

Static Pages

[Wag the Dog]

Web servers are frequently asked to send users static Web pages.

With all these web benchmarks coming out using static web pages, I'm wondering if there are some statistics available somewhere that can either confirm or deny this.

I know /. is heavily dynamic (are there any pages that are static?). I know my homepages are netscape and yahoo are dynamic. All search engines serve dynamic content. Certainly web sites designed for E-Commerce, the market for IBM's web servers, are dynamic (searching for products, filling out order forms, etc).

For some reason I don't think the statement in IBM's release is totally accurate. Why not create a new benchmark or modify an existing one that had a mix of dynamic content that matched certain typical uses for web servers. A search engine server, a portal server, an e-commerce server, etc. This would give more accurate real-world results, IMHO.

It has always been like this

[QZS4]

Special tuning has always been a part of benchmarking. There are many compilers out there with special optimizations designed only to improve the performance of a specific benchmark (detection of certain matrix operations and other snippets of code which almost exclusively occur in benchmark programs). Many compilers, for example, recognize this piece of code:

x = sqrt(exp(log(x)/t1))

which gets transformed to

x = exp(log(x)/(2*t1))

thereby skipping the sqrt calculation. Since this code almost only occurs in the Whetstone benchmark, in the general case it will never get used. (Hennessy & Patterson, CA:AQA, chap. 1.8)

Some of the compiler optimizations out there will even generate faulty code if they are switched on when not compiling a specific benchmark program. Might be worth thinking about.

And don't think that only software can be optimized. Many of the processors out there have some special hardware hacks designed only to improve some benchmark program, and which almost never is used in "real world" programs.

Bottom line is, don't be surprised that they use some special tuning, it has always been this way and always will.

Re:It has always been like this

[wik]

That's an excellent example. Hennessy & Patterson is an excellent book for Architecture (and it spends a great deal of time discussing benchmarks as well as excellent, clear explainations of Architecture, of course).

For people who are interested in benchmarks, mistakes/ways people fudge numbers and how to measure, analyze and model system performance accurately, another gem to add to your library is: The Art of Computer Systems Performance Analysis by Raj Jain. It even includes a refresher on the stats course that you slept through (although a perfect knowledge of stats is not required to understand and profit from the information in this volume). It will give you an view into real performance analysis, modeling and measurement. Including all of the tricks that everyone seems to call "unfair".

In many cases, they are wrong or unjustified. But it is not a bad idea to use tweaked hardware to perform a benchmark, although publishing the results may be in poor taste. Many times, management makes signficant decisions on what hardware to sell and what changes need to be made to future hardware, based upon these benchmarks. For instance, if a 10-by machine performs more poorly as a 3-by machine because of bad filesystem code (which is not an unreasonable possibility if the operations in the benchmark are IO-bound or depends heavily on IO performance), the management might decide to put pressure on for cleaner fs code. The only time this problem code is isolated and fixed may be during the benchmarks with tweaked code.

SPECweb99 is the one to concentrate on

[Erik Corry]

Looks much better. Tests some of the things that are difficult about web serving, espeially the question of having thousands of low speed connections open at the same time. Optimising for this benchmark might even benefit real world web serving performance. Lets hope they also specify a dataset that is much larger than RAM so you can't just use a RAMDISK to get good scores. That may be a good idea if your dataset is small enough, but most web servers have much bigger data sets than RAM so it's not too realistic.

The tuning that has gone on after Mindcraft has probably benefited performance for someone, and that's great, but I doubt it has made any difference to real web servers anywhere.

Re:Free web benchmark?

[sparty]

The c't benchmark used a Linux-based benchmarking tool that is available from them.

Benchmarks...

[Signal 11]

should be left on the bench. The only use is to justify your expenditures to management. Personal experience is the best measure of what works, and what doesn't. Let the marketing departments drool over them. Personally, I'd file them in the circular.

Benchmarks only test a very narrow spectrum of performance, and is rarely representative of real world performance. Witness the mindcraft benchmarks. Specint is a "standard" now in benchmarking, hence it's to be expected that companies will be adding special optimizations that wouldn't otherwise be done.

--

Re:Benchmarks...

[Fyndo]

Benchmarks can be relativley valuable.

Or if not benchmarks, some form of standardized teating. I don't much care for "single number" benchmarks, that claim to sum it all up, but it's nice to see how different machines handle the same load. You do need a lot of them though. computer speeds are very complicated things, and you need to look at a lot of variables before you can conclude whether a system is "right" for you.

And is pretty impractical to try all the different machines in order to decide what works.

Benchmarks are a tool, not an answer.

Grand Pa IBM shows boys how

[mudpup]

Grand Pa IBM steps up and says "Boys this is how its done if you want a good benchmark, build your own hardware, write your own software, do the test in-house, and write your own press release."

IBM HTTP Server

[Zombie]

Sure /. servers up static content. All the bitmaps! There's a lot of static content out there!

By the way, IBM HTTP Server, A.K.A. "IBM HTTP Server Powered by Apache" is Apache, now IBM's preferred webserver. So any performance improvements IBM makes to "their" webserver will go back to the community.