IBM Sets SPECweb Record

the frogspotter writes "IBM has an article on their site talking about their new RS/6000 that can supposedly do 40,161 http ops./sec. as measured by SPECweb96. Jeez...that'd be 3469910400 http ops./day. " And it only took 12 (out of 24) processors (on an unreleased server) and a hacked kernel! It seems as if modifying the kernel for benchmarking is becoming quite a popular tactic these days...

Comments

It has always been like this

[QZS4]

Special tuning has always been a part of benchmarking. There are many compilers out there with special optimizations designed only to improve the performance of a specific benchmark (detection of certain matrix operations and other snippets of code which almost exclusively occur in benchmark programs). Many compilers, for example, recognize this piece of code:

x = sqrt(exp(log(x)/t1))

which gets transformed to

x = exp(log(x)/(2*t1))

thereby skipping the sqrt calculation. Since this code almost only occurs in the Whetstone benchmark, in the general case it will never get used. (Hennessy & Patterson, CA:AQA, chap. 1.8)

Some of the compiler optimizations out there will even generate faulty code if they are switched on when not compiling a specific benchmark program. Might be worth thinking about.

And don't think that only software can be optimized. Many of the processors out there have some special hardware hacks designed only to improve some benchmark program, and which almost never is used in "real world" programs.

Bottom line is, don't be surprised that they use some special tuning, it has always been this way and always will.

Benchmarks...

[Signal 11]

should be left on the bench. The only use is to justify your expenditures to management. Personal experience is the best measure of what works, and what doesn't. Let the marketing departments drool over them. Personally, I'd file them in the circular.

Benchmarks only test a very narrow spectrum of performance, and is rarely representative of real world performance. Witness the mindcraft benchmarks. Specint is a "standard" now in benchmarking, hence it's to be expected that companies will be adding special optimizations that wouldn't otherwise be done.

--

IBM HTTP Server

[Zombie]

Sure /. servers up static content. All the bitmaps! There's a lot of static content out there!

By the way, IBM HTTP Server, A.K.A. "IBM HTTP Server Powered by Apache" is Apache, now IBM's preferred webserver. So any performance improvements IBM makes to "their" webserver will go back to the community.