Slashdot Mirror
IBM Sets SPECweb Record
the frogspotter writes "IBM has an article on their site talking about their new RS/6000 that can supposedly do 40,161 http ops./sec. as measured by SPECweb96. Jeez...that'd be 3469910400 http ops./day. "
And it only took 12 (out of 24) processors (on an unreleased server) and a hacked kernel! It seems as if
modifying the kernel for benchmarking is becoming quite a popular tactic these days...
Good point, why in the hell is hacking the kernel and getting direct support from the engineers who made the software considered fair in benchmarking??? When I first saw the Mindcraft results, I about laughed my ass off after reading that MS had direct help in the tuning of the server.
When I can get personal hacked/tuned kernel from MS, and have their engineers come out and work on my servers, I might consider running NT on the server side.
Come on IBM. Take it out of the box, do some Real Guy Tunning(TM) and then test it over serveral weeks. Then get back to us.
Linux O Muerte!
However, the "new and improved (tm)" SPECweb99 is almost ready: See the SPECwww99 home page
Web servers are frequently asked to send users static Web pages.
/. is heavily dynamic (are there any pages that are static?). I know my homepages are netscape and yahoo are dynamic. All search engines serve dynamic content. Certainly web sites designed for E-Commerce, the market for IBM's web servers, are dynamic (searching for products, filling out order forms, etc).
With all these web benchmarks coming out using static web pages, I'm wondering if there are some statistics available somewhere that can either confirm or deny this.
I know
For some reason I don't think the statement in IBM's release is totally accurate. Why not create a new benchmark or modify an existing one that had a mix of dynamic content that matched certain typical uses for web servers. A search engine server, a portal server, an e-commerce server, etc. This would give more accurate real-world results, IMHO.
Are there any projects out there aimed at modifying Linux so that it is THE leading web server platform? Obviously you can't compete with IBM and 24 processors, but what about a distro that is aimed at serving web pages, static and dynamic, at such a high level that it can't be matched in price/performance?
I know Zeus is fscking fast, but what about an operating system that is designed from the ground up to be a web server? All OS bottle-necks removed.
Is this a crazy idea? Is it possible? Seems to me like Cobalt's systems are a lame attempt at doing this (no optimizations, just easy setup).
Stew
Vive l'AIX!! The single most weirdest, scariest flavour of Unix! Has anyone seen those 21" RS/6000 monitors that are perfect squares? Even weirder.
Special tuning has always been a part of benchmarking. There are many compilers out there with special optimizations designed only to improve the performance of a specific benchmark (detection of certain matrix operations and other snippets of code which almost exclusively occur in benchmark programs). Many compilers, for example, recognize this piece of code:
which gets transformed to thereby skipping the sqrt calculation. Since this code almost only occurs in the Whetstone benchmark, in the general case it will never get used. (Hennessy & Patterson, CA:AQA, chap. 1.8)Some of the compiler optimizations out there will even generate faulty code if they are switched on when not compiling a specific benchmark program. Might be worth thinking about.
And don't think that only software can be optimized. Many of the processors out there have some special hardware hacks designed only to improve some benchmark program, and which almost never is used in "real world" programs.
Bottom line is, don't be surprised that they use some special tuning, it has always been this way and always will.
Is there any free web benchmarking tool?
Webbench from ZDNet only runs on Windows (".EXE for all platforms", right.)
SPECWeb96 costs $800!
Any suggestions?
--
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
The tuning that has gone on after Mindcraft has probably benefited performance for someone, and that's great, but I doubt it has made any difference to real web servers anywhere.
Is there any hope that vendors will stop citing tests on a platform that is never going to be commercially available unless the results are specifically labeled as R&D?
I have not gone looking yet at the SpecWeb99, but I think some of the TPC benchmarks now require the platform to be commercially available within six months.
At least IBM was honest enough to tell us that there was special tuning and new hardware involved. That gives me some hope.
Even a site that serves only dynamic pages still serves static files. Even if nothing else, the images are all static. So if you've got a busy site, it's important to have a server that's fast at serving static files. Thus, you might run an image server on a different server than your pages and dynamic content.
-Imperator
Gates' Law: Every 18 months, the speed of software halves.
Why is everybody so negative ? :)
Let them do what they want, if they can get Linux above microshit, I'm happy
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Belgium HyperBanner
http://belgium.hyperbanner.net
Linux hosting for $2.50/mo
Isn't it the idea of Linux that the source
of Linux is freely available, and that it's
purpose is to be adapted by others ?!
I say, IBM did a fine job. If they release
that pathches it would even be better...
This might lead to new and faster implementations...
should be left on the bench. The only use is to justify your expenditures to management. Personal experience is the best measure of what works, and what doesn't. Let the marketing departments drool over them. Personally, I'd file them in the circular.
Benchmarks only test a very narrow spectrum of performance, and is rarely representative of real world performance. Witness the mindcraft benchmarks. Specint is a "standard" now in benchmarking, hence it's to be expected that companies will be adding special optimizations that wouldn't otherwise be done.
--
Instead of all this stuff with httpd in software, why not just build it into the hardware? That would fry the bacon, so to speak. Imagine a ATM card with a 1 gb RAM buffer caching your web pages ready to saturate any goddam bandwidth that you can find.
Seems to me to be the obvious way to do it. They you can apply the general purpose CPU stuff to creating dynamic content where needed.
and what is /. server configuration ?
The world belongs to those who get up early. - I'm far from being the king of Earth then
Posted by Ungrounded Lightning Rod:
of the previous poster's comment. It DOESN'T
seem to ME that Cobalt was a "lame attempt".
Surely IBM can write a parellel version of httpd and blow away any numbers out there? When are they going to drop these silly standalones and market the machines with some real power?
Now if they could port PSSP to Linux, that would rule!
newton62 (56617) Karma: Bad
Posted by 2B||!2B:
Good answer!
Not that targetting benchmarks is necessarily bad. What we need is to keep making up new benchmarks in a way which deliberately targets flaws (flaws that actually make a difference, that is) in existing systems. It's much easier to make improvements if you have a simple way to measure the results on any changes.
To be truly productive, we would have to target flaws in both Linux and in NT, to give a real incentive for both sides to improve. As much as I dislike NT, I would still love to come up with ways to make them improve it. If you can't beat it, fix it.
Grand Pa IBM steps up and says "Boys this is how its done if you want a good benchmark, build your own hardware, write your own software, do the test in-house, and write your own press release."
Who owns your data?
No doubt! AIX rules. JFS, LVM, SMIT, ODM, 64bit, solid as a Mac truck. I used to work in AIX defect support in Austin, was my 1st UNIX to play with. Jaded form the get go? Yes, but I've never seen a better UNIX in my years of playing with Solaris, DGUX, Linux, FreeBSD, SCO....
See 'ya,
Jim
And what's its keyrate? :)
-Chris
By the way, IBM HTTP Server, A.K.A. "IBM HTTP Server Powered by Apache" is Apache, now IBM's preferred webserver. So any performance improvements IBM makes to "their" webserver will go back to the community.
I've worked with AIX off and on since 1991. I remember the 3.1.5 debacle. Believe me, AIX can't point fingers at anyone without having three pointing back at itself. I remember the joys of ODM corruption, I remember the text config files (there for backward compatibility) that did not reflect the true (odm) config of the system, I remember when they took away the ability for an admin to edit the ODM database directly so that you could fix the screwups caused by some of the install and system config utilities, I remember creeping JFS corruption, LVM management utils that would allow you to shoot yourself in the foot...
Geez, AIX - why did you get me started? The only thing that would make me rant more are my years with the ultimate "Crappix" - SCO EunuchsWare. I will *never* look back. The only thing that could compete with it on the Crappix scale was HP/UX 8.x (and you thought 9.x was bad... boy, have I got some stories for you).
Christ, somebody get me to alt.sysadmin.recovery before I go nuts and piss on the mains...
-- ultra1
Do they mean disk cache or RAM cache? If it's ram that's a hell of a lot of cache.
But the idea of storing frequently accessed static pages in a cache contolled by the kernel sounds like a cool idea. Wonder if there is any plan to implement this in the Linux kernel? Maybe as a module?
Steve
There is a in-kernel webserver (although not quite mature)!
It is called kHTTPd
http://www.fenrus.demon.nl
OSes are all about compromise, there are always ways to make things a little faster if you're willing to compromise reliabilty or stability. (I think MacOSX got bit by that bug with there little web bench problem not so long ago)
How many record setting benchmark configurations are good for real world usage? Do we really want development effort going in to that? I'd rather have a benchmark loser that is rock solid all the time. There are certainly cases where it makes sense though, I think mindcraft revealed some very real problems that need to be addressed but I don't think it should be our norm.
This is my signature. There are many signatures like it but this one is mine..
AIX is a royal pain in the arse. My current position is admining a whole bunch of RS6000s and while it's rock solid and quick, it's _DIFFERENT_ to everything else.
My last job was working on Data General AViiON systems running DG/UX, they are a real pleasure to use and administer, leaves AIX and Solaris in the dust.
It's a pity Data General's management now have their tongue stuck so far Microsoft's arse.
Deleted
DG/UX can do this with ease, but is a damn site smoother with it.
:(
I work with AIX, but wish I worked with DG/UX.
Deleted
While the Exokernel stuff is interesting, I think that VIA-like optimizations (http://www.viarch.org as previously noted on /., owned by Compaq/Intel/MS) are a better way to go. The idea was originally from a thesis paper and implemented for Unix. (Author not handy due to a recently deceased HD.)
VIA stands for Virtual Interface Architecture. The idea, generally, is to build device drivers that can directly interact with processes or threads with little or no interaction with the kernel. The idea is generally that the driver presents a virtual device interface to each process. The viarch.org guys appear to be concentrating on applying the idea to SAN I/O, but I think that the network/OS/Webserver (SMB, etc. also) is even more interesting given the smaller payloads and higher overhead.
What I would like is to have a process or group of threads that would interface to the NIC and receive a stream of packets and transmit a stream of packets. The exchange could be completely async with large queues and no context switches or other OS interaction (for network traffic). The service processes would contain an integral TCP/IP stack and HTTP server. The device driver would send packets for a particular IP address only to the VIA processes while handling other IP addresses normally.
This would allow optimizations just like those for the Exokernel. People could experiment with various zero copy schemes and other TCP/IP stack optimizations without affecting the operation of the system they were running on.
We need to do this now!
Is anyone working on something like this? Email me! I might even be able to provide some funding.
sdw
Stephen D. Williams
Moving static requests back up into the kernel is a neat trick, to be sure. But I'd rather have web serving where it belongs: user space. The idea of being able to serve a page to every user on the planet five times over on a given day from my web server sounds nice on paper, but not very useful in real life.
Having a bit of up close and personal AIX experience, I feel they should work on making their kernel a bit more stable, anyway. This is simply posturing for a benchmark, probably not real useful in a real-world scenario, and to be frank, I think IBM knows better.
Do you have ESP?
All this complaining that the OS & software used to set this record was tweaked!!
I don't see the problem. These people had access to source for AIX and they used it! Just the sort of thing that anyone with the skills can do to Open Source software.
while dynamic vs. static is an important distinction, i think we're looking at it wrong.
;^).
true, the majority of the data passed is static (images are almost all static), where's the big load show up? is it serving the images or generating the pages?
as a side note, i generally agree that benchmarks are mostly worthless, showing performance in only a very narrow range of situations. perhaps it's time to start an open benchmark initiative (or, more likely, i missed it being announced
/jbm, another moron who lost his password.
time to get it back.
jbm@intertek.net
The web-server IBM used was Apache (with their own name stamped on it). I believe IBM is releasing their improvements back to the community (but Apache's not GPL'd so they don't have to if they don't want to).
--Ben
On a slightly different point, I just have to laugh at Linux advocates complaining about "hacking the kernel" to get better performance. Does anyone really think the very design of Linux's major subsystems hasn't been heavily influenced by a desire to improve performance in particular situations (notably web service)? In my experience there are about a hundred times as many Linux hackers worrying about performance than about correctness, which is why we allow abominations like a non-journaling filesystem with delayed metadata writes (because if you try to force synchrony you get undetected data corruption, and the authors admitted as much in a public paper). This is the very worst sort of benchmarketing, case 4 above. Linux advocates must tread very lightly indeed when they presume to criticize other OS vendors of benchmarketing sins.
Slashdot - News for Herds. Stuff that Splatters.