Slashdot Mirror
IBM Sets SPECweb Record
the frogspotter writes "IBM has an article on their site talking about their new RS/6000 that can supposedly do 40,161 http ops./sec. as measured by SPECweb96. Jeez...that'd be 3469910400 http ops./day. "
And it only took 12 (out of 24) processors (on an unreleased server) and a hacked kernel! It seems as if
modifying the kernel for benchmarking is becoming quite a popular tactic these days...
Good point, why in the hell is hacking the kernel and getting direct support from the engineers who made the software considered fair in benchmarking??? When I first saw the Mindcraft results, I about laughed my ass off after reading that MS had direct help in the tuning of the server.
When I can get personal hacked/tuned kernel from MS, and have their engineers come out and work on my servers, I might consider running NT on the server side.
Come on IBM. Take it out of the box, do some Real Guy Tunning(TM) and then test it over serveral weeks. Then get back to us.
Linux O Muerte!
However, the "new and improved (tm)" SPECweb99 is almost ready: See the SPECwww99 home page
Web servers are frequently asked to send users static Web pages.
/. is heavily dynamic (are there any pages that are static?). I know my homepages are netscape and yahoo are dynamic. All search engines serve dynamic content. Certainly web sites designed for E-Commerce, the market for IBM's web servers, are dynamic (searching for products, filling out order forms, etc).
With all these web benchmarks coming out using static web pages, I'm wondering if there are some statistics available somewhere that can either confirm or deny this.
I know
For some reason I don't think the statement in IBM's release is totally accurate. Why not create a new benchmark or modify an existing one that had a mix of dynamic content that matched certain typical uses for web servers. A search engine server, a portal server, an e-commerce server, etc. This would give more accurate real-world results, IMHO.
Are there any projects out there aimed at modifying Linux so that it is THE leading web server platform? Obviously you can't compete with IBM and 24 processors, but what about a distro that is aimed at serving web pages, static and dynamic, at such a high level that it can't be matched in price/performance?
I know Zeus is fscking fast, but what about an operating system that is designed from the ground up to be a web server? All OS bottle-necks removed.
Is this a crazy idea? Is it possible? Seems to me like Cobalt's systems are a lame attempt at doing this (no optimizations, just easy setup).
Stew
Thats about what slashdot gets ;), well mabey not but it would be cool if it did!
Vive l'AIX!! The single most weirdest, scariest flavour of Unix! Has anyone seen those 21" RS/6000 monitors that are perfect squares? Even weirder.
Special tuning has always been a part of benchmarking. There are many compilers out there with special optimizations designed only to improve the performance of a specific benchmark (detection of certain matrix operations and other snippets of code which almost exclusively occur in benchmark programs). Many compilers, for example, recognize this piece of code:
which gets transformed to thereby skipping the sqrt calculation. Since this code almost only occurs in the Whetstone benchmark, in the general case it will never get used. (Hennessy & Patterson, CA:AQA, chap. 1.8)Some of the compiler optimizations out there will even generate faulty code if they are switched on when not compiling a specific benchmark program. Might be worth thinking about.
And don't think that only software can be optimized. Many of the processors out there have some special hardware hacks designed only to improve some benchmark program, and which almost never is used in "real world" programs.
Bottom line is, don't be surprised that they use some special tuning, it has always been this way and always will.
Is there any free web benchmarking tool?
Webbench from ZDNet only runs on Windows (".EXE for all platforms", right.)
SPECWeb96 costs $800!
Any suggestions?
--
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
>It seems as if modifying the kernel for >benchmarking is becoming quite a popular tactic >these days...
How about starting a new line of patches?
patch-2.2.11bench1.gz
:)
It is to IBM's benifit to make Linux do well. If Linux and UNIX looses out to NT there will only be Intel hardware around, and many companies will looses in teh server makret. However if Linux can in keeping up with the UNIX tradition blow away NT benchmarks, there large hardware companies (IBM, Solaris, HP, SGI) can still sell there hardware, with Linux. Linux will benifit from the support and they benifit as they have to do less for the OS. (Kernel changes mainly). This is sort of a symbiotic relationship I think.
Only 'flamers' flame!
The tuning that has gone on after Mindcraft has probably benefited performance for someone, and that's great, but I doubt it has made any difference to real web servers anywhere.
http://www.software.ibm.com/webservers/httpservers /download.html
Is there any hope that vendors will stop citing tests on a platform that is never going to be commercially available unless the results are specifically labeled as R&D?
I have not gone looking yet at the SpecWeb99, but I think some of the TPC benchmarks now require the platform to be commercially available within six months.
At least IBM was honest enough to tell us that there was special tuning and new hardware involved. That gives me some hope.
Hey IBM, how about posting the server's address so we can see if anything happens when we slashdot it?
Even a site that serves only dynamic pages still serves static files. Even if nothing else, the images are all static. So if you've got a busy site, it's important to have a server that's fast at serving static files. Thus, you might run an image server on a different server than your pages and dynamic content.
-Imperator
Gates' Law: Every 18 months, the speed of software halves.
Why is everybody so negative ? :)
Let them do what they want, if they can get Linux above microshit, I'm happy
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Belgium HyperBanner
http://belgium.hyperbanner.net
Linux hosting for $2.50/mo
Isn't it the idea of Linux that the source
of Linux is freely available, and that it's
purpose is to be adapted by others ?!
I say, IBM did a fine job. If they release
that pathches it would even be better...
This might lead to new and faster implementations...
should be left on the bench. The only use is to justify your expenditures to management. Personal experience is the best measure of what works, and what doesn't. Let the marketing departments drool over them. Personally, I'd file them in the circular.
Benchmarks only test a very narrow spectrum of performance, and is rarely representative of real world performance. Witness the mindcraft benchmarks. Specint is a "standard" now in benchmarking, hence it's to be expected that companies will be adding special optimizations that wouldn't otherwise be done.
--
Instead of all this stuff with httpd in software, why not just build it into the hardware? That would fry the bacon, so to speak. Imagine a ATM card with a 1 gb RAM buffer caching your web pages ready to saturate any goddam bandwidth that you can find.
Seems to me to be the obvious way to do it. They you can apply the general purpose CPU stuff to creating dynamic content where needed.
and what is /. server configuration ?
The world belongs to those who get up early. - I'm far from being the king of Earth then
Availability is really not a problem, TPC and Spec both have requirements about official results requiring that the system benchmarked actually ships within a reasonable period of time.
But, despite the efforts of the benchmark designers, the vendors will always find ways to tweak and tune their systems to optimize for the benchmark but not necessarily for the real world. If you are lucky, your workload looks a lot like the benchmark and the comparisons are valid, but more often than not, you are just better off getting loaners and testing them yourself.
Posted by Ungrounded Lightning Rod:
of the previous poster's comment. It DOESN'T
seem to ME that Cobalt was a "lame attempt".
Surely IBM can write a parellel version of httpd and blow away any numbers out there? When are they going to drop these silly standalones and market the machines with some real power?
Now if they could port PSSP to Linux, that would rule!
newton62 (56617) Karma: Bad
Grand Pa IBM steps up and says "Boys this is how its done if you want a good benchmark, build your own hardware, write your own software, do the test in-house, and write your own press release."
Who owns your data?
No doubt! AIX rules. JFS, LVM, SMIT, ODM, 64bit, solid as a Mac truck. I used to work in AIX defect support in Austin, was my 1st UNIX to play with. Jaded form the get go? Yes, but I've never seen a better UNIX in my years of playing with Solaris, DGUX, Linux, FreeBSD, SCO....
See 'ya,
Jim
including the homepage
Or any other design that completely removes or makes optional all system abstractions that do not contribute to performance for your application space. Read this link at MIT
And what's its keyrate? :)
-Chris
Such as network and router. To make it fair you have to have networking equipment that can feed the packets to the IBM box as fast as it can process them.
This is a growing misconception among slashdotters. Every major portal page serves some content off of static pages, yahoo a significant amount.
By the way, IBM HTTP Server, A.K.A. "IBM HTTP Server Powered by Apache" is Apache, now IBM's preferred webserver. So any performance improvements IBM makes to "their" webserver will go back to the community.
I checked the various bookmarks I have and about 95% of them are dynamically genrated. They're talking about putting a static web page generator in the kernel for Linux too. While this might generate a speed up for benchmarks, if it comes at the price of stability or security I'd just as soon do without it.
Other improvements, such as threading the TCP/IP stack, sound much more promising.
I've worked with AIX off and on since 1991. I remember the 3.1.5 debacle. Believe me, AIX can't point fingers at anyone without having three pointing back at itself. I remember the joys of ODM corruption, I remember the text config files (there for backward compatibility) that did not reflect the true (odm) config of the system, I remember when they took away the ability for an admin to edit the ODM database directly so that you could fix the screwups caused by some of the install and system config utilities, I remember creeping JFS corruption, LVM management utils that would allow you to shoot yourself in the foot...
Geez, AIX - why did you get me started? The only thing that would make me rant more are my years with the ultimate "Crappix" - SCO EunuchsWare. I will *never* look back. The only thing that could compete with it on the Crappix scale was HP/UX 8.x (and you thought 9.x was bad... boy, have I got some stories for you).
Christ, somebody get me to alt.sysadmin.recovery before I go nuts and piss on the mains...
-- ultra1
If all you're serving are static web pages, why have an OS at all? Make a web appliance which has only the stuff necessary to serve pages and a huge RAM cache. Completely get rid of the OS overhead.
Do they mean disk cache or RAM cache? If it's ram that's a hell of a lot of cache.
But the idea of storing frequently accessed static pages in a cache contolled by the kernel sounds like a cool idea. Wonder if there is any plan to implement this in the Linux kernel? Maybe as a module?
Steve
one of IBM's main goals in life is to amass
all the software patents in the universe and
use them as giant guns to destroy competition.
they could singlehandedly crush most small companies
with this war chest alone.
hope your 80,000$ hardware httpd card doesnt have any
glaring security holes, etc!
There is a in-kernel webserver (although not quite mature)!
It is called kHTTPd
http://www.fenrus.demon.nl
AIX is a royal pain in the arse. My current position is admining a whole bunch of RS6000s and while it's rock solid and quick, it's _DIFFERENT_ to everything else.
My last job was working on Data General AViiON systems running DG/UX, they are a real pleasure to use and administer, leaves AIX and Solaris in the dust.
It's a pity Data General's management now have their tongue stuck so far Microsoft's arse.
Deleted
DG/UX can do this with ease, but is a damn site smoother with it.
:(
I work with AIX, but wish I worked with DG/UX.
Deleted
After all, imagine what a c00l b3owulf you cold make with thi...
Well, if I remember correctly, the "MS engineers" were from the marketing department; if those actually finetuned the NT kernel, well, I'd reconsider a lot of what I think about marketing types.
On the other hand, the performance of the linux kernel was far improved by a few kernel hacks - which shows a remarkable response time, yet is not exactly good software engineering.
Seems to me that the reasoning behing this topic again was more along the lines of MS bad/Linux good. Waste of Bandwidth... Shrug.
While the Exokernel stuff is interesting, I think that VIA-like optimizations (http://www.viarch.org as previously noted on /., owned by Compaq/Intel/MS) are a better way to go. The idea was originally from a thesis paper and implemented for Unix. (Author not handy due to a recently deceased HD.)
VIA stands for Virtual Interface Architecture. The idea, generally, is to build device drivers that can directly interact with processes or threads with little or no interaction with the kernel. The idea is generally that the driver presents a virtual device interface to each process. The viarch.org guys appear to be concentrating on applying the idea to SAN I/O, but I think that the network/OS/Webserver (SMB, etc. also) is even more interesting given the smaller payloads and higher overhead.
What I would like is to have a process or group of threads that would interface to the NIC and receive a stream of packets and transmit a stream of packets. The exchange could be completely async with large queues and no context switches or other OS interaction (for network traffic). The service processes would contain an integral TCP/IP stack and HTTP server. The device driver would send packets for a particular IP address only to the VIA processes while handling other IP addresses normally.
This would allow optimizations just like those for the Exokernel. People could experiment with various zero copy schemes and other TCP/IP stack optimizations without affecting the operation of the system they were running on.
We need to do this now!
Is anyone working on something like this? Email me! I might even be able to provide some funding.
sdw
Stephen D. Williams
The Linux kernel has a DYNAMICALLY-RESIZING cache, much FASTER and BETTER than that lame fixed-size memory-eating smartdrive I used before under Win3.11 and DOS 6.22! When you're using the disk a lot, it grows; when you use memory a lot, it shrinks. Adding 64Mb of memory to a linux box can DOUBLE the speed after half a hour of uptime because of the caching effect!
Moving static requests back up into the kernel is a neat trick, to be sure. But I'd rather have web serving where it belongs: user space. The idea of being able to serve a page to every user on the planet five times over on a given day from my web server sounds nice on paper, but not very useful in real life.
Having a bit of up close and personal AIX experience, I feel they should work on making their kernel a bit more stable, anyway. This is simply posturing for a benchmark, probably not real useful in a real-world scenario, and to be frank, I think IBM knows better.
Do you have ESP?
All this complaining that the OS & software used to set this record was tweaked!!
I don't see the problem. These people had access to source for AIX and they used it! Just the sort of thing that anyone with the skills can do to Open Source software.
while dynamic vs. static is an important distinction, i think we're looking at it wrong.
;^).
true, the majority of the data passed is static (images are almost all static), where's the big load show up? is it serving the images or generating the pages?
as a side note, i generally agree that benchmarks are mostly worthless, showing performance in only a very narrow range of situations. perhaps it's time to start an open benchmark initiative (or, more likely, i missed it being announced
/jbm, another moron who lost his password.
time to get it back.
jbm@intertek.net
The web-server IBM used was Apache (with their own name stamped on it). I believe IBM is releasing their improvements back to the community (but Apache's not GPL'd so they don't have to if they don't want to).
--Ben
I am a Server HW designer and I am very familiar with the process my company used to test and publish results of various industry benchmarks. The SPEC suit was important as well as TPC-C (not sure if that is a subset od the SPEC benchmarks or not).
It was routine for the DB developers to work hand in hand with our OS developers and to a lesser degree the hardware developers. Yes we were tuning the entire system to optimize the results of a "Industry Standard" benchmark. However, that happened to be the best and most rigorous measure we had. When tuning the system, we had to have some measure to evaluate the merit of various tuning parameters and these benchmarks serve well technically as well as from a marketing stand point.
All this tweeking went on in development hardware, OS, and software (primarily data bases) but the optimizations were incorporated into the final products.
Further it was a requirement that any published benchmark numbers must be on commercially available HW and SW. If the HW or SW was unannounced, it was required that it be available for sale withing 6 months of the published numbers.
Ironically, the numbers we usually published were never as high as the actual numbers. They were always concerned that a tweek or patch might not make it into the final product so they hedged the numbers a bit.
For the complex tasks of servers and transaction processors, tuning to the benchmark is not cheating but rather using the best tuning measure available that also has some broad based acceptance and recognition.
Just my humble $0.02 worth.
On a slightly different point, I just have to laugh at Linux advocates complaining about "hacking the kernel" to get better performance. Does anyone really think the very design of Linux's major subsystems hasn't been heavily influenced by a desire to improve performance in particular situations (notably web service)? In my experience there are about a hundred times as many Linux hackers worrying about performance than about correctness, which is why we allow abominations like a non-journaling filesystem with delayed metadata writes (because if you try to force synchrony you get undetected data corruption, and the authors admitted as much in a public paper). This is the very worst sort of benchmarketing, case 4 above. Linux advocates must tread very lightly indeed when they presume to criticize other OS vendors of benchmarketing sins.
Slashdot - News for Herds. Stuff that Splatters.