Slashdot Mirror
Web Server Comparisons
Anonymous Coward writes "ZDNet is running a story today (yes I know it's Christmas) about several web servers. They have apparently benchmarked these servers. They tested Sun, Linux and MS servers. I'm not sure that I like the way they tested, they didn't include BeOS or tune the configs. Check it out at their web site"My fault - this is totally out of date. Blame it on the egg-nog.
May 6 1999? That's today?
;)
Man, I better get ready, the 4th is coming up soon!!! Gotta find the best fireworks vantage point...
I wasn't too concerned that they didn't include BeOS, as it's more of a desktop O/S than a server. FreeBSD on the other hand should of been included, as it's definately a good server O/S, and is used to power busy web sites such as yahoo and hotmail.
Nope, this isn't cutting it... I'm not that big a FreeBSD lover, but I am taking exception to the fact that it and its brethren are continually left out of these benchcrafts...er, marks...
Because of its speed, ease of setup and administration, and a dizzying array of add-on and development products from Microsoft and third parties, we award Microsoft's Web platform--Internet Information Server (IIS) running atop Windows NT--our Editors' Choice.
Of all the reasons NOT to choose a web server, I think they hit them all. Granted, speed is important. But ease of setup? administration? (maybe) add-on and development products? (must mean ASP, & other proprietary compoonents... that "mixed web" environment they spoke of earlier in the article must not be that important)
Whatever happened to security, or - since "ease of setup and administration" are such a factor, how about security "out of the box"? Granted, I'm glad to see Linux listed, but (no offense) Caldera's not exactly the distro I'd pick for my web serving. Nor would I use Stronghold. (personal prefs, made through experience)
Just seems like ZDNet refuses to just get things right. Ordinarily I hate to be the crybaby bitching about the testing, the methods/materials, etc..., but I'm really becoming more and more disappointed in ZDNet's lack of integrity.
Flame away...
- Mindcraft is known for making benchmarks to suit the manufacturer.
- Benchmarks can be wildly manipulated... - Hence we should call this practice benchcrafting!
The tested using a 2.0.35 kernel. This is about 1+ year old and doesn't contain support for upgraded SMP performance. While an accurate measure of Linux then, it's far from accurate now.
Figures lie, and liars figure. What else do I have to say?
They "tuned the cache" to the point that none of the servers went to disk for their workload. That right there gave NT an advantage. In a real world scenario I'll be out to the disk all the time. I'll be there to get my data during misses, to write my state info for transactions, to log things.
How much logging was the NT server doing? If it wasn't a lot then they took out the disk subsystem from the equation.
The whole reason they used the 2.0 kernel was fishy at best also. 2.2 TCP stack broke communications with their win 95 clients?
Finally, why on earth would I want to do hundreds or thousands of SSL transactions in software. If you are doing more then a few a second you really need a hardware SSL brick or card, which works with all the tested platforms. These people obviously don't understand but one set of solutions.
_________________________
they repeat bogus reasons for using Microsoft servers ("if you have existing business logic, such as pricing strategies, written in Visual Basic..." -- hello, you want to call Visual Basic script, written by an accountant's assistant from HTTP server in secure environment?), measurements made with CGI scripts on Unix (heard of anything else?), their WebBench tests that have nothing to do with real-life environments other than that it uses HTTP, never use Gigabit ethernet cards in their low-latency tests, "demonstrate" that SSL servers are dominated by Microsoft IIS by splitting branded Apache-derivatives into different categories, don't include Unix servers other than Apache and Netscape, omit *BSD, etc.
In other words, usual advertisers-driven "Editors' Choice" stuff from ZDNet.
Contrary to the popular belief, there indeed is no God.
I don't understand how the 2.2 kernel didn't work for win 95 clients (as stated in the article). Was this ever true, and if so, for what version of the 2.2 kernel?
Also, what is the state of threading in apache? Is there currently any support for it (the article states the 1.2.x version they used didn't support it at all)
Not a very bad article (it is somewhat old now), but maybe not to relevant any longer. (And they seems to thing that web based administration is easier than config files... well, whatever floats your boat.)
"It's overkill, of course. But you can never have too much overkill." - Anonymous Slashdot Coward
Linux really didn't seem to do poorly. In fact I couldn't tell a difference between the platforms..
When using CGI, they all seemed to do about the same, yes, the two that ran NSAPI and ISAPI kicked the living crap out of the ones running CGI scripts, but I'll bet if they'd slapped mod-perl onto the Apache server, Linux would have caught right up..
greg
dronf!
They don't show the formula that gives the number, but from similar web benchmarking reviews, I know that even the worst ones are serving up hundreds of page-views per second. The best are maxing out multiple 10Mbps Ethernet cards - i.e. you need a T3 line to actually provide the bandwidth you're serving.
If you can afford that, you aren't reading Ziff-Davis to make your product decisions or even find your shortlist.
These kinds of servers are only needed by the big ISPs and the eBays of the world - the whole review is only of interest to a few thousand webmasters.
My employer is a city government serving some 860,000 people with a mostly static, partly active web site about all their city services and taxes and utility bills - and it rarely exceeds a few tens of pageviews per second.
Forget all the sniping about tuning and benchmark methodology; the really stupid thing about these product comparisons is that they imply that more than a fraction of one percent of their audience should even care about which one wins. For the rest of us, a free product running on a free OS and hardware that costs less than the monthly cost of our Internet bandwidth can meet all our needs.
Maybe you'll consider (Score -1: Troll), because I'm suggesting rational discourse over religious whining.
It appears from this thread of comments that the slashdot community is unhappy about all sorts of things that don't seem central to the issue. The comments becry ZDNet's advertising vs testing integrity and methodology, Caldera distro's shortcomings, Stronghold's performance vs other Apache releases, and why they didn't choose the EndOS-BeOS of solutions.
The issue, as I would see it, is "what can Linux do, to fare better in third-party comparisons?"
It appears from the article that there are several reasons that ZDNet listed why they felt Linux/Apache/Stronghold was limited. Let's start with those.
Why did this give NT an unfair advantage? Why does Linux (or particularly the Caldera distro) solution not deal with RAM-rich servers as well as NT? I think the poster who complained of this meant the inverse: if it were a RAM-poor server, Linux would have the advantage in disk accesses.
All religious handwaving aside, why did NT fare better by spinning threads than Apache could do by spinning processes? What is the big bottleneck in managing a process, that managing a thread doesn't have? They were using a brand-new MP kernel straight from Linus. Will the Linux kernel mature to deal with SMP situations and massive numbers of similar threads or processes better?
Though I think they should have configured some PHP or Mod_Perl into their mix, just as they had to bend to the SSL3-only constraint for another platform, they have a point. Writing modules is the way to go, to get inside the server and run fast. Besides PHP and Mod_Perl, where can Linux go to improve?
Most of these seem to suggest that ZDNet could have configured their Linux servers more like real-world Linux admins would, and would find better performance. This does talk poorly of Linux's ease-of-use, though they lay it on thick when they cry about config files. This is an education issue.
Slashdot is already slanted (pardon the pun) towards the Linux solution. Not every problem is a nail, and there are even different hammers for different nails. Let's be objective and constructive, instead of whining about every possible outside excuse. Improve the tool, and the tool will become the standard.
[
It seems that ZDNet doesn't like to acknowledge any non-commercial software, and went out of their way to test commercial products wherever possible. CodeWarrior won't run on Caldera, AFAIK. :)
Gates' Law: Every 18 months, the speed of software halves.
hemos didn't say that zdnet was running it today, that stupid anonymous coward said that they were running it today. hemos was not wrong in this case. he just was kinda lazy to edit the post or add a disclaimer "actually, it's may 6th.." but, in any case, may 6th, 1999 was a long time ago. that probably explains why they were running linux 2.0.35 and not something reasonably current (i.e. 2.2.x).
-- adraken
ancient kernels available for download. 2.2.0 .... :-P
---
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
Recentally, I setup a Linux router with 8 10Mb/s feeds (full duplex)inbound with one 100Mb/s feed going out. What happened is that at about 20K interrupts a second (about 70Mb/s), the system, which was running fine (about 97% idle) started sucking up cpu time. By 30K interrupts, it had saturated the CPU. In this setup, I made use of the vlan code (it was attached to a vlan switch), and used an Alteon AceNIC which does interrupt coalessing. The AceNIC under the same loads ran with about 1600 interrupts a second, and continued to run with about 97-98% idle time at the same traffic levels that had pegged the system before. I'm wondering if special drivers that are doing interrupt coalessing are making the difference on the NT boxes. Assuming that each transfer generates on a WebBench 20 interrupts (or more), then the 1K connections a second and the CPU load are really understandable.
On another note, why not have different groups do benchmarking with a fixed $$ amount that they can use to purchase equipment, as well as fixing a $$ amount per hour that they have to spend configuring the servers. This would be a MUCH more realistic benchmark scenario as the cost of equipment and time are realistic factors in the real world.
Erik
Also, what is the state of threading in apache?
:-)
Apache 2.0 will be a hybrid forking/threading server thus giving it some of the speed advantages of threading while maintaining the advantage of multiple processes that every one of these benchmarks never mentions: stability.
If for some reason one of the threads in a multi-threading server crashes, it can bring the whole server down with it. If one of Apache's child servers crashes... Apache forks a new one to replace it. The new design will be a compromise with several preforked children, each of which is multi-threading. Then let's see what the benchmarks look like
Chris
San Francisco values: compassion, tolerance, respect, intelligence
Apparently
www.zdnet.com is running Netscape-Enterprise/3.6 SP3 on Solaris.
Why did they choose that platform if the IIS on NT solution earns the editors choice?
Or the Webmasters of Zdnet.com disagrees with the Editors?
Probably because people don't bother tuning the Linux kernel for good SMP performance on web tasks.
Why would that be? Because it doesn't make sense to pay the premium for four processors in a box when for less money you can get four processors in four boxes and quadruple your disk and network bandwidth. People use SMP on Windows mostly because of software licensing costs, because of per-box colocation costs, or because it sounds good. In my experience, SMP on Linux is mostly used for getting really high performance on numerical and scientific tasks.
Apache has a perfectly good in-process module API; mod_perl is written in it. You can write that kind of code yourself if you want to write C/C++ extensions to the Apache server.
I consider it pretty foolish, however, to write web server extensions for an E-commerce site in a language without runtime error checking or fault isolation. Applications specific modules simply can't receive the testing and debugging that Apache itself has received. You are lucky if the server crashes due to a bug; more likely, you are going to ship an unpredictable quantity of widgets to an unpredictable address as a stray pointer leads to overwriting some of your order data.
The performance difference between native code and Perl, Tcl, Python, Java, or PHP3 simply don't matter in most web applications: applications are generally bandwidth or database bound. Given that simple fact, you should use the safest and easiest language to program in. For single programmer projects, that's likely to be a scripting language. For large, component-based multi-programmer projects, that's likely to be Java, OO Pascal, Eiffel, or something of that kind.
Shit, yes, I do huge file transfers and video conferencing over HTTP all the time!
F'chrissakes, use a protocol that actually suits what you're trying to do. FTP, NFS or SMB for file transfers; one of the streaming protocols for video conferencing. (Actually, come to think of it, video conferencing over HTTP isn't really feasible; what are you doing, breaking the video stream up into frames, converting them to JPEGs and pushing them to the clients?!)
I believe that's spelled "woozy"
Since this is an Intranet, have you considered just having the Web server provide only a "file:///" URL and letting a file server handle this massive load? They're much better tuned for it.
The point is very well taken, however - bandwidth is not the limitation in a LAN. Still, the problem doesn't come up in my workplace - and we're talking 4000 seats. Our biggest Intranet server also maxes at a few tens of hits per second.
Perhaps that's an indication of our Intranet usage being backward or something, but I don't think we're all that far behind.
A larger factor is your computing philosophy - is your Intranet a highly centralized "mainframe" style with one provider of information to many-many-many? With so many diverse departments in a civic government, ours is more spread out among many servers, even though the IT department runs them all out of one room. If you put hundreds of functions from hundreds of information providers onto one web server, then its security arrangements and the tuning of the server become very complex.
Lastly, if you have very heavy web usage because your corporation is practically run from a couple of major applications - say sales management or the accounting system - it may be better to consider that this is not best done with web apps but with a "traditional" client/server app installed on every machine.
To sum up, with the options to serve lots of (or big) files with a file server, to split multiple services into many servers on the KISS principle, or admitting that not everything is best done as Web apps, I again return to my point: that not that much of the total Web server market cares about getting over 1000 pageviews/sec - even on Intranets.
Zag. You should know that it's not nice to tweak people.
Especially since you're inaccurate.
You CANNOT telnet into NT. You CAN find a shareware telnet service, but you cannot telnet into an OOB NT box. Period.
I know you're in love with your Win2K box, but sheesh. Please note that he DID mention Win2K being telnetable.
*SPANKSPANK*
Bad boy! Now you have to listen to Reba West again!
Chas - The one, the only.
THANK GOD!!!
Chas - The one, the only.
THANK GOD!!!
I know about H.323, but AFAIK the H.323 standard talks mainly about transmission over LANs and specifies nothing about encapsulating it with HTTP, so your last statement doesn't really mean much, does it? It's like saying, "A firehose is designed to carry water, so there's nothing wrong with encasing it in a garden hose." I mean, that's gotta kill your bandwidth.
My point (which applies to HTTP-DAV as well) is that most people would be better off using a server and protocol that are designed for the way they're going to use it, rather than trying to stick a square peg in a round hole. Just because your users are too braindead to use ftp/ncftp/WsFTP/CuteFTP/Fetch for file transfers, don't try and convince others that your solution is the best and web servers should be made to handle it.
Certainly, you're free to use the solution that suits your users, but saying things like:
If we have to sacrifice some bandwidth to make this happen, so what?
isn't going to make you many friends among most IT departments; not everyone runs Gigabit Ethernet (hell, the company I used to work at was using plain old 10Mb Ethernet for more than 350 clients (SMB, IPX, Ethertalk, TCP/IP) - without subnets or any other way of limiting broadcast traffic).
For some reason I thought it had always been spelled admission.
I'm a loner Dottie, a Rebel.
You're insulting him for being correct? Doesn't that make you an "utter complete moron". Ebay's backend uses Solaris with an Oracle database, their front end servers use NT. The frontend does a tenth of the work the backend does.
I'm a loner Dottie, a Rebel.
You have to remember though that M$'s site runs on a cluster on 96 Compaq Proliants running NT. Hotmail uses about as many as that.
I'm a loner Dottie, a Rebel.
Hi Ed.
ZDNet chose to tune ALL the servers to have 68Mb of web source material and at least 68Mb of memory disk cache.
Why did this give NT an unfair advantage? Why does Linux (or particularly the Caldera distro) solution not deal with RAM-rich servers as well as NT?
Well, I don't think you have it quite right. It is not that they are exposing a Linux weakness in handling RAM rich servers, it is that they are hiding a major performance bottleneck of NT -- its file system.
On the other hand, you get something for your performance hit -- filesystem journaling. It will be interesting to see what happens when ext3 and Reiser FS become more common.
In any case, I run a small web site that is 160MB in size, not counting databases. This is too large to be cached in the setup described, although not unreasonable to be entirely cached. However, if I was doing serious corporate intranet or a major ecommerce site, I would expect it to be much, much larger.
I am not a conspiracy theorist, but does it not seem a tad unrealistic to devise a web benchmark test which totally discounts disk access?
All religious handwaving aside, why did NT fare better by spinning threads than Apache could do by spinning processes? What is the big bottleneck in managing a process, that managing a thread doesn't have? They were using a brand-new MP kernel straight from Linus. Will the Linux kernel mature to deal with SMP situations and massive numbers of similar threads or processes better?
Well, on Unix, forking is cheap -- very cheap. On Windows, launching a new application instance is very expensive. Therefore multithreading is a huge performance win -- multiprocessor or no -- on windows, but a relatively smaller one on Unix. If you think about it, it hardly seems worth multithreading unless the threads are updating some common memory. If you write a thread with no critical sections, it may as well be a process on Unix, but on Windows it benefits from getting access to memory pages that have to be laboriously set up (they are simply copied in a Unix fork).
I am guessing that multithreading Apache speed improvements in Unix will scarcely be measurable unless nearly all the data being served out is cached in memory.
ZDNet suggested that in-process programming worked better for all the hairy e-commerce they decided to test.
... they have a point.
Well, that is a matter of opinion. Ebay may use IIS, but they do everything through CGIs.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.