Slashdot Mirror
Red Hat/Apache Slower Than Windows Server 2003?
phantomfive writes "In a recent test by a company called Veritest, Windows 2003 web server performs up to 300% higher throughput than Red Hat Linux running with Apache. Veritest used webbench to do there testing. Since the test was commisioned by Microsoft, is this just more FUD from a company with a long history? Or are the results valid this time? The study can be found here."
10%? 15%? Those are numbers I'd believe. But THREE HUNDRED PERCENT? I like Microsoft, and I like when somebody defends them. But this is just bull.
Let's see. A test commissioned by Microsoft says IIS is faster than Apache. The link for more information goes to microsoft.com. Is this really "news"? Seems more like a thinly-disguised press release...
Simpli - Your source for San Jose dedicated servers and colocation!
Notice the total lack of the CGI script?
What possibly possessed them to publish these results. No one in their right mind is going to believe 300% is an accurate figure under fair testing conditions.
Philosophy.
Why couldn't IIS be faster than Apache?
Is Apache/Linux the "end-all-be-all, there is nothing that can be better so let's stop trying" type of quality?
Are the guys who work at Microsoft a bunch of idiots that anyone can out-program?
I'm sure IIS is better at some things, maybe more things, maybe less.
Who cares! I don't think stats like these are why anyone chooses Apache/Linux over IIS/Windows.
Yeah...yeah...
I just wish, just ONCE that somebody would do a fair evaluation, without an agenda to forward. But I guess that'll never happen. We all have bias...but surely we could at least attempt to get above that?
Of course not, but that's not the point. Typically the guys who make the money decisions will read the headlines, and glaze over the rest, probably missing details like this study was paid for by Microsoft. Ever have to hand in a project proposal or such? I've done many, rarely does anything other than the Executive Summary get read. The rest is just there to make the document look good. Microsoft has a very big marketing department. They know this kind of stuff. Do you really think Microsoft would pay for these "studies" if they didn't show a positive return on investment?
wrong, it does tell us which is faster - linux. if Windows was faster, why would they need to benchmark against a crippled system?
sure there's a chance I'm wrong, but for me weighing the CHANCE of better performance from Windows against the CERTAINTY that they have lied about their product (or been completely incompetant) is a no-brainer.
and that's not considering costs (remember guys, using linux always requires an old, slow mainframe to be factored into the TOC!)
Under fairer circumstances, who knows, IIS might have still won, but this rigged benchmark has nothing to offer us in deciding which server is faster.
I've reached the point where I completely ignore all the studies and benchmarks like this, from both sides. It is, quite simply, far too easy to set the constraints and metrics up so as to make sure you come out ahead. What's worse, it has become absolutely standard practice to do so. Studies have become completely useless because you can guarantee that they've been cooked one way or another.
Jedidiah.
Craft Beer Programming T-shirts
I think they realized that the CXOs and other execs who make the big decisions never read the fine print anyway; and such disclaimers will never make the headline or a large-font pull-quote in any such marketing literature, so there's no harm in being up-front about it.
Even fake grass-roots efforts (astroturfing) can be done openly these days.
Oh, and to get on the Team99-bloggers-good side, I just wanted to say that Longhorn is so stunningly awesomely good that Microsoft won't have to resort to this kind of silly FUD once longhorn is released.
Wonder what that benchmark would be if you installed the FULL Norton package on it?
This bull reminds me of those advertisements for weight loss.
BEFORE................AFTER
Stick stomach out....Suck stomach in
White......................Tanned
No cosmetics..........New facial
Front shot...............Side shot
Grubby clothes........New fashions
Well, if you really want to know, you could probably do one. It couldn't be too hard to put a simple one togeather, and it would solve the question for you.
People keep saying, 'When are we going to get a real benchmark?" Well, why don't we roll our own? Seriously.
Here's my idea:
Slashdot has strong zealot^H^H^H^H^H^Hsupporters for both Microsoft and Linux. Let's have a contest to select the best qualified from each side, have them work in teams on identical hardware. Let them make any changes, tweaks or optimisations they can dream up. Then, let 'em rip.
I'm dead serious about this, by the way. Let's get off this endless roundabout and for once make a clear comparison.
For bonus points, once the first contest is finished, we should take the two servers, leave them exposed to the Internet and see which one gets 0wned first. 8^)
Crumb's Corollary: Never bring a knife to a bun fight.
So, for that piece of crap, you say "a *lot* of stuff he says is 'true'" (emphasis mine), and the best you offer is that people using appellations like "Micro$oft" burns your ass? Come on. Maybe it's the "penis envy" part? Or the part about "Setting up a server in Windows takes a couple of minutes"? Sorry, I see a bunch of wanking and sour-graping, combined with outright lies, and a few pointed pokes at the Linux Zealot community - which I don't credit with any 'hit points' because this is obviously from someone who would like to be a Windows Zealot, but isn't competent... or maybe it's a Linux Zealot making Windows Zealots look like idiots?
Thinking outside my Head
I actually thought that this was common knowledge--that Windows Server 2003 with ASPX was faster than Linux/Apache with PHP, or that Server 2003 was generally faster with static content. (I admit, I only glanced over the article, and Adobe Acrobat's search tool is the worst of crap, so sue me if it didn't mention ASPX).
.NET framework for web content delivery (get it--the 'X' makes it sound cool. Or something). .NET is compiled, and ASPX needs neither process nor thread creation. Like any .NET application, ASPX can run sort of close to native speeds (native + lots of wrapper overhead + generic memory management overhead and such.)
1)ASP (not ASPX) are fairly flaky and recent versions are roughly comparable to, but slower than, PHP4 (not sure about 5), in general.
2) Windows is not very good at creating new processes quickly. This is why CGI (not fastCGI) in the platform is so glacially slow.
Let's have an example. Let's say that you make a dynamic webpage in which all content is generated by a C++ CGI program. Ignoring database access for the time being, since that dilutes the example, on Windows, the website would be MUCH slower than the same website written in ASPX, even though the actual execution time of the C++ program is shorter (assuming a competent C++ coder).
This is because for each request, Windows must create a new process (the CGI program), and destroy the process when the request is complete.
While the execution time is low, the process management overhead dwarfs the actual page runtime, because Windows doesn't do that sort of thing quickly. This is why CGI has long been blacklistedon Windows systems by good web devs, and this is one reason that Apache 1.x was such a dog on Windows. Apache 1.x creates a new Apache process for each request.
Now Linux, on the other hand, creates processes about as fast as it creates threads, which is to say, really damn fast. Apache 1 has always worked just fine on Linux (and indeed most Unix systems) because the overhead of creating a process, while significant, isn't slower than a dead slug stuck in frozen molasses like it is on Windows.
Apache 2.x allows requests to be served by a thread or a process, or a number of processes that each create several threads (any Apache gurus please correct me if any of this is off).
It follows that this isn't a big deal on Linux (because process creation isn't really much slower than thread creation), but is a very big deal on Windows.
Windows has ASPX, which is Microsoft's marketing term for the use of the
Yet Apache is still back here creating a process or thread for each and every request (note that there are some ways to speed things up. FastCGI comes to mind, but I don't want to get into the gory details that I don't know enough about). This is not the brightest way to do it in terms of performance, but then, Apache appears to have been designed for universality and configurability over raw throughput.
It is unwise to hold the attitude that Apache can't be beaten by IIS, especially when IIS is optimized for one platform--by the vendor of that platform. Apache isn't even the fastest on Linux. Take a look at Zeus webserver. It serves circles around Apache on any platform it supports--including Penguin land.
In fact, Zeus uses a technique called SendFile() which, oddly enough, is strikingly similar Microsoft's own TransmitFile() API. Hmm.
Think of it this way: Apache is to IIS as GCC is to ICC, at least in terms of performance and generality.
Intel's compiler (ICC) consistantly blows away GCC in terms of the performance and size of the compiled code, but GCC runs on just about anything with a CPU, can cross-compile, is free, doesn't pull any PHB evil tricks, and actually compiles things like the Linux kernel without pat
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
If someone publishes a benchmark about your software, and finds out your software does not perform well, don't whine, don't behave like a child, don't start kicking and screaming, don't tear his hair out. Behave professionally.
Good starting points:
Let me summarize what I think about their test. First of all, I believe their numbers. Apache sucks performance-wise, in particular if you run a busy site with dynamic content. That's why people are using squid in local accelerator mode before Apache. This is a good indication that some performance tuning is in order. But no, people rather wait for Microsoft to find out and then they start thinking about fixing it.
If this test was meant to be unfair FUD, they would not have tested TUX, just Apache.
But now to my questions above:
Question 1: is their setup relevant?
No. Sites who answer more than 5000 requests per second are not using a single web server, they are using a load balancer and a cluster.
Question 2: Can their numbers possibly be true?
The point I find least believable is that IIS had better CGI performance than Apache. Creating a process is really slow on Windows. Their result should be independently verified.
Question 3: What weak spots about the competition does their test reveal?
They did not test a single-CPU webserver (which is what almost everyone is using).
They did not test FastCGI or APAPI dynamic web pages.
So if we wanted to do a more balanced review, we would look at these.
Question 4: What can we do to improve the results.
Document APAPI better, I'd say. Almost nobody is writing their dynamic web page modules with APAPI.
Everyone is using PHP or mod_perl. Benchmark Apache in real-world scenarios. Document best practices.
> 1. You rejuvenate and dance when you hear a windows flaw exposed, but you conveniently ignore the thousands of security flaws exposed in linux.
"Rejuvenate" means "renew, appear to grow younger". Did you mean "become jubilant"?
I don't become jubilant when anybody's security flaw is exposed. In the case of Open Source apps, patches are generally available in a couple of days.
> 2. You yell loudly TROLL! at any person's post or at any person you see posting facts that you do not want to hear about your oh so cool linux.
No, just the ones that misstate the facts or are attempts at FUD.
> 3. You know it's a classic case of penis envy, you don't have all the support, software and hardware available for linux and you have to let that anger out somewhere, but you don't have the brains to admit it.
Um, Linux supports all my hardware just great.
> 4. You hate windows, hate Microsoft, but race to emulate windows, have programs to run office from within linux, and spend a $300 on a Windows emulator, only Windows fools.
> I run Linux, Windows, and Solaris machines. I use OpenOffice.org and so have no need for Microsoft Office. But if I did, I could run it using WINE, which I can get for free. Unlike MS Office.
> 5. You cannot admit that you don't have professional usage of Linux outside server markets.
I use Linux *professionally* on the desktop.
> 6. You cannot admit that most of the joe user out there when told that there is linux will respond, what is that?
Sounds like there's a need for some consciousness-raising, then. Alothugh I've noticed that more and more people -- even Joe Sixpack types -- don't go glassy-eyed when Linux is mentioned these days.
> 7. You cannot admit that there is no professional printing capabilities in linux.
I don't have any problems printing from Linux.
> 8. You cannot admit that you are a masochist (otherwise why would someone spend hours playing with scripts, and recompiling programs that are available for Windows?)
Well, it did take me about 30 seconds to learn how to type "./configure - make - make install - make clean". Or if I'm feeling lazy, I can just double-click an RPM file icon in Konqueror.
> 9. You cannot admit that there is no professional desktop publishing done on Linux.
Sorry, mate, you're talking to someone who does just that for a living.
> 10. You cannot admit that no one in their right mind would do professional video editing in Linux.
I honestly don't know about that. But I do know that lots of movies' special effects are being generated these days using Linux-powered render farms.
> 11. You cannot admit that linux sucks when it comes for gaming/home entertainment or education.
There are tonnes of educational apps available for Linux -- many of them come with commercial distros. There are still more on the Net. As for games -- if I want to play games, I'll buy an X-Box.
> 12. You have problems in understanding Windows, and you will blame your own incompetence on Microsoft.
Over the years, I've used and administered Windows 3.1/95/98/Me/2000 and have no problems doing so. But after just 6 months, I can install, configure, and administer a Linux machine faster and more reliably.
> 13. You have problems in pointing a clicking, but have no problems in wading through cryptic scripts written by lunatics.
Pointing and clicking has its place. But there are lots of things that are actually easier via a command line. For instance, I'd much rather run a MySQL server that way than use the GUI tools. Nice thing about Linux and Open Source apps in general is that you've a choice in the matter. If you don't like the command line, don't use the bloody thing.
> 14. Nothing will get past that shit that fills your head, you will not admit to any facts.
Can't respond to an assertion that's semantically nil, sorry.
> 15. Yo
Il n'y a pas de Planet B.
I personally do not trust someone who claims to be "Veritest is an independent testing agnecy authorized by Microsoft to carry out the testing for applications developed on windows platform." to do a fair evalution of Linux vs Windows. If a company who makes a product gives you a huge pile of money at regular intervals and you are asked to compare that product to another product, who are you going to vote for? Who is your daddy? Sadly, money is everything.
9/11: Never forget it was a false-flag operation
The answer to "why now" is on the Veritest web page.
The paper is hilarious if you actually read it.
Key points...
They looked for tips on optimising Apache on the Redhat website ?!(guys next time try httpd.apache.org).
They found that Apache 2 was 50% faster than Apache 1 (without any tuning, Apache 2 offers a selection of threading models so a fair comparion would have tried each in turn, with or without tuning), so presumably didn't do any further tests with that in case it made MS look bad.
They tuned Windows for the server but effectively plead ignorance of how to tune Redhat for the server.
My guess is even then, on this hardware, sensible tuning of the kind they did to Windows would have made Apache comparable or better.
There were issues with this hardware selection at the time. Driving Gigabit ethernet is pretty demanding stuff, and you need drivers that can handle interrupt load.
However anyone who actually needs an 8 CPU machine to serve gigabit websites will probably do their own benchmarking and tuning, in the unlikely event their application software gives them a free choice of platform.
When my employers online business is big enough to need gigabit hosting, we'll probably still have SQUID on Linux with shed loads of memory accelerating the static content, because hardware is cheaper than rewriting the corporate applications.
First of all, performance is in many cases the main issue when buying a system. If you have a huge load then the choice may be between buying one well-performing system or two or three systems to take that load. And when you say "price is" I assume you mean the retail price, which is to me almost totally irrelevant (at least in the long run). The real price for any kind of software is the cost of teaching humans how to use it, costs related to maintaining it and most importantly, costs related to downtime and lost productivity if the software is slow. Performance DOES matter.
9/11: Never forget it was a false-flag operation
IIS was heavily tweaked for this benchmark. But for most uses, both IIS and apache are fast enough out of the box.
Do it like this:
-invite MS and RH (or other) for the contest, but ask what software versions to buy off the shelf and what to install. Install it yourself and have them tweak it with the configuration, without having extra's installed.
-Let them agree between each other on as many things as possible: hardware, time to tweak, do's and don'ts, type of tests
-Compile several benchmark tests: load, response time, static, dynamic. Ofcourse all on the same type of hardware platform.
-Run the tests with them witnessing and report here!
One note: Make sure to get the experts on both sides. At MS not a problem, but the linux arena is much bigger.
Well, I wouldn't use them for purchasing decisions, but this kind of studies are sure useful for pointing out weak spots in your favorite product. Time for Apache hackers get busy and fix such embarrassing performance scenarios.
I don't think Apache is the right server for static pages and simple CGIs though. It has so many modules and settings that the code path from filesystem to socket has to be much longer than necessary and longer than the feature-limited competition. They should try a simple server like Boa.
What someone should do in these kind of tests is get an expert Windows team and an expert GNU/Linux team, identical servers and let them configure them as best as they can. That seems fair.
Me lost me cookie at the disco.
Hardware support maybe more complete on the x86 platform but that's it. Linux has far superior hardware support over all.
would like to see more effort towards binary compatability in the kernel to support binary drivers a bit more consistantly though.
There is a reason that binary compatibility doesn't exist in proprietary drivers. Linux is a free system and was never intended to support 3rd party, proprietary drivers.
IMHO BSD is probably a better OS option than Linux is in a lot of ways (free/open/net)... though linux has the fame, glory, and fanatic following. I like windows, I use windows...
I see this posted all the time but It just leads me to beleive the poster has never used Linux or BSD. It seems like people are using this line to try and ward of criticism by saying "look, I like free software, just not Linux", or "I like Unix, just not Linux".
Time makes more converts than reason
How is an "independent agency" "authorized"? I would *love* to see those NDA's.
C|N>K
Don't forget that MS changed the EULA so that you are no longer allowed to benchmark windows and MS products without written permission. The only commercial people who can benchmark are those who will use a framework defined by MS.
Any other options will mean no study and no money.
He who pays the piper calls the tune.
In the end, It's all bovine dung you know
The red flag here is "300%." I don't think anyone can take it seriously with such a large desparity. That's like two hybrid car makers competing and the salesman tells a customer "Yeah, they get 50mpg but ours gets a bagillion-zillion!"
If you require identical hardware, some might complain that it's more suited to a particular system. Give them a fixed amount of money for the server. Or a fixed amount of other resources that might be the bottleneck (power, floorspace, maintenance time/month,...)
What keeps me going is my inertia.
You don't have to load any of those modules if you don't want to. Thereby simplifying the code path...
That's why they choose to make them modules rather than baking them into the application itself.
Of course, the budget has to buy both software and hardware.
:-)
After all, we want to count TCO and performance, right?
Actually what is needed is a public, non-profit benchmark competition. Both Windows and Linux enthusiasts are welcome to join in. Limit the contest to 100 teams of up to ten people. The 100 teams are all suprvised by the people who run the contest. The contest itself should make no money of any kind in order to keep away any monetary incentive. Hardware donations from the big players are acceptable with the understanding that the hardware will be returned after the compeition. In this way, the ugly little trait called "competition" gets in without any monetary incentive. At that point it's enthusiasts trying to outdo each other on both platforms. With this set up, you really get to test the performance of both OSes in a fair way because enthusiasts are likley to know all the tricks to get their OS and application to perform best. This means you'll likely see Windows outperforming a typical Windows system and Linux outperforming a typical RedHat/Mandrake/Debian/Gentoo/SuSE Linux system. Sounds like fun. SO who wants to get this party started? :)
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
I never understood why this is not illegal. Is preventing benchmarks not a restriction on trade?
Imagine if the automobile industry did it?
-b
I have to applaud the way you take a positive stance and look at how apache can be improved. I expect efforts in that direction form an ongoing part of apache development, but the positive attitude is appreciated. It's just a bit sad that your post reads as an endorsemnt of a blatant piece of paid-for propaganda
Don't let THEM immanentize the Eschaton!
Not so. If it were, there would be far less support for Open Source projects. Fortunately, as FOSS has demonstrated, large numbers of human beings are quite capable of being motivated by interesting problems and the knowledge that thier work will benefit everyone else.
Be cynical if you like, but every day you use Linux or Open Office; every day you see a website served by Apache; know that it's because some people value contribution to society enough to donate their time and creative energies.
-All that is gold does not glitter - Tolkien
www.ra
The budget has to buy software, hardware and setup labor.
This eliminates the problem of "that hardware favors Microsoft" or "that team had better engineers". It all comes down to money and value.
Of course the competition would need to state up front exactly how performance would be measured and how the various different tasks (static pages, cgi, etc.) would be weighted to come up with any overall scores. That would dictate the design choices made by each team.
One simple rule for its versus it's
Surprisingly (controversially?) enough, some EULAs forbid public criticism - I wonder if such clauses would ever be found valid in court, I seriously hope not - judges should declare void in whole any EULA that includes any anticonstitutional demands.
Ever hear of a non-disclosure agreement? Are those 'unconstitutional' in your fairyland world too?
The US Constitution only affects dealings between the government and private citizens, it has nothing to do with dealings between two private citizens. But then again, this is Slashdot. Carry on with your groundless bitching.
Usually unless they otherwise state so, benchmarks are done with the out-of-box settings, you can also do several things to enhance IIS performance, so I am not sure if your analysis is really valid. Although I do have to say IIS 6 and IIS 5.1 are nearly completely different, microsoft really took a leap forward when they designed IIS 6, so its interesting.
As to the Microsoft sponsored benchmark, Im not 100% convinced that if you pay for a benchmark its going to be biased, unless you black-mail the company doing the benchmark, I would like to think that the company holding the benchmark would be lookin out for there name also, and not skew results to make them look bad.
But thats what I would like to think, to much past with how microsoft has handled these things, but of course microsoft isnt the only one guilty of this, intel is just as guilty as well as AMD, Nvidia, ATI, etc.
If you had a business would you like your benchmarks skewed a bit to benefit your company? I'm not sure if I would or not, money is tempting but technology doesnt advance for money it advances for competition.
"If I was smarter I could rule the world!"