Slashdot Mirror
Linux Beats Win2000 In SpecWeb 2000
PraveenS writes: "While not conclusive, the SPEC group released benchmarks for a variety of systems submitted by various manufacturers (i.e. Dell, Compaq, HP, etc...) and tested their Web-serving capability. Two very similar machines from Dell, one loaded with Linux and the other with Win2000 had very different results; Linux beat Win2000 by a factor of almost 3 . Here's a synopsis of the results from LinuxToday. The actual spec benchmarks are available here for Win2000 and here for Linux."
As Marty of LinuxToday puts it, though, "What does this mean? In the real world, probably not as much as it would seem. Benchmarks in general are typically set up in an ideal environment. Real world environments tend to be quite different. However, this does indicate that Linux is moving in the right direction."
Zoran points out that "[o]ther current SPECweb99 results can be found here." They make an interesting comparison.
If you had bothered to read more of the site, you would have noticed that those results are all submitted by vendors with interests in getting good numbers. If Sun wants to enter a Sparc/Solaris combo, they can do that. If Apple for some reason decided it was in the HTTP server business (which it isn't (yet?) by any stretch of the imagination), it can run the test suite and submit the results.
Gates' Law: Every 18 months, the speed of software halves.
'TUX' comes from 'Threaded linUX webserver', and is a kernel-space HTTP subsystem. TUX was written by Red Hat and is based on the 2.4 kernel series. TUX is under the GPL and will be released in a couple of weeks. TUX's main goal is to enable high-performance webserving on Linux, and while it's not as feature-full as Apache, TUX is a 'full fledged' HTTP/1.1 webserver supporting HTTP/1.1 persistent (keepalive) connections, pipelining, CGI execution, logging, virtual hosting, various forms of modules, and many other webserver features. TUX modules can be user-space or kernel-space.
The SPECweb99 test was done with a user-space module, the source code can be found
here. We expect TUX to be integrated into Apache 2.0 or 3.0, as TUX's user-space kernel-space API is capable of supporting a mixed Apache/TUX webspace.
TUX uses a 'object cache' which is much more than a simple 'static cache'. TUX objects can be freely embedded in other web replies, and can be used by modules, including CGIs. You can 'mix' dynamically generated and static content freely.
While written by Red Hat, TUX relies on many scalability advances in the 2.4 kernel done also by kernel hackers from SuSE, Mandrake and the Linux Community as a whole. TUX is not one single piece of technology, rather a final product that 'connects the dots' and proves the scalability of Linux's high end features. I'd especially like to highlight the role of extreme TCP/IP networking scalability in 2.4, which was a many months effort lead by David Miller and Alexey Kuznetsov. We'd also like to acknowledge the pioneering role of khttpd - while TUX is independent of khttpd, it was an important experiment we learned alot from.
Other 2.4 kernel advances TUX uses are: async networking and disk IO, wake-one scheduling, interrupt binding, process affinity (not yet merged patch), per-CPU allocation pools (not yet merged patch), big file support (the TUX logfile can get bigger than 5GB during SPECweb99 runs), highmem support, various VFS enhancements (thanks Al Viro), the new IO-scheduler done by SuSE folks, buffer/pagecache scalability and many many other Linux features.
> Linux zealots scream bloody murder and inspect the process with a microscope. Someone else does a benchmark that shows Linux 3 times faster than Win 2k, and they are content that the Mindcraft fiasco has been avenged.
Well, it could be that we notice that a standard benchmark was used rather than one tailored by a company with an axe to grind. Or it could be that the benchmarks were submitted by hardware vendors, whose primary interest is in making their hardware look good (i.e., it's really hard to imagine Dell fudging a benchmark to make Linux look better than Windows). Or it could be that c't already told us how Linux and NT measured up on more equitable benchmarks. Or it could be that Microsoft's own tests showed W2K performing worse than NT on systems with > 4Mb of RAM. Or it could be that testers have been saying that W2K needs +300 MHz in hardware to perform as "well" as NT did.
In short, there's no reason for surprise at all. This benchmark is only quantifying what the attentive already knew qualitatively. If there are flaws with the benchmark, they almost certainly won't be enough to tilt it against what we already knew; if they do, we'll air our suspicions again.
I do agree that it's still a benchmark, and is therefore susceptible to all the follies associated with benchmarks. But at least this one wasn't obviously rigged.
--
Sheesh, evil *and* a jerk. -- Jade
For Linux they set the backlog at 3000. For W2k it's at 1000. Anyone see the difference? AFAIK, W2k can have a higher backlog, or even a dynamic backlog. I'd like to see a test where the backlogs are the same. Then there would actually be similar simultaneous connection counts! Right now, those numbers mean little if they are being compared.
For instance, on the Windows side you might have an 8 way xeon with 2 gigs of RAM. On the Linux side you might have (for instance) an S390 with a terabyte or two of RAM. Then just start loading them down with network clients until they start to stagger.
I'd be interested in the oucome...
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
You are confusing two completely different architectural concepts.
"threads" (which get created) and "processes" (which get forked) are 'context of execution' entities. Linux has both, TUX 1.0 uses both.
A "threaded TCP/IP stack" is a slightly mis-named thing, it means "SMP-threaded TCP/IP-stack", which in turn means that the TCP/IP stack has been "SMP-deserialized" (in Windows speak) - TCP/IP code on different CPUs can execute in parallel without any interlock/big-kernel-lock overhead or other serialization.
A 'threaded TCP/IP stack' has no connection whatsoever to a 'threads'.
FYI, the Linux TCP/IP stack was completely redesigned and deserialized during the 2.3 kernel cycle, this redesign/deserialization was done by David Miller and Alexey Kuznetsov. The TUX webserver of course relies on the deserialization heavily, but this is not the only architectural element TUX relies on.
You assume that IBM, HP, Mindcraft and Dell are all in a big conspiracy to make Windows 2000 numbers look bad - are you kidding? The reality is that there is fierce competition for best SPECweb99 numbers, and Linux/TUX is just plain faster.
The other flaw in your argument is this TUX dynamic module. Check out the source code, TUX does dynamic modules. (besides, the SPECweb99 workload includes 30% dynamic load, so all SPECweb99 webservers must support dynamic applications.)
10 LIST : REM MER : TSIL 01
If I claim that I had a fight with Mike Tyson and he won, it's relatively unremarkable; the only implausible bit is that we might meet and fight in the first place, not that he wins. If I claim I had a fight with Mike Tyson and I won, such a claim is far less believable.
Thus, if your personal experience tells you that Linux kicks the shit out of MS operating systems for Web server performance, a benchmark test whose results accord with that experience is more believable than one which contradicts it.
That's just good sense, isn't it?
--
Xenu loves you!
do not confuse advocacy with information. FreeBSD and Linux are more or less at the same place when it comes to reliability, scalability, and network performance. at this particular point in time, I'd guess that Linux has the advantage with the improvements of the 2.4 kernel, but it doesn't really matter: FreeBSD and Linux are always catching up with each other; both teams are very good and neither will let the other OS get much better without getting better in the same (or equivalent) way. I'd say that, in choosing between Linux and BSD, you need to look specifically, either at personal preference and familiarity, or at the actuall support for the programs and services that you intend to run, and choose accordingly. Neither platform is overall significally better than the other.
NO. see this :9 92/
http://geocrawler.com/lists/3/Linux/35/150/3977
its VERY important that linux users are aware that the linux devleopment process is hitting a roadblock right now. things dont look too bright.
It will be interesting to compare the comments of this article to the ones regarding mindcraft. I wouldn't be surprised if two thirds of the people here take this to be the truth simply because linux won.
It would be nice if someone would run some benchmarks with the same two machines, only have the W2k box serving up dynamically generated ASP and PHP pages, and the Linux box serving up a comparable PHP page. Whack up some identical code to perform Fast fourier transforms in the page and make it spit out the result. Once you get a database into the mix, you're also measuring the performance of it, and this is just a webserver test. Unless of course you have both boxes hit the exact same database, maybe a nice big Oracle database running on Linux. :)
Everyone here knows that MS zealots will say "Yeah, but W2k can spit out dynamic content faster...". It would be nice to have proof either way. I know I'm very interested in seeing how PHP on Linux compares to ASP on W2K.
Need Free Juniper/NetScreen Support? JuniperForum
I thought about talking about this, but I'd have to use my experience and that is fairly subjective. I'm curious what do you think the difference in human costs are? Lets see, take 30 minutes to install/configure Redhat, Take 10 minutes to install the latest updates, 5 minutes to disable services that are not needed and block the outside world from anything except 80 and 22, 10 minutes to install openssh, 5 minutes to load an existing website on the machine. Reboot the machine just to make sure and you're all set.
Looks like about an hour to me. Maybe an hour and a half if you want samba and frontpage extentions installed.
Just for kicks, lets take a look at NT.
When I installed it yesterday it took about the same about of time to install as redhat, so lets figure 30 minutes. Configure for network and reboot 5 minutes, Setup IIS 15 minutes, add webpage to IIS 5 minutes. Reboot the machine just to make sure.
Ok looks like a total of 55 minutes. Great, MS just saved you 5 or 35 minutes depending on what you're looking for. Is it really worth a few hundred dollars, if not more for an MS webserver if you really don't need one?
Also, with the linux box, I can ssh in and fix things remotely, I don't even have to be there to apply a patch when it comes out. As a consultant I find that very appealing. I just scp a file over, install it, restart the service and I'm set. NT I actually have to be there, when some of my clients are almost 2 hours away, I'd much prefer the linux method.
I find it interesting that there's no Macintoshes or Suns in the test, although there are at least one Alpha and two RS/6000s. How can they claim to be a useful benchmark by concentrating mostly on Intel hardware, and only running three HTTP servers? I'd think that the differences between different servers running on the same hardware could be just as much as between different hardware configurations; hell, even poorly configured vs. properly configured systems would be a huge difference...
How can anyone claim that any MS sponsored benchmark has any legitimacy whatsoever as long as MS insists that there be no benchmarking of their products in the EULAs?
I wonder how SPEC was able to perform this benchmark?
-Jordan Henderson
First of all, these benchmarks (both the Win2k benchmarks and the Linux benchmarks) were posted by Dell, not by some random Linux zealots. Not only is that the case, but the other WinTel vendors have very similar scores for their WinTel hardware. Does this suddenly mean that all of the W2K vendors are conspiring to make Linux + TUX on Dell hardware look good? Or could it possibly mean that all research that Microsoft funded in the Mindcraft benchmarks is coming to fruition? My guess is that the folks at Microsoft are going to start to truly understand the power of release early, release often. While W2K has sat relatively still basking in its Mindcraft glory the Linux community has targetted the specific problems Linux had that caused it to do poorly in the Mindcraft benchmarks, and has rectified them.
Second of all this is a SPECweb benchmark. The web part of SPECweb would tend to indicate that it is a benchmark of http performance. If you read the spec you would notice that it specifically measures both static and dynamic http content serving. So while this does not necessarily mean that Linux is better than Windows 2000 it probably does mean that Linux + TUX is better than Windows 2000 + IIS (for the things measured by the benchmarks).
Your observation that most Internet facing sites don't have anywhere near this sort of bandwidth is certainly correct. However, my Intranet server does have this much bandwidth (not that I would appreciate it if it saturated this bandwidth). Besides, if you are going to let bandwidth be the limiting factor then it really doesn't matter what kind of web server you are using. A 486 running Apache will happily saturate a T1 with static content.
Not that any of this matters. The two most important features, to me anyway, of Linux are 1) Freedom, and 2) Cost. Linux wins hands-down if these are the factors that you value most.
From the results you must either conclude that Dell (and the rest of the WinTel vendors) are either trying to make Windows 2000 look bad, or you must conclude that Linux + TUX is going to make one heck of a compelling case as a web platform.
Either way it looks bad for Windows 2000 as a web server OS.
No, i dont think there is any such divide, and i think TUX does not contradict Unix concepts. CPUs get faster and protocols get more complex every day. Right now the HTTP protocol is common enough to be accelerated by kernel-space - just like the TCP/IP protocol got common enough 10-15 years ago to move into the kernel in many other OSs.
The question thus is not 'should we put HTTP into the kernel', but rather '*when* and *how* should we put HTTP into the kernel'. Think of this as an act of 'caching', the OS caches and should cache 'commonly used protocols'.
Where is the limit? There is no clear limit, but the limit is definitely being pushed outwards every day. HTTP is becoming a universal communication standard, with the emergence of XML the role of HTTP cannot be overhyped i think.
And the last but not least argument, if you dont need it, you can always turn CONFIG_TUX off.
Not very close, most busy sites don't have all static content.
. net
On a side note I think you should all visit this address and see what andover.net is running:
http://www.netcraft.com/whats/?host=www.andover
Solaris eh? Whats the front page of andover say?
"Leading the linux destination" great example you're setting there.
Only the State obtains its revenue by coercion. - Murray Rothbard
This is truly fantastic news, for years linux has held the lead over Windows in stability, usability, remote access, and bugfixes. Now it's poised to take the lead in the one area in which it was lacking... meaningless benchmarks.
Now the only advantage Win2K has over linux is a transparent start menu.
"This is not a company that appears to be bothered by ethical boundaries."
Attorney General Mike Hatch on Microsoft
Are we that low? Do we pull a Mindcraft whenever we want?
These systems, although very similar, are not identical. Different drive arrangements, different scsi controllers.
And, to boot, one is running IIS 5 and one is running Tux 1.0 (whatever that is...).
What does this prove about the individual Operating systems? ABSOLUTELY NOTHING!
It shows that operating system 'A' running web software 'B' on machine 'C' is faster than operating system 'X' running web software 'Y' on machine 'Z'.
What the hell is 'Tux 1.0?' Yes.. I could look it up. WHy not at least benchmark Apache, so at least you could say 'benchmark of most common intenret platform for each OS' or something..
I'd use Linux if Windows was 200% faster..
A faster Windows still locks me into it's
stupid upgrade treadmill... Benchmark
results are just statistics.. and as you
know, there are "Lies, Damn Lies, and Statistics".
You can't just jump up and down when Linux
beats Windows on a benchmark. Then you're
setting yourself up to hang your head when
Linux loses one every now and then (Mindcraft)..
In so doing, you're missing the point:
The speed, usability, or even stability of
free software is not the driving force behind
its existence, It's the FREEDOM!
On Independence Day, of all days, you lose
sight of this? I'm so tired of these benchmarks.
-- The Funk, The Whole Funk, And Nothing But The Funk
I'm absolutely loving reading all these comments right now. Its very amusing to me how all of you Linuz Zealots have jumped on this bandwagon proclaiming, "Linux is the greatest OS".
Yes, there were problems with the Mindcraft benchmarks - and yes there are problems with this one. Namely, what in gods name are they comparing? They certainly arent comparing operating systems - there are way too many differences in this case to do that objectively. Next time somebody runs benchmarks between the two OSes, please try to keep the following things in mind:
(1) USE THE SAME HARDWARE! I cannot stress this point enough. What you people may call minor differences may often have a MAJOR effect on the outcome of a benchmark such as this.
(2) Use the same Webserver Software. How in gods name can you blame or claim any of these benchmarks on the operating system? Both are using completely different HTTP servers (one which is isnt even publically available and shouldn't have been used = TUX). If you want a legitimate operating system benchmark and not and HTTP server benchmark - try to compare Win2k running Apache for NT and Linux running Apache. Otherwise climb off your high horses right now - these are webserver benchmarks NOT OS benchmarks. I for one will say that Apache for NT consumes a lot less memory than IIS 5.0 - though on my small intranet site I've yet to notice any speed difference.
(3) The results are unrealistic. What kind of server has 4 gigabytes of bandwith?
(4) Also - make a point to configure both servers equally - it seems to me you guys scrimped here and there on IIS configuration - I wonder why?
If the Linux world wants credibility - its time to grow up and earn it. You guys sure talk a great game - but when it comes down to the numbers you are either whining when you get trounced or creaming in your pants over benchmarks which are obviously flawed.
While Im on my soapbox - let me say this also: Its amazing how many of the news story's of this so-called "News for Nerds" site appear to be blantantly attacking Microsoft and promoting Linux. Its obvious that whatever sense of objectivity Slashdot once had (if ever) has long been lost to the horde of pre-pubescent teenagers who only have one goal: To get something for nothing.
So there you go - take it or leave it - I dont really care. You may either post a reply or email one to darkgamorck@home.com
Gam
I love idealists not because I am one, but because they make life bearable for pragmatists such as myself.
Suns, sure, but Macintoshes? I don't think I'm aware of anybody using Macs for even semi-serious webserving. Neither the OS (OSX is a different ballgame, granted) nor the hardware is designed for this kind of thing. Correct me if I'm wrong, please :)
As regards to the number of HTTP servers, maybe they just ran out of time and money to benchmark ten squillion different configuration, and chose the ones that they believed were in most common usage. Testing more of them would certainly be a good thing, though.
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
When the Mindcraft benchmarks came out, every Linux zealot screamed and cried that there were problems with the benchmark. They were right. Some sensible people pointed out something interesting I remember..
They said that when someone performs a benchmark in the future and it shows Linux outperforming Windows NT or 2000 by a sizeable margin, the Linux zealots will claim that THIS benchmark is the correct one and Mindcraft will be PROVEN wrong.
This post seems to me like exactly that behavior. Mindcraft doesn't tune Linux the right way and WinNT trounces it. Linux zealots scream bloody murder and inspect the process with a microscope. Someone else does a benchmark that shows Linux 3 times faster than Win 2k, and they are content that the Mindcraft fiasco has been avenged.
Take a look at yourselves. I'm not a Linux lover. I think it has a long long way to go before the mainstream starts to take it seriously. There are so many problems with it right now..installing programs, removing them, x windows interface complexity, simple text editors..the list goes on. Honestly, I don't think it will ever become mainstream - it will get replaced by something else that will before long.
I don't love Windows either. There are of course many problems with it. However, it's not the spawn of Satan and Linux is not the Great Hope or messiah.
Be objective, people. Please. You'll do your "cause" some good.
The two major distinctions between these benchmarks and the unjustly-maligned Mindcraft benchmark that were later confirmed by PC Labs:
1) these tests compare Win2k to Linux. By contrast, the Mindcraft study compared WinNT4.0 to Linux.
2) in the "Operating System" column of the Linux boxes, we see a revealing note:
Operating System: Red Hat Linux 6.2 Threaded Web Server Add-On
It seems as though RHAT has taken the trouble to render its TCP/IP stack into a multi-threaded model, rather than the forked model I understand it used to be. This was identified as the primary deficiency in the previous benchmarks.
At the time, Linux afficianados claimed that the superiority would be short lived. Assuming these stats are otherwise legit, it seems as though they were right, and in such a brief period of time as well. I'm impressed! Keep pumping out impressive turn-arounds like this one, and very soon commercial entities will have to give open source its just props as a development model.
I am slightly curious whether this "web server add-on" is available to consumers, and also whether it is a fully-featured web server. If not, and this is just a hack, that might cast a pall of illegitimacy. Anyone have the inside scoop?
-konstant
Yes! We are all individuals! I'm not!
-konstant
Yes! We are all individuals! I'm not!
1) the maximum filesize in the SPECweb99 benchmark is 900kb, this is why there is a 1MB limit set. Your claim that there are 1MB objects in the benchmark is false.
2) the CGI executable is mandated by the SPECweb99 Run Rules. A process must be created and destroyed. But the total amount of CGI requests is 0.1%! All the other 99.9% of the workload was handled with IIS 'low application priority' modules, which is a DLL loaded into IIS's address space, not a .EXE.
3) the IIS object cache was set to 2GB (not 2MB). It's set to 2GB because Windows 2000 + IIS has a serious limitation, threads (such as the IIS threads) can only address 2GB. This is a design flaw in Windows 2000, which hunts them in the enterprise now.
4) are you really seriously promoting the idea that the top 4 PC OEMs (Dell, IBM, Compaq, HP) and Microsoft did not tune IIS to the max and somehow conspired in making Linux+TUX numbers look good?
Fact is, the only reason why the TUX result was compared to the same Dell system is that the Dell system also happened to have the fastest Windows 2000 results. Your whole line of argumentation is obviously flawed if you compare IBM's similar Windows 2000 SPECweb99 result to the TUX result.
The problem here is that these figures aren't aimed at freedom lovers - most of us take benchmarks with a pillar of salt anyhow. They are aimed at business users. Most business users don't play politics - they want results, and they want comments from the press they can show to their managers.
So while the benchmarks don't directly impact on us, their influence over business computing does give benchmarks some significance.
tangent - art and creation are a higher purpose
postmoderncore - art and creation are a higher purpose
These numbers seem hugely high to me. I mean... 4,200 simultaneous connections at 350kbit/sec is around 1.5Gbit/s. To do that you'd need some fairly serious NIC's. A closer inspection of the test setup reveals the server was pushing 4 networks through 4 gigabit alteon network cards.
:)
Reality check guys. Does anyone have 4 gig of external connectivity? And doesn't 4,200 simultaneous connections of 350kbit/sec each represent, like, Yahoo? (without doing the sums)
This would also seem to spurn a more serious debate in terms of web performance testing. If we can get a single server to munge through this kind of quantity of throughput - why have clusters of servers at all? Clearly real world servers perform nothing like as well as this, and we need to have a better look at why.
Dave
I write a blog now, you should be afraid.
Suns, sure, but Macintoshes? I don't think I'm aware of anybody using Macs for even semi-serious webserving. Neither the OS (OSX is a different ballgame, granted) nor the hardware is designed for this kind of thing. Correct me if I'm wrong, please :)
You mean besides the military. A G4 running OS X Server is nothing to sneeze at, and if memory serves correctly, web serving on the mac using WebObjects is a pretty sweet combo... If you are going to include commodity x86's then you should include macs...
course if you're running linux PPC, then you can run Apache on a G4 and really rock and roll......
~ a low user id is no indication I have a clue what I'm talking about.
Comment removed based on user account deletion