Slashdot Mirror
Mindcraft Study Validated
!ErrorBookmarkNotDefined writes "Another study has appeared validating the
Mindcraft comparison of Linux and NT. This time,
PC week benchmarked Solaris, Linux and NT. Using a monster machine, NT handily defeated Linux. The study found fault with Apache, mostly. (For low-end machines, Linux would easily beat all comers; but how far along is Linux in the highend market?) "
Posted by ZeeC:
I love Linux I run an ISP with it. I am also working for a company that wants to relese linux on their high end Intel boxs (Hint, they are currently supporting NT and do UNIX on relly colorful boxs), anyways I am having a tough time finding drivers and HOWTO's on doing high end stuff with Linux. like Fiber channel scsi, Gigabit eathernet, heck even handling more then 9 scsi drives at a time. The OS needs to grow out of the "Keep a 386 useful" to a higher level now.
It shows that NT needs a behemoth computer to run well, whereas it is targeted at the low-end market
which does not use behemoth computers. NT needs the hardware to run well.
Believe me, NT does not run well, for instance, on my 450 Mhz PIII with 128 megs of RAM, all things considered.
Linux has a history of keeping abreast with reality. When nearly everyone has a four or eight CPU monster, then Linux will run like hell on them, and so will applications such as Apache, etc.
When everyone had a 386, Linux ran well on a 386. When everyone had a 486, Linux ran well on that (and still does!). Linux is made to fit a need,
not to participate in olympic events.
I have an 8MB 486 at work on which I need Linux to run well. It does. In all likelihood, NT 4 won't even boot on such a machine. The machine has no keyboard or monitor, yet I can completely administer and upgrade it. NT would be useless on it since it requires a graphical display, mouse and keyboard for administration.
For those interested in IIS and benchmarking, please take some time to read this rather long article. It's from a friend of mine who was the manager of the O'Reilly WebSite Pro development team. Some of the key issues were that MS made changes to the Winsock API specifically for IIS (AcceptEx, TransmitFile, Fibers and IOCompletionPort). Should Linux do this to make Apache/Zeus faster just for benchmarks, when really it does fine in the real world? No. Of course not.
,hacker Perl another Just)'
The other interesting point is the fact that ZD came up with the IIS benchmarks specifically to show how good IIS is. Such things as fitting the test harness in the cache, and only doing ISAPI dll's for dynamic content (vs CGI on other servers).
There are lies, damn lies, and ZD benchmarks. I'll use what works, and live happy in the fact that I won't have to reboot my server this year.
Matt.
perl -e 'print scalar reverse q(\)-:
Matt. Want XML + Apache + Stylesheets? Get AxKit.
These benchmarks weren't a complete loss to all of those. Hidden in those benchmarks was a rather interesting admission found in this story on Linux Today (note, I don't work for them, but they do have some good stories sometimes):
http://linuxtoday.com/stories/5906.html
This story reveals that Linux with Samba achieved 197Mbps, which was significantly higher than the Mindcraft benchmarks, severely invalidating the original Mindcraft benchmarks. Also, Apache did MUCH better on these benchmarks than on the original Mindcraft tests.
The article also shows that NT achieved only 150Mbps against NT clients, 31% slower than Linux. In tests with 60 clients, Windows NT managed only 110Mbbps throughput, compared with 183Mbps for Samba.
So, we got something out of these benchmarks. Linux serves Samba to NT clients 31% faster than NT on high end hardware!
Now, if we only tested IIS against Zeus to make a more fair benchmark for static tests, Linux wouldn't look so bad after all overall.
I don't see how these new benchmarks validate Mindcraft at all.
It appears that most Linuxheads have finally come around to admit that Linux doesn't perform well as a server. Yet.
But it's pretty well acknowledged that NetBSD kicks ass in that department.
Time for Linux groupies to take the blinders off. Quit getting your shorts in a knot about the unfair Mindcruft tests, quit trying to pit Linux against NT in server applications...
...and start *heavily* promoting NetBSD as the ultimate server solution. Mob the media with it.
As long as you play by Microsoft rules, you lose by Microsoft rules. And fiercely protecting one's "turf" is a Microsoft rule.
Step out of that box. Quit promoing Linux as the be-all and end-all. Promo NetBSD as *the most appropriate solution* to server needs. Promo BeOS as *the most appropriate solution* to multimedia needs. And so on.
This tactic will emphasize to the media that people should make active choices re: their OS needs; emphasize that Windows is not the most appropriate OS for most cases; and emphasize that the Linux community plays big and puts the user first and foremost.
It's a no-lose situation. Choice is the ultimate goal.
--
Don't like it? Respond with words, not karma.
Every day, ./ chooses a web server on the internet at random. It then presents a link to that server somewhere on the start page, calling it the "benchmark link" or whatever (so people know what it's for). It is then ./-ed by the readers, and at the same time monitored for its uptime. Its server OS and software are determined (if possible, should be) and as the days pass, statistics are put together for the average time a server OS lasted under that strain.
Not entirely serious, but a good "real world" benchmark, and I'd enjoy that.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Well, I work for a very large worldwide online/television news organization. Our main web site gets about 130,000 hits/minute on a normal day. That comes out to about 2200 hits/sec. This doesn't include any of our partners, which each generate a good bit of traffic. We have 6 T3s and an OC3 for our bandwidth, and we run multiple servers to balance the load.
What do we run? Netscape web servers on Solaris. When big news like the Starr report came out, all the servers at MSNBC running NT came crashing down under the load, but we didn't. That's what UNIX (and Linux) are about, reliability. Apache can be performance-tuned if you need it to be fast (Netscape's server is the same code base as Apache), but for most of us it's fine as-is. I bet that Microsoft.com doesn't get 2200 hits/sec.
- Vincit qui patitur.
Admit it folks, if the tables were turned and Linux was beating NT in these benchmarks, we wouldn't be hearing all these excuses about the relevance of the benchmark.
Not that this is a new thing, since it happens every single time that someone shows that Linux might not be the best solution for everything under the sun. Whether it's lack of certain quality applications available on other OSes, or poor performance by Linux on a certain benchmark, we can always be assured of hearing the shriek of, "But nobody needs to do that anyway!"
Uh-huh.
And no, Linux doesn't actually suck at this current benchmark, but it definitely doesn't measure up to NT or Solaris in it.
Cheers,
ZicoKnows@hotmail.com
Slashdot Realist
...unless you make it matter. Remember what Linus said at his keynote address? He said: let's focus on the low end server and desktop market. It may sound "sexy" but we're not focusing on high end machines with bunches of processors.
These are two totally different areas, and Linux was always designed with the lower end in mind. How convenient then for them to do all these tests on huge computers nobody would actually use for a web server, unless they run one of the top 100 sites on the internet! Not to mention the fact that this is more of an apache benchmark than a linux one.
If you run a huge smp machine and want to squeeze every last drop of speed out of it, you probably won't run linux anyway. It's not that linux isn't "good enough", it is designed for a different purpose. For a job like that, you would want Solaris or FreeBSD (still not NT)
NT has its own design purposes, which are different from any unix type system. There are two main design goals I can see in NT: 1. Be easy for even an idiot to maintain, since most of the time all he will have to do to is follow wizards or reboot the machine. 2. Be monolithic and slow, but for benchmarking purposes, have a way for those few people who know the OS inside and out to tweak it to insane levels for one or two particular services at the expense of stability and resembalance to "real life" situations.
Vidi, Vici, Veni
If you look at the numbers and not just the graphs, aren't these numbers just a little ridiculous? I'm not real sure exactly what is meant by these numbers, but here's my reading of them. I mean at 60 clients linux has 2000 requests per sec, solaris has 5000, and NT has 4000. That means each client is requesting ~30 items/sec for linux, ~83 items/sec for solaris, and ~66 for NT. That's either really quick clicking, or really complicated pages. Then there's the other test, which is more understandable, throughput vs client load. Even so its not bad. Look 300 Mb/sec vs 200 Mb/sec for 60 clients is not bad. Really each client gets about 5Mb/sec or 3Mb/sec. For most uses this is probably fine. I expext that multi-Mb document to take a while to open. Even from my own hard drive. Bandwidth is more of a concern I'd imagine at this point.
That said let's actually look at the graph for a minute. On the WebBench test 60 clients is about the point it seems that NT levels off, can't really tell they cut the graph off. Yet the quote below has you believing different "Solaris and NT had plenty of CPU cycles to spare." And linux wasn't exactly losing any ground at that point - ok a little lower but not much. It seemed to hit a stable point. What about more clients? And then there's the Netbench graph. I mean look at NT plumet. Linux hits 16 clients and levels off at 200 Mbps. Nt hits 48 clients with 350Mbps then falls down to 300Mbps by adding 12 clients. Linux added 44 cleints and lost maybe 50Mbps. To me this looks like linux acts like a marathon runner, getting to a distance then setting cruise and holding steady. NT on the other hand is like s sprinter, burning itself out and working hard quickly but won't last real long. Yeah the sprinter will beat the marathoner in a 1 - 2 mile race, but look out for that 5 - 10 - 26.2 mile race.
My point is you get fine, predictable preformance, regardless of the amount of work asked of linux. Meanwhile NT seems fine for small amounts but the more you ask the less likely you are to get it. I want to see the benchmarks with higher numbers. I'd expect linux to hold around its same mark, and NT to fall steadily. Why 60? Why not 100? 100 is even (ish), why not 50? 60 just seems like an odd number.
-cpd
NT is far less mature than the Unix family, of which Linux is a member. M$ foolishly ignored 30 years of research and accumulated wisdom. As a result, they've been repeating all the old mistakes.
Most of NT (and other M$ code) was written by lower echelon programers, under the direction of computer scientists and managers. Many of them had only recently graduated from MSD training classes. In generaly they were operating under marketing imposed time constraints. This shows in the quality of the product.
If you want proof, try working with the IP routing table metrics under NT, or look at their publicly released code, ie the frontpage extentions for apache. Also look at a security model that requires everyone to buy third party virus scanners.
In contrast, most of Linux was based on an established tradition. Most of the major holes were already known. It was written by people who cared about the quality of their code. They loved programming, and their personal reputations were at stake. Then that code was reviewed publicly, and contributions were fed back to the author.
I forget who said "If I have seen far, it is by standing on the shoulders of giants". M$ forgot it was ever said.
The pure humor of it all...
From behind the scenes of www.microsoft.com
Hardware
Six internal Ethernets provide 100 megabits of capacity each
2 OCI2s provide 1.2 gigabits of capacity to the Internet
Runs on Compaq Proliant 5000s and 5500s, with 4 Pentium Pro processors and 512 megabytes (MB) of RAM each.
Software
Microsoft Windows NT Server 4.0
Microsoft Internet Information Server 4.0 (IIS)Microsoft Index Server 2.0
Microsoft SQL Server 7
Other Microsoft tools and applications
Powerful Solutions
www.microsoft.com started out as a single box beneath a developer's desk in 1994, handling about a million hits a
day. That seems almost laughable now. A sleek data center in Redmond, Wash., receives more than 228 million
hits a day while data centers in London and Tokyo shoulder the international load of about 12 million daily hits.
How has the site handled its explosive growth while keeping its hardware to a minimum? How does it administer
one of the largest databases in the world? How does it manage the challenges of a decentralized publishing
environment? How does it come close to achieving 100 percent site availability? The answers lie in the
strength of its software, according to site architects. The whole shebang runs on Microsoft Windows NT 4.0,
IIS 4.0, and SQL 7.0. "Our site showcases Microsoft technology," says systems operations manager Todd
Weeks. "We prove every day that we can run one of the largest sites in the world 100 percent off of
Microsoft technology."
The Challenge
Not only is www.microsoft.com an enormous site with hundreds of thousands of pages of content. Not only
does it receive millions of hits a day. Not only has its growth been unrelenting. Those are some of the
easy challenges, site architects say. One of the most interesting challenges is that www.microsoft.com
functions within a decentralized publishing environment. More than 300 writers and developers working in more
than 51 locations around the world provide information for the site. These content providers are able to update
their sites within the www.microsoft.com umbrella as often as eight times a day. In fact, 5 percent to 6
percent of the site is updated every day. The complexity of that publishing environment is daunting
when you consider that each of the 29 content servers in Redmond contains the nearly 300,000 pages of
information that comprise www.microsoft.com. But the end result is that the information on www.microsoft.com
is as current and up-to-date as possible. A team of about eight people staffs three shifts around the clock
to ensure www.microsoft.com stays up and running 24 hours a day, seven days a week. "Our goal is to make the
site available to users 99.8 percent of the time," Weeks says. So how do we reach that lofty goal of 99.8
percent availability? (The 0.2 percent down time is required for routine maintenance.)
First, the Hardware
The physical architecture behind www.microsoft.com seems surprisingly modest. Twenty-nine servers host
general Web content; 25 servers host SQL, 6 respond to site searches; 3 service download requests along
with another 30 in distributed data centers; and 3 host FTP content. Additional servers overseas handle
some of the international load.
Did you count all of that? That's 96 Compaq Reliant 5000s & 5500s (Quad Pentium Pro boxes with 512Mb RAM) running
www.microsoft.com using NT, IIS, Index Server, and SQL Server.
Standard
This machine is a P6/200 with 1GB of memory & 1/2 terabyte of RAID 5.
The operating system is FreeBSD. Should you wish to get your own copy of
FreeBSD, see the pub/FreeBSD directory or visit http://www.freebsd.org
for more information. FreeBSD on CDROM can be ordered using the WEB at
http://www.cdrom.com/titles/os/freebsd.htm or by sending email to
orders@cdrom.com.
Now, which site do you suppose has set more download records?
--- A Jesus Fish eating a Darwin Fish only proves Darwin's point.
How many times does the "Linux Community Inc." need to tell these people that Apache wasn't ment for speed!? Why is Apache designated as the One True web server? Benchmarking static Apache vs. static IIS is pointless. Any programmer worth his salt could cook up a few dozen lines of code that would outperform both servers on pure static content.
:)
They should benchmark how many dynamic perl generated pages NT can vomit out
--- A Jesus Fish eating a Darwin Fish only proves Darwin's point.
Somebody please help me out here, because I evidently just don't get it.
Why is SPEED the overwhelming issue? IMHO, there is so much more involved in choosing a server OS. Do we really need to measure the number of milliseconds it takes to rename a file on the server? Isn't that a little silly?
Picking a hardware/operating system configuration is not a drag race. You care about cost. You care about uptime. You care about security. You care about support.
The skills of your existing personnel are important too. If you have a staff of freshly-certified MSCEs, it's very unlikely that you will use a Unix-like system. OTOH, if your network admins love Unix, they will want to work in a familiar environment.
In the end, speed is not really the same thing as "performance". Benchmarks like these provide nice soundbites for the winner (whoever it may be). They also improve magazine sales and web traffic for the publications. If you choose to commit your organization to an operating system based on them, however, then maybe you deserve what you get.
As my mom used to say, "When that lawnmower cuts off your feet, don't come running to me."
Save the whales. Feed the hungry. Free the mallocs.
If the question is "how many people deploy low end solutions?", then it is important to note that the situation being tested has no relation to the real world. If someone needed to serve thousands of static pages per second, they would be out of their gourd to select a quad Intel box in the first place, regardless of OS. Better to have a small, cheaper farm of lesser computers to do this job.
Given that one has chosen Linux on any hardware platform for this task, one would also be out of one's gourd to choose Apache. Apache engineers will tell you this. Apache is built for flexibility at the expense of performance. Thus, the simpler the job, the slower Apache is, on any platform, for the job.
If you are comparing OS speeds for Web serving, either use the same Web server on both sides or use optimal Web servers on each platform. Apache engineers will be the first to admit that other Web servers can outperform Apache, on any platform, for this test.
But for high volume dynamically generated content, for example, or commerce, or databases, NT is more mature and benefits from being developed by engineers rather than hackers. DEC, from whence Cutler came, are very serious about this.
Maturity is relative. NT has more runtime hours than Linux, so there has been a longer time to detect bugs. Linux, due to its huge potential developer base, may well have more developer hours invested in it. It also has more debugging hours invested in it, because most Linux users are potential debuggers. When NT fails, one just reboots. When Linux fails, one often looks at the messages file (or has a sysadmin do the same) and track it through the tech support infrastructures.
The specification for Linux draws heavily from Unix, which is an incredibly mature model for high-volume computing. Most of the specification for Linux predates DOS, never mind any flavor of Windows.
Linux is developed by engineers at their best. The best engineers are hackers, really; they're the ones who build software for the love of building software. And Linux hackers contribute with only their best.
If you are building a commercial software product, you are expected to put 5-6 days of work into the project each week. You cannot maintain top productivity, top quality, on that sort of a schedule. Employers understand this, and they deal with it. The code produced in a commercial setting tends to be "good enough".
When someone contributes software, the same drive that causes them to want to do something like this for no money causes them to work at peak performance. It also causes them to work at precisely what they're good at. Sure, they may only put in 100 hours in 3 months, but you will get their best 100 hours, easily worth 3-400 average engineering hours. Commercial software is produced by marathon. Free software is performed by relay sprint.
Besides, if commercial OSs are superior by virtue of being developed by engineers rather than hackers (that is, by virtue of being commercial), then why are shops like Sun putting so much effort and money into Linux? Methinks that the Solaris guys see something in Linux that they envy, and I don't think that it's just the salaries.
However, for midrange work, linux simply isn't up to par yet. I seem to recall Linus himself stating that he believed OS design was well understood by the 1970s, and he considers microkernels to be "stupid", plan9 to be "stupid" etc etc. Whatever you think of Linus' talents as a kernel hacker, the fact remains that Linux works. It works in commercial production environments. Sysadmins have been disobeying management by deploying Linux where NT was requested--and they've been doing it for years. This isn't politics.
A sysadmin has one overriding virtue: laziness. Larry Wall gave us the prototype: more on this in the Camel book. They want to do the job once, they want to do the job right, and they want to forget the whole affair afterwards. These sysadmins have been putting Linux in back because they do their jobs and are easy to handle--and because the performance boost gives them fewer boxes to administer (and fewer hassles with acquisition budgets).
While he is undoubtedly a highly talented programmer, I think that there are engineers in the world who are at least, if not more, skilled working for Sun, CMU, Microsoft, DEC and suchlike whose work has proved Linus to be very wrong. And as such, linux is crippled.
Linus doesn't have to be the best programmer on the planet. In fact, he needs never write another line of code. There probably are better kernel hackers writing code for their respective companies--and also writing code for Linux.
Linux isn't an optimal OS. There are places that it is the best one out there, and other places where it does poorly. Like every OS, however, it evolves. Its openness simply lets it evolve faster than the competition. Per Darwin, evolve or die.
It doesn't matter how wrong Linus is in his coding, because it does well enough for commercial use today. Maybe microkernels have overriding advantages. GNU has a microkernel OS (GNU Hurd) in beta or GA by now. If it outperforms Linux, it will only be a matter of time before somebody crosspollenates them. Then RMS will have a better case to call it GNU/Linux. Whenever somebody finds a major improvement one can make to an OS, somebody else will port that improvement to Linux. Perhaps every line of Linus' original code will be optimized out.
Linux is far from crippled. By my lights, it is the first OS to sprout wings.
--The basis of all love is respect
I realise that the webserver benchmark is just a small part of the tests, but practically all of the benchmarks make claims like "NT beats Linux" and substantiate that by giving a lot of numbers on how much webpages were served by a Linux server and a NT server.
Apache runs on a whole bunch of other platforms, even on MS-Windows. Probably even NT... Wouldn't it make more sense to make claims like "Apache on NT beats Apache on Linux"?
That wouldn't prove the superiority of NT over Linux either, but it would IMHO make just a little bit more sense...
The same goes for Samba: Samba runs on Linux but also on other systems.
All these tests only test NT-running-some-software versus one-of-many-Linux-distros-running-other-software and then make claims like "NT kicks Linux' ass".
"Linux" is just the kernel... or have I gotten things completely wrong?
Benchmarkers should at least prove that bad scoring is caused by Linux (kernel) and not a program they're running on top of that!
If a webserver running Apache on freeBSD is doing better than Apache on Linux, that would be an indication of shortcomings in the kernel (although some people may dispute that as well).
Ah well, I never really cared for benchmarks anyway...
190,080,000
Thats what 2200 hits/sec gets you. You'll be doing 190 million hits/day. Pretty damn impressive. I'd like to work for you, considering the monster bw you'll have.
I'd basically ignore any current benchmarks because they're based of versions of linux that have known issues.
You're also comparing a multi-process server, which works faster at lower loads, to a multi-threaded server, which scales better, although might not/does not return documents back faster.
I'd like to see the avg connection times on these things.
Ah, but the PC Week test was just static documents! Redhat 6.0 comes with an RPM for Squid, but instead of installing that, they use Apache and then gripe about how expensive it is to fork for each request.
It's unclear to me what use there is for a web server that is eating bandwidth about the way ftp.cdrom.com does, anyway. That doesn't strike me as a typical "enterprise application". That part of the benchmark is obviously contrived.
Java: the COBOL of the new millenium.
The numbers get even more interesting when comparing the results of NetWare with and without Opp Locks. When we turned on Opp Locks, NetWare's overall performance improved by about 40 percent.
However, this gain is deceiving. With Opp Locks enabled, almost every operation in NetWare actually slows by 25 percent. The exception is file write operations, which are faster by 300 percent. Because writing files takes up almost 40 percent of the NetBench test, it's no wonder we saw a huge overall performance boost in our results.
These people haven't a clue what they are benchmarking. Opportunistic locks allow the client to do whatever it likes to a file (or regions thereof) without synchronizing with the server. Of course write speed increases; the network isn't involved anymore! You haven't increased server performance one whit, but rather prevented more than one client from opening the file for writing at the same time.
Java: the COBOL of the new millenium.