Slashdot Mirror
ESR and the MindCraft Fiasco
Microsoft's latest FUD (Fear, Uncertainty and Doubt) tactic may be backfiring.
A 21 April ITWeb story reported results by a benchmarking shop called Mindcraft that supposedly showed NT to be faster than Linux at SMB and Web service. The story also claimed that technical support for tuning the Linux system had been impossible to find.
Previous independent benchmarks (such as "Microsoft Windows NT Server 4.0 versus UNIX") have found Linux and other Unixes to be dramatically faster and more efficient than NT, and independent observers (beginning with a celebrated InfoWorld article in 1998) have lauded the Linux community's responsiveness to support problems. Linux fans smelled a rat somewhere (uttering responses typfied by "Mindcraft Reality Check"), and amidst the ensuing storm of protest some interesting facts came to light.
- The benchmark had been paid for by Microsoft. The Mindcraft press release failed to mention this fact.
- Mindcraft did in fact get a useful answer to its request for help tuning the Linux system. But they did not answer the request for more information, neither did they follow the tuning suggestions given Also, they forged the reply email address to conceal themselves -- the connection was made after the fact by a Usenetter who noticed that the unusual machine configuration described in the request exactly matched that of the test system in the Mindcraft results.
- Red Hat, the Linux distributor Mindcraft says it asked for help, reports that it got one phone call from them on the installation-help line, which isn't supposed to answer post-installation questions about things like advanced server tuning. Evidently Mindcraft's efforts to get help tuning the system were feeble -- at best incompetent, at worst cynical gestures.
- An entertainingly-written article by the head of the development team for Samba (one of the key pieces of Linux software involved in the benchmark) described how Mindcraft could have done a better job of tuning. The article revealed that one of Mindcraft's Samba tweaks had the effect of slowing their Linux down quite drastically.
- Another Usenet article independently pointed out that Mindcraft had deliberately chosen a logging format that imposed a lot of overhead on Apache (the web sever used for the Linux tests).
So far, so sordid -- a fairly standard tale of Microsoft paying to get exactly the FUD it wants from a nominally independent third party. But the story took a strange turn today (22 Mar) when Microsoft spokesperson Ian Hatton effectively admitted [8] that the test had been rigged! "A very highly-tuned NT server" Mr. Hatton said "was pitted against a very poorly tuned Linux server".
He then attempted to spin the whole episode around by complaining that Microsoft and its PR company had received "malicious and obscene" email from Linux fans and slamming this supposed "unprofessionalism". One wonders if Hatton believes it would be "unprofessional" to address strong language to a burglar caught in the act of nipping the family silver.
In any case, Microsoft's underhanded tactics seem (as with its clumsy "astroturf" campaign against the DOJ lawsuit) likely to come back to haunt it. The trade press had largely greeted the Mindcraft results with yawns and skepticism even before Hatton's admission. And it's hard to see how Microsoft will be able to credibly quote anti-Linux benchmarks in the future after this fiasco.
I am worried by the general attitude of the community towards the Mindcraft test. IMHO it would be much more productive if we said:
"Thanx Mindcraft for working in the Free|Open spirit, test our software for us, and report problems back". The study shows at least that it is not difficult to get sub-optimal performance from a Linux box. Since nobody got such a monster at home, the programs were never tested on it before, not to mention tuned or even developed for such a machine. I sincerely hope that Kernel, Apache and Samba hackers will take this opportunity to improve their software for high-end servers:
- SMP
- big-memory support
- better|alternative defaults
- optimization guides
Demonstrate performance on SPECweb96 on a 1, 2 and 4-way Xeon. All vendors can tune their solution to the max. Presto: even playing field!
Compare against existing NT4, Solaris, and NT2000 benchmarks on similar 4-way Xeons.
http://www.specbench.org/osg/web96/
--G
there was something posted on /. a couple days ago where someone did a test of a poorly tuned linux server against a poorly tuned NT server (the controversial tests here were a poorly tuned linux vs. well-tuned NT).
linux won.
I couldn't agree more. We have been using a mainframe for almost 20 years, and just recently the decision came down to replace it with SAP running on some heavy IBM servers. The only problem is we got about 5 seconds into production of one section of the organization and the whole thing screeched to a halt.
Now we'll likely go back to SAP running on a newer Mainframe.
Sure we use Novell, NT and Linux for desktop authentication servers, passthrough stuff, web etc. But for anything vitally important, it's running on a Mainframe, or really, incredibly large servers that can not be classed as PC's.
PC's didn't win, they're just more visible.
Linus uses a quad xeon at home (don't remember the exact RAM) and he's most interested in SMP...so I would think developer support is there.....
Dave
The important things to remember are:
the NT system will never get faster, modulo new releases from Microsoft. (And those releases tend to be infrequent, bug-ridden, and bloated.)
the Linux system will never get slower.
the Linux system will get faster as developers get access to such high end systems.
Finally, 4-way Xeon servers aren't exactly a consumer item. Any department that will buy one probably has an annual budget well over $1M. If I were that manager and I wanted a screaming server, I would buy at least *five*.
Two would be deployed (for redundancy).
One would be used for final testing of any changes (and would be a hot spare).
One would be loaned to the Apache team
One would be loaned to the Samba team
The ones loaned to the Apache and Samba teams would be carried on the books as spares (and the terms of the loan would spell this out)... and internally the cost of the systems would be offset by the advertising benefits of donating hardware to those popular projects.
Demonstrate performance on SPECweb96 on a 1, 2 and 4-way Xeon. All vendors can tune their solution to the max. Presto: even playing field!
Compare against existing NT4, Solaris, and NT2000 benchmarks on similar 4-way Xeons.
http://www.specbench.org/osg/web96/
--G
Halloween Letter LXII or maybe I Quit IV
This Zippo character is dumber and pettier than Raymond is (no easy task, IMHO
On the OTHER hand, as has been mentioned above by several posters, the actual purpose of Raymond's post here is to put all the valid claims of the debunkers together in one spot, associated with a name that trade-press editors have heard of. The point is publicity, along with what passes for credibility in the trade press. Raymond can provide that. He can get these statements out of the advocacy newsgroups and into column inches in IT publications. Doing so is important. IMHO he should be more cool, calm, and collected when he talks specifically about Microsoft, but on the whole this post is just what we all like best to see him do: Advocate free software armed with facts and logic. This post has no silly infighting with competing OSS Rock Stars and no bullshit. I like it. From where I sit, Raymond's earned back some of the respect that he lost (from me, anyway; YMMV) over the last few months.
Zippo just doesn't get it, that's all.
If anything, they are nothing more than a thin shell for Microsoft PR. It would be a great story for some good investigative IT reporter to check out the background and funding of this supposedly 'independent' lab. I think it's clear from their reports who they're actually working on behalf of. Witness their home page:
"Windows NT Server Outperforms Linux"
"What's Wrong with the Quicktime Plug-In and How to Fix It"
"White paper compares Window NT Server 4.0 and NetWare 5 as file servers"
"White paper compares Windows NT Server and Solaris Web Servers"
In all cases, Windows comes out on top, and in all cases, Microsoft funded these supposedly 'independent' reports. Gee, wonder what's going on here? I also wonder how much they paid to have their findings published on Mindcraft's home page. Some clever reporter from ZDNet (how about Dvorak?) could make this into a fantastic expose.
Blah, blah blah, blah blah, blah. Who cares about all these thousands of comments?
The only thing that should be of interest is the performance numbers on the same four-way Xeon/RAID system with both platforms tuned to the gills.
I dont know if there is a way for the client to to determine which server it is talking to, but if it is then a spider could be written to accuratly count Apache vs IIS servers. Anyone ?
Perhaps ESR is referring to a different document than I think he is, but the results that mindcraft posted at http://www.mindcraft.com/whit epapers/nts4rhlinux.html do claim that Microsoft sponsored the tests.
Has anyone actually sent DOJ the Mindcrap report and the rebuttals from the Linux community?
Not only is M$ guilty of fraud, but Mindcrap is also an accomplice. Last I checked, fraud is a criminal offense in the US.
Even if no criminal charges happen, I think that Mindcrap is finished as a software testing house. Their credibility is nonexistant.
We use an SP2 here at work, and My experiences are the opposite. Soid as a rock and pretty darn fast. I've never used the SP2 down at OSU, so I can't comment on that. (if that's the one your talking about)
Wow, totally the opposite from our SP.. we have the same configuration serving Lotus Notes databases/email for 14,000 users and it hardly breaks a sweat doing it.. It's one of the fastest machines I've ever worked on, and the "SP Switch" is blazingly fast.. it can send 4 100 mb files at the same time in about one second between nodes.. works for me.
Sure I can read the +1,500 returns on Deja news from server tuning but nobody wants too.
How about a performance tuning for all servers with subsections.
1. SAMBA
A. RAM
B. smb.conf
2. NFS
3. APACHE
4. Hardware
A. Supported RAIDs
B. Supported NICs
a. why you should/shouldn't pick chipset X
You are clearly not a manager with this sort of budget, or you would realize how unrealistic this situtation is.
5 machines to effectively have 1 server??
Sounds like an IT manager's wet dream.
to watch Linux users jump up and down when they see NT vs. Linux tests and see MS funding them.
MS is trying to sell its operating system while Linux users are thinking "my car is bigger/faster/better than your car". Who cares if NT kicks Linux's ass? I use what works. If Linux is currently working.. I'm using it. If NT is currently working then thats what I use. Quit being a marketting droid and just use the software. There is no way to compare two completely different operating systems. It'd be damn near impossible to create a benchmark program that runs the exact same speed on all platforms if one platform is slightly faster than the other.
We also use a proprietary dbms that searches roughly 2 terabytes of data. The IO performance is bad, and if CPU usage is high, IO drops to about one read a minute.
I'm unfamiliar with the SPs, but this sounds like it is very blatantly the software - if the industry reports good results with the hardware, and you're not running the same software as the industry, if you think the hardware's crap, it's probably your software. Furthermore, I've frequently found that you can substitute 'buggy as heck' with the word proprietary, and it usually works. Occasionally I find decent proprietary software, but it's as rare as decent non-proprietary software, and generally speaking, once you start using proprietary, you can't get out, the management wants to sink with their investment.
My experience with mainframes has been, by and large, very favorable. However, I'll point out that most of that has been on some ancient Sequent DYNIX boxes, so I can't really speak for IBM hardware either way.
About the same as Linux vs DOS look like on a 286.
Such a test is pretty useless. It won't convince anyone to go Linux in their company, no more than it would when you port Linux to a toaster. Companies buy using either "we have so much to spend, what's the best we can buy for it?" or "we need this and this, what is the best deal we can get?". A 486 will seldom be an option.
I disagree. Perhaps 486 is extreme, but at least they should test low-end pentiums. Many companies have at least a few older computers lying around doing nothing. If you have 5 pentium 90s lying around, install Linux or BSD with Apache and load share. You get a much more reliable server (one box goes down, you still got 4) that can handle large loads, all for the low low price of free! Of course Samba is a slightly different story, but they did benchmark Apache, too.
This is not unrealistic. My best friend works at Blue Cross and they have literally hundreds of such machines in the "grave yard" just waiting in case they can use them for something. The problem is that they are currently a (mostly) NT shop. My friend has convinced them to take a look at Linux, though. They currently have setup a test server on one of those older boxes.
Just maybe this is what Microsoft was trying to get at in the first place. Linux is not the only threat to them at this time, the OSS movement as a whole is a threat, and the reaction by its supporters discredits us all. If we are made to look like a bunch of kids raising a ruckus when things don't go our way, then Microsoft has gained something.
I'd love to see some more information in the Slashdot headlines and elsewhere about the professionalism of Open Source users. We need to shape the minds of the young Open Source crowd in a positive manner so that there is organization and cooperation before individual users start screaming back incoherantly at people like Mindcraft. (Yes, death threats, virus threats, hack threats etc.)
Imagine with me for a minute the effect we would have had if not one user had lashed out in anger at them or posted irrational comments in various public areas. Microsoft would have been exposed for the childish tactics it is using against us. It would have been obvious.
Now we look like two corrupt political candidates. IT shops all around the world are trying to pick between the lesser of two evils instead of being able to pick a sure winner. It doesn't have to be that way.
-FS Left the password on the other computer.
He hasn't added anything new. His only intention is obvious, to aggrandize himself yet another time. All he's done is show us that he is willing to feed at the table after everyone else has finished and left. Pathetic.
This benchmark won't work. If you tweak NT down, it probably won't boot.
Last time i used a low-end Sun (SparcStation 5/85Hz) it was about the speed of mid-level PC (486dx5-100) at about 10x the price.
"Testing companies serve clients with the same service a street-lamp provides a drunk."
forgot where that's from...
Some Linux org ,say Red Hat , should _hire_ the Mind Craft folks to run the bench again
on a properly tuned Linux box. Mind Craft's claimed independence would make refusing the assignment difficult. Red Hat would work hand in glove with Mind Craft to optimize the bench scores on the same tests & same hardware used in the MS sponsored bench.
Ideally the best way to have an impartial test is to allow Microsoft partisons to tune the Microsoft machine and Linux partisons to tune the Linux machine.
While this may simply be an exercise in who is the better system administrator it would at least insure that nobody purposefully mis-tweaked something.
Ideally both parties should also have access to the same help facilities that any end user would have access to.
Except they are guilty.
They already have mention of the survey on their site here:
http://www.microsoft.com/Window s/dailynews/042199.htm
A worker at a fairly reputable computerstore told me that a certain advertising for a certain game on a certain computer was heavily optimized for a 15second 'slot' that shown how 'superior' this videocard was to that videocard. Blah,blah,blah.
Presentation can be sensationalism can be snake-oil. The general public would never know 'the magician's sleight-of-hand'.
Real world benchmarks, to me, are done on real world servers doing everyday real world tasks not pansy advertising to push a point...
Point 1: The benchmark got nothing more than a passing mention.
Point 2: Has anyone proven that Microsoft told Mindcraft to run the test in the fashion that they did? Everyone seems to be assuming this point, but for all anyone knows Mindcraft did this all on their own.
This is just plain ol' bullshit.
The study has shown that if you take a bunch of experienced MSCEngineers with a high end server designed for NT and a support contract for NT and let them try replacing their carefully tuned server with a basically out-of-the-box standard Linux setup, the performance will significantly drop.
Now while this testing scenario is certainly not fair, it is relevant. We have just this combination in the industry. We have to aim for the point where Linux wins even in this setting so that people will be willing to abandon previous investments and commitments to NT.
So Linux distributors and Apache developers better take care to address all the points in that study, not by commenting on it, but by making sure that it becomes easy for power-users to get optimal performance without having to first be LCE. Only when stack Linux solutions outperform a well-setup NT box will we win over current investments in NT. Of course, *keeping* commitment once we won them is going to be quite easier.
This reminds me somewhat of something from history. "Will someone ever rid me of this annoying bishop?" or some such.. can't remember which English king said it. Maybe Microsoft said something like "Will someone ever prove that NT outperforms Linux?" Mindcraft heard, and performed the test.
Hey presto -- Microsoft never actually asked for the tests to be biased, but they've got the result they wanted.
Cynical? Me? Never.
I don't have the link now, but someone on one of the Corel WordPerfect newsgroups posted a link to a detailed Novell rebuttal of the Mindcraft comparison of Netware with NT. The rebuttal was on the Novell web site, and it seems many of the same techniques were used.
pem@televar.com
So, who is getting terminated over the mindcraft incident?
I'm one of those IT Managers, a CEO, in fact. And ESR has absolutely NO CREDIBILITY AT ALL with me, or with anyone I know. We are doing what we do because we make money by using the best technology. We learned to do this before ESR was born. We know all about the advantages of UNIX, and sometimes of Linux, over NT.
With your attitude, you'll never even get in my door, because you can't tell the difference between a self-aggrandizing bullshitter and the real deal. When you're as good at understanding life as you are at kidding yourself, try again.
I mean, he simply either lied or is guilty of ignorance. ESR is allowed to get away with shit like this. That's why we need him. So that somebody on our side can lie and be ignorant and get away with it. Isn't that what this kind of kiddie journalism is all about?
The entire idea of comparing the "speed" of two different OSes is completely stupid anyhow. You might as well make up the results since they are not indicative of the value of any OS.
The only thing that bothers me is that the supposedly "tech-knowledgeable" community believes that Linux is "faster than" NT or, God forbid, vice versa and that you can produce a benchmark that measures this.
Stupid.
I'm one of those IT Managers, a CEO, in fact.
Both? Oh, lord. Your mom's calling, you'd better go.
ESR has absolutely NO CREDIBILITY AT ALL with me, or with anyone I know.
He's got some with me. Not a whole lot, but some. Then again, I'm all grown up and I'm actually in the industry (development, not IT) and I know what the fuck I'm talking about.
this is hardly analytic rigour at its finest.
your piece is full of ad hominem and
shaky analogies.
i dont care if microsoft says that white is black,
you dont disprove it by saying 'microsoft sucks like clinton'.
anyways, who gives a rats ass about MS saying linux is bad, the tao of computers teaches that
the goal is to be and not to seem, no preaching or shouting will do no good.
what MS really does that is bad is shady international law breaking type of crap.
You're all grown up? And there you are talking about my 83 year-old (when she died) mother. And we've even got your assurance that you know `wht th fck' you're talking about. That's about as impressive as it gets here in slashdot fantasy island, where being all grown up means you've got a job and a girlfriend at the same time. Oh, and you've got printouts of ESR's literary trivia on your shelf. Go Get `em Tiger!
The kind of fiction he generates requires that he no be held to such rigorous standards as accurately determining the sources. Plus, it cramps his style - shoot from the hip after the party's over.
sheesh and look at the companies name for heavens sake Mindcraft, definately a PR "testing" company.
Why the hell would this benchmark be taken seriously when its obvious that they are unqualified to even setup a Linux machine. Shit thats like me getting a bunch of people that never learned to drive to test out how fast a bunch of cars are and I get a indy driver for the car for the company's that sponsered my test.
Some politician said of another (paraphrased)
"He uses statistics as a drunkard uses a
lamp post: not for illumination but for
support."
Because it is now available as a marketing tool for Microsoft ; see http://www.microsoft.com/wind ows/dailynews/042199.htm. From now on, expect that every time Linux is mentionned, Microsoft (or its supported), will answer "oh yes, this unsupported crappy OS 2.5 to 4 times slower than NT".
I think the following headlines describe well what kind of a relationship Mindcraft has with Microsoft:
"Windows NT Server 4.0 Is Four Times Faster as a Web Server Than Solaris 2.6 with Sun Web Server 1.0 and Has 10.3 Times Better Price/Performance"
"What's Wrong with the QuickTime Plug-In and How to Fix It.
"We'll tell you what Apple's bugs are..."
"Windows NT Server Outperforms Linux"
"Microsoft Windows NT Server 4.0 Is 25.5% Faster Than Novell NetWare 5 as a File
Server and Has 2.7 Times Better Price/Performance"
Anybody who could mistake you for a responsible adult would have to be on crack.
Get over it.
Sue Mindcraft for misinforming the consumer.
This is a great idea. But.... Not everybody will have this experts available when setting it's own network. So, this would not be REAL world performance.
Not everybody can tune his server like these experts can.
This contest would be "who can tune it's own server better" and not "wich platform is faster".
yeah, but the EULA's invalid anyhow, right?
Have there been any legal procedings in the aftermath of Windows refund day?
Netcraft is site surveying site that has always been objective and fair. MINDCRAFT are the idiots in question.
Demonstrate performance on SPECweb96 on a 4-way Xeon. Compare against existing NT4 and Solaris benchmarks on similar 4-way Xeons.
--G
Does it really matter whether Linux outperforms NT on a high-end SMP server? We know that Linux outperforms NT at the low end, and other Unixes outperform NT at the high end. Linux has NT looking like a poor choice for both the small-scale departmental server that has always been its bread-and-butter, and Unix has always made NT look like a poor choice for the high-end enterprise servers, and that doesn't seem to be changing.
So what is left for NT?
You have to be real careful performing benchmarks against commercial software. If you read thru some of the Micro$oft EULA's there's specific prohibitions on publishing benchmarks. In short, without an agreement from Micro$oft, it's a good way to get sued!
The commercial product I work on would love to publish direct comparison benchmarks against the competing MS product. Our legal department won't let us.
Doesn't anyone else find it interesting that after funding this Mindcraft study, MS didn't put out a press release touting the results?
Personally, I think MS realized that Mindcraft didn't do a good job of running the tests, so they declined to advertise the study. Mindcraft just put out a release on their own.
You have to understand how these things work. All good software companies run tests of their product under various conditions on various hardware configurations. When they find one that beats some portion of their competition, they try to advertise that fact. To get credibility, they hire an independent lab to reproduce the tests.
In this case, it looks like even MS was suspicious of how poorly Linux fared. My guess is that MS knows that even a well-tuned Linux will lose this particular test, otherwise they wouldn't have hired Mindcraft to run it in the first place.
With the recent report on the untuned NT vs Linux with Oracle server, many of the marks were 20 times faster on Linux than NT. So, I got to wondering, wouldn't it be funny if someone was able to achieve similar/better scores in a Linux system with a less powerful machine (how many of us have quad xeons?). If it were shown that a dual-p2 or something running Linux was just as good as their xeon beefed up NT box, that would make quite the statement, both on the study and on NT.
As the guy above said, a quad-cpu machine isn't terribly high end, but more than the average desktop certainly.
:)).
But with regards to the study, wouldn't it have been more truthful to tune the Linux box appropriately? Then if the results showed that NT was better, a point could be made that Linux needs more work in these areas. I really believe people would accept that as honest criticism (again, IF the results showed that NT was faster, I have my doubts that would happen
Mindcraft obviously didn't make much of an effort in tuning the box (ie hire a decent Linux admin), posted a message to a couple newsgroups (lacking sufficient details, and not responding when someone requested more details), one call to RedHat (which was directed to the wrong group as ESR stated). And yet, they build an entire study around these points, making claims that Linux support is bad and all. For these reasons they were blasted, not for saying NT is better than Linux (as some people I've talked to think is the motive for Linux people's outrage).
In the Performance Testing section on their web page, second paragraph they say flat out:
"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."
Since they say Microsoft sponsored the test, we can replace "you" with "Microsoft." So they worked with MS to define the test goals (NT is 2 or more times better than Linux). Then they put together the tools to do that, hacking the registry and all to beef NT up, slowing Linux apache/samba servers. And finally, report the results back in a form that satisfies the test goals, lo and behold NT is 2-3 time faster than Linux. Such a surprise, right?
Very interesting, What was the total cost of your system?
Things I could see improve your system:
drop the Apache RPM's and compile it.. Specificaly with the PGCC compiler, I have heard of 30+% speed improvements with it. you could even go for the PGCC based distro stampede for your base install.
look at mylex RAID boards.. they are supposed to work a little better than the AMI cards that Dell uses.
It'd be fun to send that box to mindcraft, and have them test it as same preformance or 90% preformance of the NT box.. but cost less than $5k
not $18,000
Xenophobia. Managers are scared of Linux because it's not NT, which they're used to.
Managers are used to NT? Where? Managers that propose NT for high-end stuff definitely never seen NT for long enough to be "used to" it -- but they are "used to" being bombarded by M$ advertising/propaganda of NT, and this, not mythical "they are already used to NT" should be counteracted.
Contrary to the popular belief, there indeed is no God.
an impartial third party that can optimize both Linux *and* NT fairly.
Please note that this report was (deliberately?) released a week before Comdex, when most Linux vendors were running around like chickens with their heads cut off trying to get everything straight for their Comdex displays. And most of the top people for Linux vendors have been at Comdex all of this week, while the home offices are understaffed and slowly going crazy because there's not enough people to do everything that's needed to keep the business running...
Given all of that, it's a little early for sponsored benchmarking by members of the Linux community. This stuff takes time, and if it's a choice between doing Comdex right and repeating a discredited benchmark, doing Comdex right comes up at the top of the list every time.
Send mail here if you want to reach me.
Isn't that what I said in the "Mindcraft Reality Check"?
There are valid limits to Linux scalability, problems that need fixing, and honest benchmarking can help us find those limits. Unfortunately, Mindcraft's benchmarking was so flawed by misconduct and poor judgment that it is not useful for that purpose.
BTW, I do agree with the Microsoft spokesman who said that he was certain that NT would have come out on top even with an honest test. I suspect the SAMBA results would have been quite competitive, within 3% (more or less) of the NT numbers, but the Apache server has never been known for its static file serving speed (though mod_mmap_static may change that!). On the other hand, there is a big difference between the 5%-10% advantage that I bet Mindcraft would have found, and the ridiculous numbers that they actually reported. They actually shot themselves in the foot here, because if they'd reported the real numbers, the Slashdot Crowd would have howled, but Jeremy Allison and other technical heavyweights would have stayed on the sidelines working on fixing the problems found, and the media would have ignored the Slashdot Crowd.
Just count it as another example of Microsoft Arrogance (tm) outweighing their good sense. It's amazing how such bright people can do such stupid things.
-- Eric
Send mail here if you want to reach me.
Tuning Linux properly does not have to involve cryptic commands...a Tcl/Tk (or Perl/Tk...or Python...ad infinitum) applet could do the trick just as well, thanks to the sysctl stuff in /proc. I've written quite a few of these for my personal use...perhaps I should spiff them up and release them.
Posted by kkkotta:
Everyone is complaining how Microsoft rigged this test, but no one is surprised or doing anything about it. Why doesn't someone just sponsor an HONEST test, and let the best system win.
Posted by smich:
Wasn't the test supposed to be about how nt can scale up but Linux can't? We all know the test was rigged, and I'd like to see it run again, but over a range of machines.
What does nt vs Linux look like on a 486 66 w. 8 meg of ram? Hmmm?
Posted by kewlmann:
I work for a fairly large company thats coming out with a kick ass machine pretty soon. Its enterprise class for sure. Lots of processors. I would love to see linux tuned for it.
Posted by The Masked Miscreant >:):
Bear in mind that Intel's integer core is more primative than either AMD or Cyrix's current design. That means that for the vast majority of non-3D/CAD software you'll see a marked improvement over a PII 300 when you use even Cyrix's relatively weak MII chip (whatever P rating actually runs at a 300MHz clock speed).
I sat through an Intel marketing presentation back when I worked retail, and was told flat out that floating point power was more important for word processing, and that every processor that runs at a given clock speed generates the same heat, because it's the little clock crystal that generates all the heat (not the resistance as electricity flows through the chip).
Just goes to show you can't trust marketing to give you straight facts.
The difference between you and MS, obviously, is that you have integrity and pride.
Microsoft is to software what Budweiser is to beer.
Yes the study was flawed. But I remember a comment by Matt Welsh in which he said that Linux is not properly represented in the high-end machines.
That is a natural consequence of the open source
development. Many of the features of Linux are there because some users needed them (scratch an itch, as ESR says...)
So, bearing in mind that there aren't many Linux
users with quad xeons with 4 Gb Ram, it's only natural that issues relating with that kind of machine have a lower priority to the Linux user community
The PC won over the mainframe? Perhaps in the general-utility-computing arena, but for truly obscene loads and outrageous availability requirements, mainframes still rule. Ask E-Schwab, for example, or REI.
Mainframes will never die. The legacy system of tomorrow will be mainframe transaction-processing systems fronted by SP/2 analysis clusters, with something like Linux or NT gating the whole mess to the web. I can almost guarantee it.
Of course, Linux is being ported to the S/390...
Remember that what's inside of you doesn't matter because nobody can see it.
I'm curious about this too. I wonder if people who pay Netcraft money get that kind of breakdown.
One thing to remember is that Netcraft counts domains, not IP addresses. ISPs that host sites for clients probably dominate the survey and most of them run Apache. Companies that run their own web site off a T1 or ISDN line probably favour NT, but they would be in the minority.
I like to take each months numbers and pretend that nobody is switching (clearly not true, but might be a good approximation). This month, for example, Apache gained 423063 sites and MS gained 132943. So new Apache sites outnumbered new MS sites by MORE THAN 3 TO 1! That's impressive.
It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. - Abraham Maslow
I would still like to see some more conclusive stuff on Linux's highend SMP abilities (4 or more), bother on i386 and on Alpha and UltraSPARC. Alan Cox claimed there were some speedups to SMP late in 2.1.x (i think) that should have significantly improved hi end perf. Perhaps VAR would sponsr a test?
If Linux doesn't beat all on up to 16 or so processors, we should fix it.
It's not that simple. Currently the biggest problem with the Linux SMP implementation is the IO subsystem. (SCSI and IDE, etc.) The problem is that this subsystem isn't SMP safe. So whenever the kernel enters this portion, it grabs what is known as the kernel lock. Thus disk activity can only happen on 1 processor at a time (bad). In Linux 2.3 they're going to strip this away, so that Linux 2.4 (or 3.0, or whatever) will probably scale far better than Linux 2.2 (of course they'll be making other nice optimizations along the way). But this isn't something you can just fix on a whim.
One could interpret it that way, but considering that they willfully avoided being helped to tune Linux (ask for samba help anywhere but on the samba lists/newsgroups, call the wrong helpline, don't ask for the right one, when they accidentally do find a helpful person, ignore him, etc...) it seems that the goal must have been to find that NT outperforms Linux.
It reminds me of an old Mad Magazine comic, A father orders his steriotypical hippy son living at home to go get a job. The son, dressed in his most casual attire (torn cutoff shorts, open shirt, many beads etc), goes to a clothing store and says "You ain't got no jobs do you?". Later to his father, "Well, I TRIED!".
The only difference is, Mindcraft actually did get an offer, so they had to ignore it.
I feel the need to ask why this piece was worthy of "airtime", and can reach only one conclusion: That a refutation can somehow become more valid because it comes from the pen of Eric Raymond.
I'm now happy that Katz periodically contributes his dozen screenfuls of drivel, and I don't have a problem with Eric either, but would I have got a whole item on /., had I summarised the obvious flaws of that ridiculous benchmark? I doubt it.
Matthew
- what happened to our great meritocracy?
the PC community *won*?
what have you been smoking?
with the rise of n-tier architecture and thin client, mainframe-style computing is getting stronger than ever. centralised data and logic, just the display on the desktop.
www.mindcraft.com
www.netcraft.co.uk
I've seen very little comment that Linux might actually really be slower on a 4-way. I would be disappointed, but not amazed if Linux were slightly slower on a 4-way given the maturity of NT SMP compared to Linux.
I would like to know if Linux does scale as well or better than NT with 4 and 8 processors -- both systems properly tuned and using the same webserver. When that question is answered, I'd like to know what to expect in the future. Is Linux going to leave NT in the dust, or will this be the key niche ground for NT servers that Microsoft will defend to the end, and Linux will never conclusively defeat?
As an extremely unhappy user of an sp/2 system, I hope the sp/2 isn't the system of the future. I can do some things faster on my pentium 90 with Linux than the SP/2 system. We have 32 nodes with 4G each and 2 Terabyte of disk spread out over them. With this setup, you'd think things would be fast. The only thing that happens fast is the corruption of my data by their filesystem. In short, the SP/2 system isn't made for databases or web serving. It's meant as a compute box (and it doesn't even do that well!)
Mike Mangino
mmangino@acm.org
It isn't the sp at OSU, I don't even know if they have one. We also use a proprietary dbms that searches roughly 2 terabytes of data. The IO performance is bad, and if CPU usage is high, IO drops to about one read a minute.
Mike Mangino
mmangino@acm.org
I think it'd be best to have one set of benchmarks for default configuration, one set of amateur tuning, and one set of tuning by experts for each of the platforms tested. Get a linux expert... someone from the Samba team, someone from the Apache team... get an NT expert, someone who works at Microsoft, I guess (though not just anyone, obviously). And let them go to town.
Sometimes people set their systems up by just leaving them in the default because of laziness/time constraints, whatever. And sometimes a minimal amount of configuration is done, but not a whole lot. And then there are performance freaks. Got to show the spread...
Run a pencil-and-paper RPG campaign with your far-off friends: Gametable!
While our box is not SMP, I very much doubt that SMP on 4 processors (which even 2.0.36 was known to cope quite well with, especially for simple tasks like web serving) would cripple a box to the extent provided by the mindcraft survey.
I can see an SMP system to only improve performance up to a point, eventually hitting a limit (e.g. 8 processors won't be twice as good as 4 because of time spent scheduling), but to see the effects that mindcraft saw you would have to do something pretty crafty...
Matt.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
I think the total cost was ~£3500 UK.
If you read what I wrote you'll realise I did drop Apache and recompile - I just used RPM's. I don't really think the problem is the compiler used for Apache - I think most of the work is in the kernel, managing processes and file caching. I guess a pgcc compiled kernel would be a better option - but that's not something I'm desperate to get into - this system took me 1/2 a day to build - I don't have the time to really increase that.
A better option would be to just use Apache for mod_perl processes, and use thttpd for static content, and use squid to proxy requests to the right port. I think we'd probably blow away even our own estimations with thttpd.
Matt.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
You're only right on the kernel front - that hasn't really been used enough on high end machines AFAIK.
But both Samba and Apache have been heavily tested on very high end servers. The Samba crew have even been heavily involved in making Samba fast on high end servers.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
Please note that the dejanews reference that ESR links to is quite wrong. The presence of %h does _not_ cause host name lookups under Apache - only the directive "HostNameLookups on" causes that to occur. I don't believe this to be the case.
I strongly believe however that their httpd was running under inetd, and that would cause the effect they saw.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
At a large company I'm working with we're trying to prove to the phb's that Linux is a good thing. The mindcraft study set us back a ways. So what did we do? We did our own tests.
/proc fs to speed things up. Stock kernel options selected from "make xconfig". Apache was the apache+mod_perl srpm found on redhat/contrib, compiled with no configuration changes. We didn't test NT on this box - we were trying to compare against Mindcraft's results.
Server:
- Hand built by our best hardware guy
- PIII 500 (single CPU)
- Adaptec 2940U2W SCSI Adapter
- 10,000 rpm LRW drive. 1 drive only.
- 100Mb/s network card
- 256Mb PC100 RAM.
- Linux 2.2.6, upgraded from stock Linux-Mandrake box
- Apache 1.3.6, configured for best performance.
No changes to the
Want to know the results so far?
Well, we can get about 2200 requests per second out of that box. The Quad Xeon NT box that mindcraft tested got 3700 requests per second at its maximum rate. We are at very early stages so far, and I think I can squeeze more out of the box by dumping Apache and using thttpd or something else that uses a threaded model. But since this is to be a pure mod_perl box I don't think that's important.
Things to remember:
The mindcraft server had 1Gb of RAM.
The mindcraft server had RAID (RAID/0 I believe).
The mindcraft server had 4 10/100 network cards.
We're so far pretty pleased with our little Linux box... It was a fair bit cheaper than Mindcraft's server....
Matt. Want XML + Apache + Stylesheets? Get AxKit.
From the linux-kernel list:
Mindcraft also used the v0.92 MegaRAID driver. An SMP race condition was fixed in v0.93 which was almost certainly available from the AMI web site long before the Mar 10-13 test. So SMP NT "beat" a non-SMP Linux on a quad-Xeon server. Big hairy deal.
Original poster is "Doc" Savage. Original post 14 Apr 99.
Hmm Linus has one so I would expect that the kernel at least would operate very well on it.
If the kernel operates well then it is highly likely that other software will get a boost.
Hettar.
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying "the study was paid for by Microsoft" and that "a very highly-tuned NT server was pitted against a very poorly tuned Linux server".
That is, the claim attributed by Eric to Ian Hatton was really made by reacting Linux supporters.
What Hatton did admit, was:
"Microsoft did sponsor the benchmark testing and the NT server was better tuned than the Linux one."
This isn't much, but it is sufficient. Hatton admits that "the NT server was better tuned than the Linux was" and even without adjectives that invalidates the report.
-cjr
Now that EGCS is available under a more rapid release schedule, you probably won't need GNUPro just to get your code to compile, but it might be a good deal if you want the latest PII/PIII optimizations that haven't been rolled into the public codebase yet. You also get a visual debugger and some other goodies.
Here's a press release on the PII/PIII optimized GNUPro tools that will be available next quarter.
--
Jake
"Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself."
Well, you have to define "won" pretty carefully. If you mean that the group that controlled the centralized (mainframe) resources was forced to give up complete control of information management services, then yes, the PC "won".
But keep in mind that big dollar, mega-user, high-bit-rate applications are almost always run on IBM (or compatible) mainframes. Or on mainframe-class minis (Sun, etc) that are designed, installed, and operated using mainframe class operations discipline. And centralization seems to be on the rise at the moment, not on the decline.
sPh
I am not suprised by how "blatant" this whole episode is. Relatviely speaking, this is nothing to the FUD heaped on OS/2 by MS after the split with IBM.
Those who witnessed "OS Wars" of the early, mid-90's are well aware of the ability of MS to bludgeon superior technology into submission through marketing.
The main (and important) difference I can see is that today, MS has less credibity, and their target, being Linux in this case has no corporate "owner" like IBM. IBM sat idle while MS (and the IBM PC Company, "MS' biggest customer") and the trade press trashed OS/2 into oblivion.
I do not see the Linux community standing idle and taking it, ESR's post is a fine example of this. Note that benchmarks like mindcrafts were done with NT vs OS/2 over and over with no real response from IBM. The OS/2 users who protested were categorized as "zealots" and written off. On Compuserve, false user accounts (see "Barkto") were alleged to have been created to depict "real users" who then went on and on about serious OS/2 errors that "trashed my hard disk" and "my backups", ad nauseum. (Such reports were then published in PC Week, Infoworld, Computerworld to drive home the FUD).
MS has it's hands full trying to FUD Linux into obscurity. But be assured, they are experienced at this type of "warfare" and will attack furiously. With such deep pockets, I expect they feel a war of attrition is winnable.
This remains to be seen. The Linux community is not an impotent IBM. And today, we have a maturing internet to get some real facts distributed that the traditional "legacy" trade rags tend to not report.
You guys bring tears to my eyes, matching a fully tuned $25k NT box with an untunes $3000 linux box.
:). Hey, if it's good for /., it must be good enough for everyone else.
If you want some more performance, I suggest moving to a dual system. My experience in SMP linux has only been with dual systems, and it's been very good. In addition, there are some very nice, cost effective dual motherboards out there. Tyan makes one with onboard aha2940 SCSI, Intel 10/100mbit ethernet, and sound too
I wouldn't be surprised if with some tweaks to your box configuration you could make it as fast as the NT box without any hardware mods, perhaps by following some of the advice Eric links to...
----
I would still like to see some more conclusive stuff on Linux's highend SMP abilities (4 or more), bother on i386 and on Alpha and UltraSPARC. Alan Cox claimed there were some speedups to SMP late in 2.1.x (i think) that should have significantly improved hi end perf. Perhaps VAR would sponsor a test?
If Linux doesn't beat all on up to 16 or so processors, we should fix it...
how good PGCC is on p6-core processors. In my experience, the key on p6 is not pairing, like it was on the Pentium, but avoiding partial stalls, which empty the pipeline and really fsck things up.
In our own tests, we found that VisualC++ 5.0's (otherwise an excellent compiler) ftol() stalled like crazy on pII's, eating 10% processor power in Fire and Darkness. How good is PGCC at avoiding similar problems.
Are there tools under linux (analogous to Intel's VTune) for analyzing this?
Remember that a certain number of sites really need big-iron servers (hey, slashdot isn't exactly gentle on its hardware, although in that case I suspect database performance may be more of an issue), and even when they don't it's the results from high-end server tests which impress the management the most.
Having seen Linux/SMP in action and made some subjective judgements I'm quite confident that, properly configured, it ought to scale fairly well onto hardware of the class Mindcraft were `testing'. But it would still be nice to have some number...
I think you're a little bit out of touch with the vast majority of the Linux community.
One of Linux's prime virtues is what it can do with older hardware. I have a friend who wouldn't believe that the Linux machine she was using at my house was a P90 because Netscape/WordPerfect/etc. "felt" as fast as W95 on her PC (a PII/333).
I have three Linux boxes, the (dual) P90 mentioned above, a K6/166 that acts as my web/mail/ftp/telnet/IRC/MOO/everything server and my 386/25 laptop that I use to do my homework.
I would wager that the vast majority of Linux boxes are not high-end monsters but old machines that are "worthless" in the eyes of many people. That is the true power of Linux in my eyes, an attribute that I think is all-too-often ignored here lately.
IMHO, YMMV.
--
Posted with Mozilla
Actually he said that he was comparing results with the NT box, which was tuned to the max. Nice results!
ElpDragon.
I don't know where you get your definition from, but where I come from there's nothing good about fear, uncertainty, or doubt.
You definitely do NOT have to say anything nice to sling FUD. It's not in the definition.
I didn't see anywhere a reference to duplexing or
ethernet card configuration. In redhat's default
configuration I have noticed that it selects 10Mbit most of the time. And HALF duplex all the
time..
Only way out of this is to pass the options= command when loading the ethernet module.
-- You can be a geeklord too
then it shouldn't be tough to distribute freely, just one person buys it, and then gives it to the world.
First - sponsoring a rebuttal in the
form of another benchmark will have to
be left to the Distro guys making money...
Cause it costs money to put together such
a test system, or purchase NT for that matter!
On the other hand - putting out a decent
rebuttal in the form of accurate criticism
such as ESR has done(I REALLY like his
article) is perhaps the best way to point
that Emporer Bill isn't wearing any clothes.
The only remaining trick is to get that
rebuttal circulated amongst the press
widely. ESR has the credibility to get
quoted in such places. Looks like a
good combination, and the right path
to me.
Steve
Have you compiled your kernel today??
Actually GUNPro is not GPL'ed, only the compiler part is. Also the $7000 cost also includes a support contract.
I rather ambivalent about some of erics latest moves, but he has written software, and tried to do a good job, but these rather petty comments begs the question. What have you done for the open movment?
"Think of it as evolution in action."
Good summary.
Seems to me that what we really need is a bench marking rebuttal; is there another
benchmark going on? I saw that in Jeremy Allison's article he was working with PC Week,
does anyone else know any other active bench marking going on?
I think that the only way to prove against FUD is education, bench marking can go a long
way.
I have about 7 Linux servers with no down time, great performance on lesser hardware then
my commercial servers in my company, that should be proof enough; but my pointy haired
boss still asks "Why not NT?". I do not need any more fuel for that fire.
We need Benchmarks on larger servers, with more memory, RAID, and a high-end server
guide.
"Think of it as evolution in action."
What we DO need is a HOWTO describing the idiosynchrocies of doing this under Linux. What parts of the tuning are in the Kernel and need recompilation? Where are the tweakable parts? What needs to be frobbed under /proc?
From there, it's a short jump to the developers of Apache and Samba to say "increase the PROC table", or "increase the file buffer area". This advice would apply to all architectures. (But even if they don't tell us how, a good admin can probably figure most of this out.)
Get off my lawn.
Sure, maybe a good admin CAN figure this out. But I am an Oracle DBA primarily, not a Unix Administrator. Sure, I can keep a box up and do the necessary maintenance and generally perform that job. But not with the nuances and expertise that I can administrate an Oracle database. I would like to have a good source to learn this stuff without having to change careers to do it..
And then there are those who aren't GREAT administrators. We all know 'em and have met them.. They can do their job (well some can't) moderately well but not very well..
Stick Boy
--- "The problem is not that the world is full of fools, it's that lightning isn't being distributed correctly." -- Mar
Good article Eric, and I'm proud of the Linux community for the way we've reacted to the Mindcraft "benchmarks". I think the Linux community fought back, but in a mature balanced way. I think it is important that we continue to do so, and try not to appear too much like reactionary fanatics which somtimes happens too.
Also, I must say I really learned a lot by following the debates. Next time I need to install Apache and Samba you can bet I'll be referencing the responses to Mindcraft to see the proper way to optimize this stuff.
Kudos and thanks to the Linux Community!
Without being too fanatic, I think that we all should inform any magazine publishing the Netcraft results (and thus concluding Linux is sh** compared to NT) of the facts and unreliability of this survey.
Admit, "M$ guilty of consumer fraud" is a better headline than "NT beats Linux on all fronts".
It would, IMO, not be a good idea to get Mindcraft to do another test.
It would be a far better idea to get a truly _impartial_ party to re-do these tests, with proper help from the Linux community. Then we'll see the results!
I simply don't _trust_ MindCru^Haft.
The VMS guys have been used to this kind of FUD for a while. They call it 'BenchMarketing', which I think is a clever term.
:^)
One of the things to keep in mind is that you're never going to get more out of this (1 CPU, 1 NET card) configuration than the mindcraft NT performance figures: The peak NT performance was above 100mbps.
Some people are suggesting to use squid to direct requests to apache for the complicated stuff, and to thpptd for the simple stuff. I personally would try to make a tool that would change all local URLs to include the portnumber for all references that are a simple file. This might not be allowed for the benchmark, but it sure would help in the real world.
Roger.
The language makes it very unclear whether the quote came from the Linux supporters or from the Microsoft SA release.
Of course, it's unlikely that the quote came from the Microsoft SA press release (It is referring to the original brag release: "Independent research shows NT 4.0 outperforms Linux"). I think that cjr was quoting Linux users and therefore Eric was wrong to attribute it to Microsoft, much less to Ian Hatton directy.
In the wake of mounting "corroboration", it may now, more than ever, behoove the Linux Industry to respond to the Mindcraft challenge. "False", "rigged", "FUD" - these are the cries of the outraged knowing, but where corporate IS is concerned they fall on deaf (or "endeafened") ears. We must remember that up until narely a couple years ago, Linux had practically no coverage in mainstream media and was principally the purvue of relatively small group dedicated Internet hackers and computer enthusiasts. Linux did not make its way into the hallowed halls of big business by atrition the way Microcrap did. Rather, Linux found its way into corporate IS by way of covert installs and closed-door set-ups thanks to the efforts of IS geeks and tech-heads. Unfortunately, however, these are not the people that dictate corporate computer-systems policies... IT management does. If it weren't for the graces and virtues of Linux (stability, availability, cost-of-ownership, etc.), IT management would likely have mandated its removal long ago, relegating Linux's hope of survival back to the domain of devoted hackers and geeks. Now, with the claims produced by Mindcraft and their corroborators, and the direct challenge by Microcrap, IT management can be expected to watch closely the Linux Industry's response to these claims and challenges. An outraged and snorting response is not likely to engender sympathy, but rather apathy! This can not and will not be good for the future of Linux in the eyes of corporate IT management. Big vendor applications will not be enough, big vendor investments will not be enough, user support will not be enough, media support will not be enough, the voices of evangelism will not be enough! These things and more are already on the side of the corporate IS O/S's Linux is climbing up against. In the end, it will be the non-geeky, non-technical IT management force that will decide whether Linux lives or dies in their venue. They do not read Slashdot, they read the Wall Street Journal, they read the New York Times, they read PC WEEK and PC Magazine, but most of all, they listen to the BS of Microcrap! Win or lose, meeting the Mindcraft challenge can only be in the best interests of the Linux Industry.
"LinuX - Dropping the c u r t a i n on Windoze." -- Vee Schade, vschade at mindless dot com
Ummm That should be Mindcraft right?
This is an alramingly common mistake - poor netcraft.
Well, I think this incident has damaged microsoft's credibility, but that's beside the point. Microsoft isn't talking to us, the technical community. They aren't trying to convince us that NT is better. For those of us in server closets, in the operations center, and in system administration - we already know the truth. We don't need benchmarks and statistics to tell us NT is unreliable.
The plain fact is, Microsoft did this to appeal to middle/upper-management, not us. They need to keep feeding them reasons to keep their NT investment without looking stupid. Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.
--
I remember when I first implemented the RC5DES cow on my home Linux box (AMDK5PR90) and on a mostly idle Win95 workstation at work (Intel P166), and my home Linux box was running circles around the blocks of the Win95 workstation (in fact, a few of my friends were embarrassed by their WinXX Intel PII's for not being all that much better than my Linux AMDK5PR90).
So, it makes me wonder about this test as well... Is it possible for someone to tweak a common-joe-affordable Linux box to outperform a supercharged, out-of-affordability-range NT box? Can someone duplicate the server load from the Mindcraft test on a highly tuned Linux PC and show that Linux can beat NT even when Linux is on a smaller machine?
~~~~~~~~
Signature illegible, could be somebody else.
What strikes me about the entire "Mindcrap Affair" is the resulting coverage. I can recall seeing only one press article covering the original story (the "benchmarking"), but I have seen many press articles covering the resulting controversy. Of course, my impression may be biased because I take pointers to news stories from Slashdot and Linux Today. On the other hand, I have done some looking outside of the "linux community", at sites such as CNet News, and they definitely seem more interested in covering the fiasco than in the original benchmark. Maybe these sites too can smell a rat.
--JT
An anonymous user wrote: That's very easy to say, particularly from your anonymous vantage point. I don't see any reason to suppose, when you add up all the positive tweaks that were done to NT and the negative tweaks that were done to Linux, that the result has any connection with reality.
If MS really thought Linux would lose a fair test, we'd now be seeing the test results with a detailed, reproducible description of what was done and it would clearly indicate NT the victor, comparing apples to apples.
Instead, we have an extremely flawed study which Microsoft paid for, which looks to exhibit systematic bias. Coincidence? Could be. ;)
Please call the imposter MindCraft, not netcraft...
I'm begining to think that this is all a conspiracy to undermine Netcraft:)
-- To dream a dream is grand, but to live it is divine. -- Leto ][
I think your version would test the abilities of the respective administrators much more than the software, and is nowhere near the real world. A sysadmin has available plenty of resources to find information and ask questions, not just a limited 24 hours. I think two *groups* should be given identical machines, have each set the box up as best they can in a week, and then run the test. Each group can get outside help, of course, but it must be limited to public information. To prevent "we applied patch xyzzy which makes the kernel faster when doing z but six times slower at y, but the test only tests z", the groups could instead generate a list of things to do to the box to tune it and have someone who can just get around do the monkey work...
i have tested a dual pII/400 with pretty similar specs in Linux:
3COM 905 10/100
onboard SCSI on Gigabyte Motherboard
256 megs of RAM
9 gig Quantum SCSI hard drive
The results were about 20% above the quotted figures for the quad xeon. Perhaps i'm wrong but my $1649 system shouldn't be faster than a $25,000 system. Linux SMP is not nearly as stunning as BeOS SMP. Is there issues in Linux with 4 gigs of RAM slowing down the system last I checked Samba only supports 2 gigs.
By the way the Mindcraft config favored NT quite obviously by using NTFS file system and Raid 0, which should alone roughly double HD access speeds.
Sorry, but a test (particularly one that is supposed to be objective and scientific/engineering in nature) has but ONE GOAL: to determine the facts as objectively as possible.
That isn't 'your' goal or my goal, or Saddam Hussein's goal. It's THE goal, PERIOD. Anybody who makes statements like that admits they run a bovine excrement factory instead of a testing facility.
Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.
Of course, IBM didn't help themselves by trying to alienate their customers in the early 90s by trying to withdraw all their mainframe source code. That was about the time we started looking into unix systems. BTW we're going to dump our entire machine room full of IBM mainframe equipment at the end of the year
Hasn't anyone considered for a second that perhaps their definition of test goals is "determine which OS is a faster web server in a RAID/SMP system" or some other criteria?
:) but I read that sentence in a completely different way.
Not to defend the report (it's not Scottish, so it's craaaaaaaap
A Quad Xeon is the highest end box that NT runs on (barring Alpha).
Note that WinNT is driving the "high-end" x86 hardware market. Vendors like Dell and Compaq make boxes with only 4 CPUs because that's all vanilla NT will support. When Win2000 comes out, it will support 8? processors, which means the hardware companies will immedeatly follow with 8 CPU iron. (Implictly making this hardware available to some Linux folks.)
Of course a better benchmark would be the $50,000 NT/Dell box versus the $50,000 Sun/HP/DEC box, etc.
--
Business. Numbers. Money. People. Computer World.
A few years ago, nobody in their right mind would have proposed any x86 (Novell, OS/2, NT, Linux)solution for anything other than workgroup filesharing or a ccMail postoffice. The fact that you can now seriously consider Linux or NT as a contender at the low end of the midrange market is primarily due to the advances in Intel hardware.
In a few years, Solaris, Tru64, HP/UX, Linux, and NT will all be running on essentially the same Intel IA64 hardware. At this point, the appeal of NT's one-size-fits-all design is going to start breaking down. But on the other hand, hardware equality is going to get Microsoft's salesmen in the door for midrange solutions that were previously above their heads. And Microsoft is more price competitive than commercial Unix, so NT deployment is probably going to increase in this market, not decrease. (Same argument for Linux.)
--
Business. Numbers. Money. People. Computer World.
If I understand correctly, those numbers are public webservers only. MS IIS's market strength has been internal Intranet solutions (where there's probably an existing NT file+print setup). IIIS's intranet market is probably going to being going up, not down, as things like the Office 2000 server get deployed.
--
Business. Numbers. Money. People. Computer World.
Following myself up, Allison does have a message for those who take the Mindcraft bench at FUD value alone:
The study has also shown that the knowledge of how to tune filesystem performance in Linux is equally obscure. We need to do a better job of educating Linux administrators about how to get the most out of their systems.
I'm sure there'll be more benchmark disappointments in store for us. After all, how else do we learn what we need to fix? But the strength of open source is that we can face the errors without trying to deny them. We just fix 'em and move on.
--
Business. Numbers. Money. People. Computer World.
The 'Rush Limbaugh' principle is a very valid point, especially in this context. Don't forget the target market for this study is Microsoft partners and WinNT-based shops.
Aside from all the meaninless numbers (who cares if your web server can saturate a 100BT line with static pages!), the study drives home an important point to NT Administrators - If you've invested in a high end IIS system, and you've got it tuned, there's probably no good reason to switch that box over to Linux. If the Linux box was tuned correctly, I doubt the difference would be that great performance-wise.
Of course, the study didn't address stability, which is the number one problem with IIS.
--
Business. Numbers. Money. People. Computer World.
I just took a look at the linked article written by Jeremy Allison of Samba.
/proc filesystem.
A few interesting points -
* In the often referred-to ZD Samba versus NT benchmarks (where Linux+Samba wins), the Samba/Linux configuration was tuned by a Samba team member. Objectively, this makes the ZD benchmark actually less valid as the Mindcraft study, because as far as we know, a Microsoft-employeed SMB developer wasn't actually there tuning the server.
* Tuning Linux properly involves cryptic commands such as:
echo "80 500 64 64 80 6000 6000 1884 2" >/proc/sys/vm/bdflush
echo "60 80 80" >/proc/sys/vm/buffermem
While I'm sure these commands are documented somewhere, this sort of tuning makes the NT Registry Editor look like a model user interface. Low level tuning like this really needs a nicer front end, or preferably, a daemon which monitors system activity and dynamically tunes these settings.
It sounds like the Mindcraft study has been a kick in the pants for the Linux community to get some high performance documentation together. I'd like to see a nice How-To which lays out some of the more obscurantist tricks such as echoing strings to the
--
Business. Numbers. Money. People. Computer World.
Okay, so Netcraft says that Apache's market share is 1.3% greater than the previous month, and IIS's market share is 0.41% smaller.
But what does that mean?
Netcraft also says that the total number of web servers just exceeded five million. Is all of this Apache vs. IIS activity happening on existing web servers, on the new ones? Is Apache growing slowly-but-steadily across the board, or is it growing like a weed on new web servers, while market share on the existing ones remains frozen? That's good news, too, but it's different news. Among other things, it suggests that people aren't so unhappy with IIS that they're willing to put up with the annoyance of moving to a different server.
I dunno, I'm just wondering.
"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"
"Christianity neither is, nor ever was a part of the common law." --
The truth or falsehood of the Mindcraft study is irrelevant to its intended audience. The point is to give NT "believers" something to quote in arguments, that's all. It's the Rush Limbaugh Principle. In a disagreement, it's helpful to have official-sounding statistics to back up your point. It doesn't matter where they came from, and it doesn't matter whether they're even remotely accurate. What counts is that somebody "important" (read "well-known") said it in public, which "validates" it. This "validation" isn't about truth. What it means is that the proper forms have been followed, and so it's acceptable to introduce the "evidence" in an argument. What's being offered is not evidence in the conventional sense, but the appearance of evidence, or the outward form of evidence. In poker, what does the four of diamonds mean? It means the four of diamonds. It's pure, disembodied symbol.
Disagreement and debate in our culture (especially on the net) isn't a whole lot less stylized (nor a whole lot less predictable) than Noh drama. You have to play by the rules and observe the forms. The content of the Mindcraft study is arbitrary. The study is a signifier, or token. A yacc parser says, "hey, this token is a function, hey, that one's an operator." The actual content of the token is not significant; what matters is what kind of token it is.
Everybody should learn at least a bonehead popularized minimum of semiotics (which is all I know, obviously
While we're at it, let's be honest with ourselves: How many of us are going to check Eric Raymond's facts for ourselves -- even to the minimal extent of clicking on the links he provides? And how many of us who don't check the facts are going to run around repeating them? Quite a few, probably. Dammit, I think Raymond's right on the money with this, and I'm confident that he's done his homework -- but I don't have the time to go about proving it. As far as many of us are concerned, Eric has given us a counter-signifier. Some "good spin" to match against the "bad spin". (That makes it sound dishonest, but IMHO if the "good spin" is factual and accurate, then "good" is a perfectly reasonable thing to call it.)
Think about it.
(Experienced sysadmins are a bit of a special case here. They can judge for themselves. The Limbaugh Principle applies mainly to people who are arguing in an area outside of their field of expertise -- I don't recall who it was who said that "every man is gullible outside his specialty", but it's true even of the best of us.)
"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"
"Christianity neither is, nor ever was a part of the common law." --
Why don't we just send a couple of linux guru's down to netcraft and tune the linux side of that machine? I'm sure they still have the NT side still rigged up. Then run the tests again and watch linux beat NT on every test!
You might want to check out Cygnus's GNUPro utilities, which have lots of PII/PIII optimizations, I think. THey're fully GPL'd, too (not free, though, they cost $).
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
Nice try, Eric, but pretty transparent. I've gotta give this one a thumbs down and await Halloween Letter LXII or maybe I Quit IV, aka "But Mommy, I Use Linux Too So Why Does Everyone Hate Me?"
Cheers,
ZicoKnows@hotmail.com
Ya know, Oscar Robinson was a damn good basketball player in his day, but I don't think there are too many teams that would take him as a player today. He's still a great person, but his best athletic days are long behind him.
Similarly, Raymond's better days of advocacy are long behind him, back around the Cathedral and Halloweens I and II days. Ever since then, it's seems like he's turned into a self-serving egomaniac who can't take one lick of criticism, committing one blunder after another.
Sure, he didn't do any damage with his post here, but I remember seeing at least a dozen other people discussing this much more effectively. Not to mention days ago. If someone's going to weigh in with their opinion so long after the fact, it better be good. This wasn't by a long shot. It was basically, "I am ESR, I have now come to allow you to listen to my infinite wisdom on the subject. Feel grateful, I command it!" What a boob. The guy only remains a player because people like Rob (or whoever posted it) take anything he says, no matter how inconsequential, no matter how many other people said it earlier and better, and elevate it to Topic status. It creates a self-fulfilling fame, like Zsa Zsa Gabor or Charles Nelson Reilly, where a person ends up being famous simply for being famous, and you eventually can't even remember what made them famous in the first place. Lame.
Cheers,
ZicoKnows@hotmail.com
What is missing in all this -- and I'm afraid in ESR's rather sloppy summary as well -- is any sense of whether benchmarking is reliable in the least for analyzing application processing. Things are bad enough with processor benchmarking -- remember the tuning code in Quake?
In databases, things are much worse. The TPC has been wrestling with these issues for a decade now and still doesn't really have a good handle on it. It is too easy to put your thumb on the benchmarking scale without anyone noticing, and make the results go the way you want. This is true even if the vendors themselves do the tuning.
Even if you can equalize the platforms, it still gets down to issues like the mix of instructions in the test suite. The TPC has wrestled with that one for years, trying to avoid tipping the balance toward any given vendor.
Schematically, of course, Web servers are database servers (aside from the issue that they may request data from an actual database).
Frankly, I'm even more appalled at Mindcraft (and by extension, Microsoft) for pushing this "study" the way they did. It was at least borderline unethical, given their admittedly lame-if-you-are-being-kind-about-it effort to equalize tuning between the two systems.
Bill Gates Is My Evil Twin.
Well, it probably depends on what youre doing with the machine. At my job we have a Sun running our Web Server and were secretly running a PII 400 under the table with Linux. Basically, for progamms (usually Perl scripts) requiring large amounts of drive access, the Sun generally blows the Pentium away; obviously this is barely even a matter of which processor the machine's got, but rather I/O capability and Memory(-management). However for scripts which are just doing a lot of 'text-crunching ' the Pentium is faster. Admittedly it's usually running a lighter load, but I still think it shows that the Pentium is a contender in some areas.
As far as I know, Suns and the like are optimized for moving around large amounts of data, whereas x86's are more optimized to crunch numbers. For a home-system with one fulltime user I'm pretty sure a multi-processor Intel with a bunch of memory will give a lot more bang for the buck.
chris
San Francisco values: compassion, tolerance, respect, intelligence
If somebody will buy me a system like Mindcraft used, I'll be more than happy to benchmark it myself! It might take me a while though, so be patient with me. If I have the box back to you in say 5 years, is that sufficient?
---
This sig has been temporarily disconnected or is no longer in service
Actually it's not so much the `crystal' (I don't think they are crystals; rather ring oscillators in PLLs I would think), but the clocking that draws a very substantial part of the power (50 %?).
Keep in mind the clocking circuit needs to drive a lot of transistors, and this takes quite a lot of current !
But decreasing voltage levels will have a bigger impact on power than frequency will.
The angry man always thinks he can do more than he can. -- Albertano of Brescia
According to the CraftyMind survey, the peak performance on that quad box was 1,000 rps', 3,770 for IIS. This implies that a $25k Dell Linux box is more than 2x inferior to you $3k solution.
The Mindcraft box was 1.7 times faster under NT/IIS. The single-cpu box was 2.2x faster that the Mindcraft box under Linux. So this says nothing about Linux's SMP capabilities.
No, not Mindcraft, then they would win! Give the job to an independent group. Better yet, have MS and a Linux corp. each give input into what happens to a particular server, and see what happens. Even better, take an identical Dell system, possibly with a different RAID, and beat Mindcraft's NT results.
You have taken a quote your first quote COMPLETELY out of the context of the article.
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying 'the study was paid for by Microsoft' and that 'a very highly-tuned NT server was pitted against a very poorly tuned Linux server'. In response, Ian Hatton, Windows platform manager at Microsoft SA, says these comments are valid."
Someone posted to COLA that one of the orginal posts for help came from Microsoft (tide71.microsoft.com which resolves but does not return a ping). So we can be sure that what you say is absolute fact.
Anyone care to check who really owns Mindcraft?
No no! They just used a variation of "Inductive Reasoning" as outlined in Science Made Stupid by Tom Weller (copyright 1985). The outline is as follows, with Mindshaft's specific steps explained:
Formulate Hypothesis
In this case "NT is faster than Linux"
Apply for Grant
This is a bit out of sync, because I think the "grant" came in before the hypothesis was ever formulated
Perform Experiments/Gather Data to Test Hypothesis
Benchmark! Benchmark! Benchmark!
Alter Data to Fit Hypothesis
Here they varied from the outline again. Instead of altering the data, they altered the test conditions.
Publish
Press release, etc.
So while they varied from the outline in some areas, it is clearly a valid use of the inductive method. Of course, I had no idea anyone took Science Made Stupid so seriously....
--
Okay, I got Linux installed. So where's the free beer everyone keeps talking about??
You benchmark a server by setting it up on a network, and connecting many clients to it. The clients all run exactly the same stuff. The clients collect the data.
The reason that these tests are expensive is that it takes a lot of clients to load down a large server.
The entire point of benchmarks is to determine if one machine (or OS on the same hardware) is significantly faster than the other. The tests under discusion are run on the same hardware.
On a desktop machine, what works is what you are most comfortable with as a user. On a server what works is not having your clients call you on the phone.
-sh
This is not a political statement. This is not legal advice. It's a frick'n Slasdot post. However: I'm Running For
My idea for a real benchmarking test...
1) Define the machine's specific role(s); e.g.,
web server and SMB server.
2) Define a series of Y hardware configurations.
3) Allow Microsoft to recommend X number of
NT administrators, allow Linux community (maybe
coordinated by VA?) to recommend the same number
of Linux administartors. (X should be a multiple
of Y, perhaps 3 to 4 times.)
4) For each testing round, a randomly selected
NT admin and a randomly selected Linux admin will
be given a random hardware configuration and an
equal amount of time (1-24 hours?) to configure
the machine for the test.
5) Run the tests, averaging the results for each
configuration.
6) Write a press release saying how badly Linux
kicked NT's ass.
I think Microsoft wants to put the big play on the idea that Linux tuning documentation and technical support is inferior to Microsoft's.
And that type of spin fairly documents Microsoft's style of marketting. They definetly don't play nice up there in Redmond.
As usual, ESR has done an excellent job of refuting some anti-Open Source FUD.
He wrote and excellent summary of the FUD, and then proceeded, in an air tight beautifully researched fashion, to blow the FUD out of the water. ("Blew 'em out th' tub!", as my wife's dad used to say.)
However, Microsoft's credibility is now so deeply in the sewer with ANYONE with even a modicum of real computer literacy nowadays that all one needs to say to refute anything they are behind is say, "The study was paid for by Microsoft" and everyone just goes, "Ah, yes, I see now..." and the story is refuted!
Well, not actually having one it's hard to say but... I personally would spend the same dollar ammount in a different way, like four boxes with 512MB RAM on a 100meg switch. Think of the possibilities!
Competition Good, Monopoly Bad.
Ummm, how does one consider a four processor Intel box a "high-end" machine. To my way of thinking, that's very much a mid-end machine. A high end machine would be one with > 16 processors and a ton of memory.
That's an area where Linux truly doesn't play at the moment, but with SGI's donation of an O2K, we may see some big changes there soon. That said, I doubt if we'll ever have great scalability to lots of processors, simply because the user/developer community for those type of machines is comparatively small. Then again, maybe I'm wrong.
Check out Linux University
Yea sure, You can do that with just about any program GPL'd or not. But thats not the point. Accually I am slighly confused about this? Is this one of the licenses that stopped people from using the code to produce products they sell unless they pay for the product, like Qt. Or is it just a support thingy?
Are you saying Microsoft has credibility? That's almost as unbelievable as the Mindcraft "study!"
Buzzing the information Superhighway at Warp speed
I agree that you would expect Mindcraft to want to clear their name by performing the test again, but it strikes me that your parenthetical comment, "But since microsoft isn't interested in a real test that isn't going to happen" may have missed the mark. If I were Microsoft, I would be very interested in a second, more fair test. I would be very interested to see that it didn't happen.
Food for thought.
I thought FUD meant the author/rep/person said *something* good about a product/issue/thing, but then brings a lot of negative things about. There weren't any good things said in the Mindcraft survey, well atleast none that I personally caught.
Saying something good about something, then attacking it gives a false sense of credibility. If someone only criticizes you and praises someone else, you will take this as "just another marketing thing" whereas if someone says "well you have foo, but we really personB had foo, bar and a lot of other features.
PS. I took my definition from Linuxtoday.com, actually.
//d
Don't forget that Linus Torvalds computer is something like a Quad-Xeon. I've got confidence that Linux on 4 processors would be pretty competitive. Better or worse than NT I don't know, but it wouldn't have to hide.
I think a contest would be better than a benchmark.
:)
In a benchmark their are great odds that the benchmark will be sponsored by one of the party (M$ in this case).
If you do a contest, like the best ratio performance/price : you benchmark the performance of all the competing teams and then divide by the price the team involved in the hardware (not the software because due to Linux openness many people would say Linux price biased the contest).
If someone do so you can have a M$ team which will try to tune NT to is best, a Linux/Samba/Apache team which will try to tune Linux to his best, a Novell team, a Sun team...
You could choose your hardware so small team can try to compete. Even companies unrelated to NT/Linux/Novell/Another OS could compete so that can do a lot of publicity to these companies if they are well placed in the results.
It would be a good thing so every people supporting an operating system and so knowing how to tune it would be able to compete and their would be a greater range of results than in a single benchmark.
Of course we now need to find somebody to finance the contest
"The obvious mathematical breakthrough would be development of an easy way to factor large prime numbers." Bill Gates,
Monty
Monty
Technically, it could be done. Whenever an HTTP request is sent to a server, one of the servers repsponse headers is the type of server. For example, Slashdot returns 'Apache/1.3.6 (Unix) mod perl/1.18'.
Maybe we can get google to do a count of this when they do their spidering. Then again, I would assume that the people who are releasing number of different types of servers are already doing something like this themselves. I could be wrong though, since I haven't checked.
If you can see through the reason why Microsoft sponsors and eventually published the Mindcraft report, you know exactly why the Linux community need to say "NO".
Suppose that your manager receive a phone call from a Microsoft salesman and want to switch their Linux server to NT (for whatever non-technical reason). Without other rationale, you, as a technical support, will nearly definitely says it will be difficult to manage, cause performance loss, requires more money, etc., and finally ask the manager to reconsider and abandon the decision.
But if the managers hold a copy of Mindcraft report and says that he need more performance and it cannot be accompolished with Linux, you won't have much to say.
Except that you also hold a copy of rebuttal telling exactly where the Mindcraft report went wrong. That is, it is not that Linux cannot be configured to beat a well-tuned NT. It can, and it can do it easily. Just that it requires some expertise which Mindcraft didn't have, and that such expertise will be easier to be obtained after the Mindcraft report.
This is exactly what the LinuxWorld article of Jeremy Allison is for. Essentially, it says "its not difficult to configure Samba much better and outperform a well tuned NT, here are the details. And we will find ways to teach system adminstrators about what's needed in the next release."
I apologise if anyone's suggested this before, but perhaps what's needed is a HOWTO for performance tuning of high-end servers, with a special mention that benchmarkers are advised to adhere to the information in the document.
The good thing here is that those "real" high-end servers run Un*x. There is alomost complete source-code comatibility (meaning you can just type make in most cases) from Linux / *BSD to those systems.
Looks like ESR's back again, fighting the good fight. I fail to see how
it's going to do any real good, though. In his piece he
nicely enumerated the many complaints people had concerning the tests, but
ultimately failed to get to the heart of the matter:
Xenophobia. Managers are scared of Linux because it's not NT, which they're used to.
My first reaction in this case would be to try a second test.
Unfortunately, tests like these are probably
somewhat expensive, but I think it would be worth the cost.
The distrib. co.'s would likely have to fund it, but fairness could be made explicit.
Despite the controversy, does anyone *know* if the tests were actually invalid?
I'm not a sysadmin, and I don't run Linux, so I can't comment.
Everyone criticizes the method, but what about the results?
Regards,
--
JCA
I don't have a whole lot of experience with Suns and I have zero experience paying for them, but....
When I read the report, the first thing question that popped into my head was, "If I had the money to by a quad-Xeon Intel, why would I? Wouldn't it be smarter to put the money into a Sun?"
It seems to me that quad-Xeons are getting into the upper range of Intel-based machines. Why not go for a mid-range (low-end?) Sparc-based system. I'm sure Solaris would spank NT in uptime (and probably speed), and I would have the ability to scale up even further if necessary.
SMB would be a non-issue to me. I would just use an NFS client on the NT workstations (I've done this before).
I'm thinking more in terms of a real-world "Here is your budget. Get the best you can afford" situation. I'm a missing something here?
Save the whales. Feed the hungry. Free the mallocs.
As long as they're credible with the deep-pocket types, they don't have to be credible with the techie/geek types. Of course, if the techie/geeks start gaining credibility with the deep pockets at the expense of Microsoft...
--The basis of all love is respect
The interesting question to me isn't "How much power can you get out of hardware X with OS Y", but rather "How much hardware do you need to throw OS Y on to do job Z".
From what I've been reading, NT does better SMP than Linux does. Frankly, Linux doesn't need SMP nearly as badly as NT. If uniprocessor Linux can do the same job as SMP NT, who cares how good or bad SMP Linux is?
--The basis of all love is respect
I do not see the Linux community standing idle and taking it, ESR's post is a fine example of this. Note that benchmarks like mindcrafts were done with NT vs OS/2 over and over with no real response from IBM. The OS/2 users who protested were categorized as "zealots" and written off. On Compuserve, false user accounts (see "Barkto") were alleged to have been created to depict "real users" who then went on and on about serious OS/2 errors that "trashed my hard disk" and "my backups", ad nauseum. (Such reports were then published in PC Week, Infoworld, Computerworld to drive home the FUD).
This, of course, completely ignores the fact that:
1. Barkto was a complete and utter asshole.
2. He was fired for what he did.
3. Any other MS employee doing what Barkto did, or otherwise found to be deliberately misleading the public will be terminated on the spot.
Coming soon - pyrogyra
Ideally the best way to have an impartial test is to allow Microsoft partisons to tune the Microsoft machine and Linux partisons to tune the Linux machine.
While this may simply be an exercise in who is the better system administrator it would at least insure that nobody purposefully mis-tweaked something.
Ideally both parties should also have access to the same help facilities that any end user would have access to.
This is the most intelligent comment I've seen on the matter so far -- and the best way to ensure a fair test.
Though someone would have to oversee things to make sure that both sides don't use dubious hacks to get more performance.
Coming soon - pyrogyra
So, who is getting terminated over the mindcraft incident
Show me someone who's behaving badly in online forums and/or making ludicrous claims about competitors *without admitting their relationship to Microsoft*, and I'll show you someone who's getting terminated. It's as simple as that.
Coming soon - pyrogyra
There's someone who does this already it's netcraft (don't mistake it with mindcraft)..
the url is www.netcraft.com and shows that apache kicks IIS's arse.
Not that their credibility is looking all that good as it is, but this can only hurt them in the anti-trust trial. Mindcraft was an "independent" lab that contradicted what Apple was saying about Microsoft trying to undermine Quicktime for Windows. Mindcraft released a fix that would make most of the problems go away, but I wonder if that fix wasn't given to them by Microsoft to patch Microsoft's code instead of Quicktime.
You don't want to fight FUD with FUD in that way. It's not just a matter of morality, either. Microsoft has clout because of their success as a big corporation with an established monopoly. They can afford to lose a little credibility by spinning a few lies. The Linux community has only one source of credibility -- that their stuff *works* -- and that's the very thing M$ is attacking. If you bend the truth and are caught, your credibility will suffer a lot more than Microsoft's. You'll be helping their FUD campaign, not hindering it.
Keep the high ground, folks. It's really in your best interests.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."
"Define test goals"?
"..satisfies the test goals."?
Isn't the goal of testing to reach unbiased, truthful results? Mindcraft simply takes the cash and criteria, and gives you the results you want. I couldn't sleep at night if I worked for a company devoid of any morals.
---
aaron barnes
part-time dork
---
aaron barnes
part-time dork
"I like cheese. Yum."
Yeah, that's the way I've viewed this affair myself. The constant backlashing at Microsoft really isn't helping much. The IT guys I work with are literally in bed with Microsoft. They take what ever Microsoft says as gospel and their software as manna. Many people in IT groups just don't look at servers and software like we do. They want to keep buying the newest servers and the newest Microsoft OSs. It dosen't matter what you show them about Linux vs. NT benchmarks are whatnot. They take this one study, funded by MS and carried out by a reputable testing group, and say "look, I told you so".
In the end Microsoft is going to win in the majority of markets simply because you can't get the "big brother" feeling from open-source that most people seem to get from Microsoft. Linux seems on the right path for that. As long as we as developers and operators continue to support Linux and support the groups that are working to produce standards and support, the future looks pretty good.
Madhatter --It's no wonderland out there.
On Mindcrafts Reports Page They say "our .. reports give you the peak performance of various systems"
Many things were pointed out here and elsewhere, that changes they made *reduced* the performance of the Linux setup
Microsoft addmitted, via Ian Hatton, that the "NT box was better tuned than the the Linux Box
So, here we have a report on performance, where is seems to be clear that the linux box was running nowhere near its performance peak, but Mindcraft by publishing this report are certifing that it was at "peak performanance"
If it can be clearly shown that they are certifing something that is false, could they be forced to retract the "certified" report?
Now, *That* would be nice PR
Exigo spamos et dona ferentes
Hatton also admits that the Linux system would have performed better if it had been better optimised. "Having said that, I must say that I still trust the Windows NT server would have outperformed the Linux one."
Trust? Obviously you don't have too much confidence in NT, Ian.
We're going down, in a spiral to the ground
I see lots of calls for doing another benchmark that's "fair" to prove the Linux system superior. The problem with that is that it could never be "fair" if carried out by Linux partisans. Even if it were, likely there would be one missed tweak which would throw the whole thing in doubt.
:-)
Instead, why not fight FUD with FUD? Mindcraft claims the study's still valid even though the systems weren't tweaked equally. If that's the truly the case we're home free! Do a study designed to show how *badly* an NT server can be tweaked, and publish the results. As long as you promote the results as "just as valid as the Mindcraft benchmarks", you are being perfectly honest.
So next time MS throws out the invalid Mindcraft survey (NT 2.5 times better), don't attack the survey. Just throw out the new Linux survey (Linux 153 times better) done using the "Mindcraft method".
So here is one of that persons that brings
the Linux community into a bad light.
As i know myself there are always several
news messages or press releases concerning
a special event. The mentioned URL is in
fact the report itself. There its mentioned,
but probably not in the one that ESR pointed
to. How many reporters had the time to look
deeply into that report?
Bye, AlexS.
--- Linux has no limit !!!
What about a real shootout party?
The machines are required only once.
Invite some important guys, like:
- Microsoft
- Novell
- SGI
- BSD folks
- Linux
and others important.
Why not do testing on
- single intel
- dual intel
- quad intel
and proprietary systems?
Such an event would be very intersting.
--- Linux has no limit !!!
It's not that simple. Currently the biggest problem with the Linux SMP implementation is the IO subsystem. (SCSI and IDE, etc.) The problem
is that this subsystem isn't SMP safe. So whenever the kernel enters this portion, it grabs what is known as the kernel lock. Thus disk activity
can only happen on 1 processor at a time.
Issues of I/O, interrups and memory access are very non trivial using a shared memory SMP approach.
Indeed for the kind of web and file serving which Mindcraft were bench marking using an SMP machine is not the best way to do things anyway. There is
simply no need to couple the processors tightly, giving each processor it's own PSU, mboard, RAM (1G) and NIC (and a smallish HDD to hold the OS)
will solve most of the memory, I/O and interrupt
issues. The only remaining issue is connecting
to the RAID array, though connecting this to a
server and using 100M ethernet will be quicker
than SCSI.
So how do 5 (upping to 7 if you want the redundancy needed for real world situations) Linux machines compare with the NT "monster". Even when you factor in the extra costs of extra networking
equiptment (including a watchdog driven automatic
patch panel), the extra space, etc (How expensive
is the system not working, too.)
drop the Apache RPM's and compile it.. Specificaly with the PGCC compiler, I have heard of 30+% speed improvements with it. you could even go for the PGCC based distro stampede for your base instal
You would be better off with the latter option, becuase it's not just the binary itself which matters but it's also the shared libraries and the kernel. In you went the first option then you'd end up recompiling your whole distribution.
Though remember for the second option the distribution would need to be built for the processors you were using, i.e. does "stampede" optimise for a Xeon? (Or at the very least have a Linux system built with the same optimisations as were used to build NT Enterprise...)
The other thing with the settings is that IIS was set to only write it's log to disk in "block mode" (with large blocks) whereas Apache usually writes its log line by line (which is the norm for logs, since it ensures that logs get written and enables
real time monitoring of the log.)
A couple of the big companies that are pushing Linux (Oracle for instance) should have put on a new benchmark session with SEVERAL test systems. They should have a variety of hardware, from low to very high-end.
Get some NT and Linux gurus to tune the hell out of their respective OS (lets make this damn fair). Run a series of tests that reflect theoretical and real world conditions.
Get the tech media involved as much as possible and be very honest about it.
Write it up in several formats from guru to PHB level (lots of pretty charts).
Let the results speak for themselves. I would love to read them.
Someone needs to perform this test on a Dell PowerEdge 6300/400 server (anyone have one?:).
Even if NT can beat Linux fairly, MS could not take the chance of a close race. Since their server is much more expensive, they had to "prove" that there system is worth the cost.
I did notice that there was no bench mark on down time! I wonder how many times the system crashed before they got it working.
Steven Rostedt
-- Nevermind
truly beutiful, completely evil
I love it . . . make it so
x.x = -1 no, your just imagining it . . .
Doesn't it seem just a little odd that Microsoft is so blatant about this. Reminds me of a certain "hacker" type that likes to leave an obvious cookie trail to let the detectives find him and then says "yeah so, now what are you going to do?"...
Microsoft may be trying to snowball the general public, but I don't think they even care if we know what they are up to or not. Them's politics for ya.
This space for sale
The Mindcraft study proves that you might get adequate perfomance from NT if you lavish hardware on it. Quad Xeons and a gig of RAM could get the NT servers on my network up to speed. Perhaps NT's design flaws are intended to push us toward high end machines and pricey upgrades!
The point has already been made, but I will make it again -- I get better performance from a cheap PC running Linux than I do from the expensive boxes I have running NT.
[but what room do I have to talk? I got sick or RedHat and decided to try out FreeBSD for awhile.]
The above article implies Mindcraft concealing their email address was a bad thing.
I don't know why it was done but it may have been done so they were treated like an average user and not given special attention or extra help.
You think Jerry Pournelle gets the same level of support we do?
Regards,
David.
How about this for a nice little conspiracy theory?
(1) MS arrange for a benchmark that will annoy the heck out of the Linux community.
(2) The Linux community respond. Inevitably, the less evolved amongst them will flame the publishers of articles about the survey with stupid threats of hacking and virus attacks. There will be many letters to trade publications by illiterates, referring to "Windoze".
(3) MS ensure that publicity is given to these reactions, thus associating Linux with that sort of reaction from that sort of person. Just what is needed to promote the OS amongst professionals.
In short, the mindless amongst Linux users may have done a lot more harm than the benchmark itself.
Good idea Iffy. The natural extension is a public contest -- a "Battle of the Operating Systems". You get a credible and impartial organization to oversee the contest. A magnanimous gesture I think the Linux community should make is to allow Microsoft to choose the hardware setup. Make it that much harder for them to blink.
Let them put up their best against the Linux best and settle this thing once and for all.
About the same as Linux vs DOS look like on a 286.
Such a test is pretty useless. It won't convince anyone to go Linux in their company, no more than it would when you port Linux to a toaster. Companies buy using either "we have so much to spend, what's the best we can buy for it?" or "we need this and this, what is the best deal we can get?". A 486 will seldom be an option.
--- Abigail
Ian Hatton didn't say "A very highly tuned NT server was pitted against a very poorly tuned Linux server." That quote was an excerpt from an email received by Microsoft.
I remember that shortly after the report was published MindCraft semi-officially said something to the effect that : "if we were to run the test again....we would not make those optimizations" (e.g. the ones that slowed samba down).
My question is: Why cant they do it again? Just do the tests again....
I realize it will be expensive. But someone paid for it originally and came up with flawed results. Most companies would be looking at how to do it right the second time instead of saying
"Well ya know if we were to do it again we would not screw it up (But since microsoft isn't interested in a real test that isn't going to happen)"
Most companies do their best to cusion bad publicity. But microsoft seems to be proving time and again that ANY publicity is good.
Even if its bad.