Slashdot Mirror
ESR and the MindCraft Fiasco
Microsoft's latest FUD (Fear, Uncertainty and Doubt) tactic may be backfiring.
A 21 April ITWeb story reported results by a benchmarking shop called Mindcraft that supposedly showed NT to be faster than Linux at SMB and Web service. The story also claimed that technical support for tuning the Linux system had been impossible to find.
Previous independent benchmarks (such as "Microsoft Windows NT Server 4.0 versus UNIX") have found Linux and other Unixes to be dramatically faster and more efficient than NT, and independent observers (beginning with a celebrated InfoWorld article in 1998) have lauded the Linux community's responsiveness to support problems. Linux fans smelled a rat somewhere (uttering responses typfied by "Mindcraft Reality Check"), and amidst the ensuing storm of protest some interesting facts came to light.
- The benchmark had been paid for by Microsoft. The Mindcraft press release failed to mention this fact.
- Mindcraft did in fact get a useful answer to its request for help tuning the Linux system. But they did not answer the request for more information, neither did they follow the tuning suggestions given Also, they forged the reply email address to conceal themselves -- the connection was made after the fact by a Usenetter who noticed that the unusual machine configuration described in the request exactly matched that of the test system in the Mindcraft results.
- Red Hat, the Linux distributor Mindcraft says it asked for help, reports that it got one phone call from them on the installation-help line, which isn't supposed to answer post-installation questions about things like advanced server tuning. Evidently Mindcraft's efforts to get help tuning the system were feeble -- at best incompetent, at worst cynical gestures.
- An entertainingly-written article by the head of the development team for Samba (one of the key pieces of Linux software involved in the benchmark) described how Mindcraft could have done a better job of tuning. The article revealed that one of Mindcraft's Samba tweaks had the effect of slowing their Linux down quite drastically.
- Another Usenet article independently pointed out that Mindcraft had deliberately chosen a logging format that imposed a lot of overhead on Apache (the web sever used for the Linux tests).
So far, so sordid -- a fairly standard tale of Microsoft paying to get exactly the FUD it wants from a nominally independent third party. But the story took a strange turn today (22 Mar) when Microsoft spokesperson Ian Hatton effectively admitted [8] that the test had been rigged! "A very highly-tuned NT server" Mr. Hatton said "was pitted against a very poorly tuned Linux server".
He then attempted to spin the whole episode around by complaining that Microsoft and its PR company had received "malicious and obscene" email from Linux fans and slamming this supposed "unprofessionalism". One wonders if Hatton believes it would be "unprofessional" to address strong language to a burglar caught in the act of nipping the family silver.
In any case, Microsoft's underhanded tactics seem (as with its clumsy "astroturf" campaign against the DOJ lawsuit) likely to come back to haunt it. The trade press had largely greeted the Mindcraft results with yawns and skepticism even before Hatton's admission. And it's hard to see how Microsoft will be able to credibly quote anti-Linux benchmarks in the future after this fiasco.
In the Performance Testing section on their web page, second paragraph they say flat out:
"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."
Since they say Microsoft sponsored the test, we can replace "you" with "Microsoft." So they worked with MS to define the test goals (NT is 2 or more times better than Linux). Then they put together the tools to do that, hacking the registry and all to beef NT up, slowing Linux apache/samba servers. And finally, report the results back in a form that satisfies the test goals, lo and behold NT is 2-3 time faster than Linux. Such a surprise, right?
Isn't that what I said in the "Mindcraft Reality Check"?
There are valid limits to Linux scalability, problems that need fixing, and honest benchmarking can help us find those limits. Unfortunately, Mindcraft's benchmarking was so flawed by misconduct and poor judgment that it is not useful for that purpose.
BTW, I do agree with the Microsoft spokesman who said that he was certain that NT would have come out on top even with an honest test. I suspect the SAMBA results would have been quite competitive, within 3% (more or less) of the NT numbers, but the Apache server has never been known for its static file serving speed (though mod_mmap_static may change that!). On the other hand, there is a big difference between the 5%-10% advantage that I bet Mindcraft would have found, and the ridiculous numbers that they actually reported. They actually shot themselves in the foot here, because if they'd reported the real numbers, the Slashdot Crowd would have howled, but Jeremy Allison and other technical heavyweights would have stayed on the sidelines working on fixing the problems found, and the media would have ignored the Slashdot Crowd.
Just count it as another example of Microsoft Arrogance (tm) outweighing their good sense. It's amazing how such bright people can do such stupid things.
-- Eric
Send mail here if you want to reach me.
Yes the study was flawed. But I remember a comment by Matt Welsh in which he said that Linux is not properly represented in the high-end machines.
That is a natural consequence of the open source
development. Many of the features of Linux are there because some users needed them (scratch an itch, as ESR says...)
So, bearing in mind that there aren't many Linux
users with quad xeons with 4 Gb Ram, it's only natural that issues relating with that kind of machine have a lower priority to the Linux user community
I've seen very little comment that Linux might actually really be slower on a 4-way. I would be disappointed, but not amazed if Linux were slightly slower on a 4-way given the maturity of NT SMP compared to Linux.
I would like to know if Linux does scale as well or better than NT with 4 and 8 processors -- both systems properly tuned and using the same webserver. When that question is answered, I'd like to know what to expect in the future. Is Linux going to leave NT in the dust, or will this be the key niche ground for NT servers that Microsoft will defend to the end, and Linux will never conclusively defeat?
You're only right on the kernel front - that hasn't really been used enough on high end machines AFAIK.
But both Samba and Apache have been heavily tested on very high end servers. The Samba crew have even been heavily involved in making Samba fast on high end servers.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
Please note that the dejanews reference that ESR links to is quite wrong. The presence of %h does _not_ cause host name lookups under Apache - only the directive "HostNameLookups on" causes that to occur. I don't believe this to be the case.
I strongly believe however that their httpd was running under inetd, and that would cause the effect they saw.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
At a large company I'm working with we're trying to prove to the phb's that Linux is a good thing. The mindcraft study set us back a ways. So what did we do? We did our own tests.
/proc fs to speed things up. Stock kernel options selected from "make xconfig". Apache was the apache+mod_perl srpm found on redhat/contrib, compiled with no configuration changes. We didn't test NT on this box - we were trying to compare against Mindcraft's results.
Server:
- Hand built by our best hardware guy
- PIII 500 (single CPU)
- Adaptec 2940U2W SCSI Adapter
- 10,000 rpm LRW drive. 1 drive only.
- 100Mb/s network card
- 256Mb PC100 RAM.
- Linux 2.2.6, upgraded from stock Linux-Mandrake box
- Apache 1.3.6, configured for best performance.
No changes to the
Want to know the results so far?
Well, we can get about 2200 requests per second out of that box. The Quad Xeon NT box that mindcraft tested got 3700 requests per second at its maximum rate. We are at very early stages so far, and I think I can squeeze more out of the box by dumping Apache and using thttpd or something else that uses a threaded model. But since this is to be a pure mod_perl box I don't think that's important.
Things to remember:
The mindcraft server had 1Gb of RAM.
The mindcraft server had RAID (RAID/0 I believe).
The mindcraft server had 4 10/100 network cards.
We're so far pretty pleased with our little Linux box... It was a fair bit cheaper than Mindcraft's server....
Matt. Want XML + Apache + Stylesheets? Get AxKit.
From the linux-kernel list:
Mindcraft also used the v0.92 MegaRAID driver. An SMP race condition was fixed in v0.93 which was almost certainly available from the AMI web site long before the Mar 10-13 test. So SMP NT "beat" a non-SMP Linux on a quad-Xeon server. Big hairy deal.
Original poster is "Doc" Savage. Original post 14 Apr 99.
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying "the study was paid for by Microsoft" and that "a very highly-tuned NT server was pitted against a very poorly tuned Linux server".
That is, the claim attributed by Eric to Ian Hatton was really made by reacting Linux supporters.
What Hatton did admit, was:
"Microsoft did sponsor the benchmark testing and the NT server was better tuned than the Linux one."
This isn't much, but it is sufficient. Hatton admits that "the NT server was better tuned than the Linux was" and even without adjectives that invalidates the report.
-cjr
Remember that a certain number of sites really need big-iron servers (hey, slashdot isn't exactly gentle on its hardware, although in that case I suspect database performance may be more of an issue), and even when they don't it's the results from high-end server tests which impress the management the most.
Having seen Linux/SMP in action and made some subjective judgements I'm quite confident that, properly configured, it ought to scale fairly well onto hardware of the class Mindcraft were `testing'. But it would still be nice to have some number...
Good summary.
Seems to me that what we really need is a bench marking rebuttal; is there another
benchmark going on? I saw that in Jeremy Allison's article he was working with PC Week,
does anyone else know any other active bench marking going on?
I think that the only way to prove against FUD is education, bench marking can go a long
way.
I have about 7 Linux servers with no down time, great performance on lesser hardware then
my commercial servers in my company, that should be proof enough; but my pointy haired
boss still asks "Why not NT?". I do not need any more fuel for that fire.
We need Benchmarks on larger servers, with more memory, RAID, and a high-end server
guide.
"Think of it as evolution in action."
Good article Eric, and I'm proud of the Linux community for the way we've reacted to the Mindcraft "benchmarks". I think the Linux community fought back, but in a mature balanced way. I think it is important that we continue to do so, and try not to appear too much like reactionary fanatics which somtimes happens too.
Also, I must say I really learned a lot by following the debates. Next time I need to install Apache and Samba you can bet I'll be referencing the responses to Mindcraft to see the proper way to optimize this stuff.
Kudos and thanks to the Linux Community!
Without being too fanatic, I think that we all should inform any magazine publishing the Netcraft results (and thus concluding Linux is sh** compared to NT) of the facts and unreliability of this survey.
Admit, "M$ guilty of consumer fraud" is a better headline than "NT beats Linux on all fronts".
Ummm That should be Mindcraft right?
This is an alramingly common mistake - poor netcraft.
Well, I think this incident has damaged microsoft's credibility, but that's beside the point. Microsoft isn't talking to us, the technical community. They aren't trying to convince us that NT is better. For those of us in server closets, in the operations center, and in system administration - we already know the truth. We don't need benchmarks and statistics to tell us NT is unreliable.
The plain fact is, Microsoft did this to appeal to middle/upper-management, not us. They need to keep feeding them reasons to keep their NT investment without looking stupid. Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.
--
What strikes me about the entire "Mindcrap Affair" is the resulting coverage. I can recall seeing only one press article covering the original story (the "benchmarking"), but I have seen many press articles covering the resulting controversy. Of course, my impression may be biased because I take pointers to news stories from Slashdot and Linux Today. On the other hand, I have done some looking outside of the "linux community", at sites such as CNet News, and they definitely seem more interested in covering the fiasco than in the original benchmark. Maybe these sites too can smell a rat.
--JT
The 'Rush Limbaugh' principle is a very valid point, especially in this context. Don't forget the target market for this study is Microsoft partners and WinNT-based shops.
Aside from all the meaninless numbers (who cares if your web server can saturate a 100BT line with static pages!), the study drives home an important point to NT Administrators - If you've invested in a high end IIS system, and you've got it tuned, there's probably no good reason to switch that box over to Linux. If the Linux box was tuned correctly, I doubt the difference would be that great performance-wise.
Of course, the study didn't address stability, which is the number one problem with IIS.
--
Business. Numbers. Money. People. Computer World.
I just took a look at the linked article written by Jeremy Allison of Samba.
/proc filesystem.
A few interesting points -
* In the often referred-to ZD Samba versus NT benchmarks (where Linux+Samba wins), the Samba/Linux configuration was tuned by a Samba team member. Objectively, this makes the ZD benchmark actually less valid as the Mindcraft study, because as far as we know, a Microsoft-employeed SMB developer wasn't actually there tuning the server.
* Tuning Linux properly involves cryptic commands such as:
echo "80 500 64 64 80 6000 6000 1884 2" >/proc/sys/vm/bdflush
echo "60 80 80" >/proc/sys/vm/buffermem
While I'm sure these commands are documented somewhere, this sort of tuning makes the NT Registry Editor look like a model user interface. Low level tuning like this really needs a nicer front end, or preferably, a daemon which monitors system activity and dynamically tunes these settings.
It sounds like the Mindcraft study has been a kick in the pants for the Linux community to get some high performance documentation together. I'd like to see a nice How-To which lays out some of the more obscurantist tricks such as echoing strings to the
--
Business. Numbers. Money. People. Computer World.
The truth or falsehood of the Mindcraft study is irrelevant to its intended audience. The point is to give NT "believers" something to quote in arguments, that's all. It's the Rush Limbaugh Principle. In a disagreement, it's helpful to have official-sounding statistics to back up your point. It doesn't matter where they came from, and it doesn't matter whether they're even remotely accurate. What counts is that somebody "important" (read "well-known") said it in public, which "validates" it. This "validation" isn't about truth. What it means is that the proper forms have been followed, and so it's acceptable to introduce the "evidence" in an argument. What's being offered is not evidence in the conventional sense, but the appearance of evidence, or the outward form of evidence. In poker, what does the four of diamonds mean? It means the four of diamonds. It's pure, disembodied symbol.
Disagreement and debate in our culture (especially on the net) isn't a whole lot less stylized (nor a whole lot less predictable) than Noh drama. You have to play by the rules and observe the forms. The content of the Mindcraft study is arbitrary. The study is a signifier, or token. A yacc parser says, "hey, this token is a function, hey, that one's an operator." The actual content of the token is not significant; what matters is what kind of token it is.
Everybody should learn at least a bonehead popularized minimum of semiotics (which is all I know, obviously
While we're at it, let's be honest with ourselves: How many of us are going to check Eric Raymond's facts for ourselves -- even to the minimal extent of clicking on the links he provides? And how many of us who don't check the facts are going to run around repeating them? Quite a few, probably. Dammit, I think Raymond's right on the money with this, and I'm confident that he's done his homework -- but I don't have the time to go about proving it. As far as many of us are concerned, Eric has given us a counter-signifier. Some "good spin" to match against the "bad spin". (That makes it sound dishonest, but IMHO if the "good spin" is factual and accurate, then "good" is a perfectly reasonable thing to call it.)
Think about it.
(Experienced sysadmins are a bit of a special case here. They can judge for themselves. The Limbaugh Principle applies mainly to people who are arguing in an area outside of their field of expertise -- I don't recall who it was who said that "every man is gullible outside his specialty", but it's true even of the best of us.)
"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"
"Christianity neither is, nor ever was a part of the common law." --
You have taken a quote your first quote COMPLETELY out of the context of the article.
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying 'the study was paid for by Microsoft' and that 'a very highly-tuned NT server was pitted against a very poorly tuned Linux server'. In response, Ian Hatton, Windows platform manager at Microsoft SA, says these comments are valid."
I think a contest would be better than a benchmark.
:)
In a benchmark their are great odds that the benchmark will be sponsored by one of the party (M$ in this case).
If you do a contest, like the best ratio performance/price : you benchmark the performance of all the competing teams and then divide by the price the team involved in the hardware (not the software because due to Linux openness many people would say Linux price biased the contest).
If someone do so you can have a M$ team which will try to tune NT to is best, a Linux/Samba/Apache team which will try to tune Linux to his best, a Novell team, a Sun team...
You could choose your hardware so small team can try to compete. Even companies unrelated to NT/Linux/Novell/Another OS could compete so that can do a lot of publicity to these companies if they are well placed in the results.
It would be a good thing so every people supporting an operating system and so knowing how to tune it would be able to compete and their would be a greater range of results than in a single benchmark.
Of course we now need to find somebody to finance the contest
"The obvious mathematical breakthrough would be development of an easy way to factor large prime numbers." Bill Gates,
The interesting question to me isn't "How much power can you get out of hardware X with OS Y", but rather "How much hardware do you need to throw OS Y on to do job Z".
From what I've been reading, NT does better SMP than Linux does. Frankly, Linux doesn't need SMP nearly as badly as NT. If uniprocessor Linux can do the same job as SMP NT, who cares how good or bad SMP Linux is?
--The basis of all love is respect
You don't want to fight FUD with FUD in that way. It's not just a matter of morality, either. Microsoft has clout because of their success as a big corporation with an established monopoly. They can afford to lose a little credibility by spinning a few lies. The Linux community has only one source of credibility -- that their stuff *works* -- and that's the very thing M$ is attacking. If you bend the truth and are caught, your credibility will suffer a lot more than Microsoft's. You'll be helping their FUD campaign, not hindering it.
Keep the high ground, folks. It's really in your best interests.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
Hatton also admits that the Linux system would have performed better if it had been better optimised. "Having said that, I must say that I still trust the Windows NT server would have outperformed the Linux one."
Trust? Obviously you don't have too much confidence in NT, Ian.
We're going down, in a spiral to the ground
I see lots of calls for doing another benchmark that's "fair" to prove the Linux system superior. The problem with that is that it could never be "fair" if carried out by Linux partisans. Even if it were, likely there would be one missed tweak which would throw the whole thing in doubt.
:-)
Instead, why not fight FUD with FUD? Mindcraft claims the study's still valid even though the systems weren't tweaked equally. If that's the truly the case we're home free! Do a study designed to show how *badly* an NT server can be tweaked, and publish the results. As long as you promote the results as "just as valid as the Mindcraft benchmarks", you are being perfectly honest.
So next time MS throws out the invalid Mindcraft survey (NT 2.5 times better), don't attack the survey. Just throw out the new Linux survey (Linux 153 times better) done using the "Mindcraft method".
I remember that shortly after the report was published MindCraft semi-officially said something to the effect that : "if we were to run the test again....we would not make those optimizations" (e.g. the ones that slowed samba down).
My question is: Why cant they do it again? Just do the tests again....
I realize it will be expensive. But someone paid for it originally and came up with flawed results. Most companies would be looking at how to do it right the second time instead of saying
"Well ya know if we were to do it again we would not screw it up (But since microsoft isn't interested in a real test that isn't going to happen)"
Most companies do their best to cusion bad publicity. But microsoft seems to be proving time and again that ANY publicity is good.
Even if its bad.