Slashdot Mirror
Mindcraft Fun Continues
LinuxOnEveryDesktop sent us Mindcraft's comments on a
third benchmark that
will be open to a wider array of Linux Experts (the second
benchmark took tips from Linus which raised a lot of eyebrows: my
favorite being 'did they require Bill to be involved too?')
There are quite a few restrictions, but overall it seems
like a solid chance to show what Linux can really do. Check
it out.
If IIS with these configs can't beat the "swiss army knife" Apache then IIS is really bad ! I mean look at what those configs actually meant
p p05_5xtf.htm
:)
These were some that i bothered to investigate.
ObjectCacheTTL = 0xffffffff.
This it to be used if all contents fit into memory and is static (hmm, and guess what the benchmark measures...yes, static performance.) It supposedly turns of the cache "garbage colllector". See http://msdn.microsoft.com/library/sdkdoc/bdg/bdga
OpenFileInCache = 0x5000 = 20480 (max number of cached open file handles) Well caching up 20 k open filehandles should speed up serving static pages
ListenBackLog = 200 (couldn't find it on MSDN), presumably means that listen() can have 200 pending connections.
And what is my point ? well IIS was totally geared up to handle a website that fit in it's primary memory (anyone how big is the "site" in webbench?)
but,pardon me if'you've got a quad xeon running some kind of "enterprise" website wouldn't the amount of data be _much_ more than 1(or 4) GB ? And would hardly be static either, but generated from scripts,databases and whatnot ?
In humble respect of all computer gurus out there, i hope i haven't got this totally wrong
/Micke
The fact that Linux has done this well is interesting, but really folks, take a look at the test -- it is specifically designed to be a test that Linux cannot win at this current time. 4-way processing. RAID support. And with third party utilities which were created for their usefulness, not their ability to shine on a benchmark.
I don't think we're going to find a 4x boost in the results. One has a good argument saying that Linux is *not tunable* for the reason that it has a decent design with reasonable defaults and there's not much that needs to be tuned! [If someone does a partial quote of that last line and puts it in a magazne, they are vermin.]
BTW, a few semi-related notes. If I was going for performance on a web server with Linux, I would chose an architecture far different from a monolithic 4-processor, 4-network box with RAID storage. I would go with multiple simple systems, which would allow me a path to grow that is not tied to upgrading to expensive bleeding edge technology.
If I wanted to do a performance based web server, I'd put all the files in a memory based filesystem. (Or have sufficient memory to make sure that I'm going to get a high cache hit rate.)
If I had to use drives, I'd use solid state drives (the kind with built in battery and automatic disk backup) for ultimate performance.
I would spread the documents (images, html, etc) across several machines. Additionally, I might have "www.myhost.com" assigned to several IP addresses so that each new connection is semi-randomly distributed to a new machine.
There would be no need for 4 x 100Base-T connections for each host. (BTW... are these going to the same network or different networks? Meaning... is NT doing *IP trunking* with these (all four 100base connectors "striping" output on the same network), which Linux won't? That could explain a 4x difference and wouldn't it be funny if one simple feature is all that this test demonstrated?)
The design given for the "web server" and "file server" aren't really good hardware choices for the job. BTW... two different services on the same box? Aside from the fact that it is bad design on the NT side, the two functions really require different configurations.
And unlike in the previous hatchet job, we are going to be right there watching as they tromple over our OS and publicly defame it.
We are going to be standing there with our mouths open saying, "What happened?"
Folks, this is a battle we can only lose. Bruce has his LIVELIHOOD at stake here. NT is not going to lose no matter how open it may seem. Whatever it was that happened in that second test Mindcraft did gave them the confidence to do this test so they could appear to the public as "fairminded" and "open".
It ain't so. What can we do to prevent this? I hope noone important falls into this trap. It could be a black day for us all as the press trumpets "NT prevails in open tests where linux gurus try their best!"
I'd like to point out a few issues with this: 1) Motivation; in point #1, Mindcraft says the purpose of the test is to verify the results of the second tests. No Linux experts ever even had the opportunity to view the second test's results. Why are we rerunning test #2 which no one has had the opportunity to see or critique? Why not retest #1, the one that everyone says is false and inaccurate? 2) Machine used; this machine will not be the Dell PowerEdge that was used for the first test. It may be a machine that was specifically chosen for it's weak Linux drivers or other reasons. 3) Mindcraft configuration; why must the Linux experts use the configuration that Mindcraft used? 4) Microsoft's ability to use the results in press releases; I admit, this part is the reason why the tests are run in the first place; but why is this starting to look like a press campaign? Microsoft has incredibly tricky attorneys; why should we play their game? 5) Will the Linux experts have the opportunity to edit the 'joint' press release that includes quotations, possibly out of context, from the Linux experts? --Curious in Atlanta
It's a little obvious when you think about it. Mindcraft says that the hardware configuration is "set in stone". And, in fact, when they posted messages on the newsgroup, people told them that they needed to change their hardware configuration... but they won't. The key to this entire test IS the hardware.
Where did this specific hardware configuration come from? Microsoft. And Microsoft didn't just pull this configuration out of thin air. They've been doing all sorts of internal benchmarks with Linux systems to see what kind of numbers they could get. In their tests, I'm sure they've come across many favorable Linux comparisons.
But, as expected, they've found a few sub-optimal configurations for Linux which NT does well with. Microsoft has run many tests, and found the hardware that works the worst with Linux but good with NT. This is what explains the email from a Microsoft email address regarding the Xeon configuration... the email that Mindcraft said they did not send. It was done by Microsoft's internal testing.
Once they have the bad configuration, they need to send it off to another third party for "independent verification". Which, I believe, it isn't Mindcraft that was responsible for the low numbers... they were handed a benchmark that Microsoft ran ahead of time and already knew the results for. They just needed an outside party to go discover it for themselves.
This creates a simple smoke-and-mirror effect. Microsoft isn't blamed for this... Mindcraft is. And it disguises the real issue... it isn't software/OS tuning. It is hardware de-tuning done in advance by microsoft.
Consider that each piece of the system is pretty much a non-optimal configuration for linux. CPU, RAID disk, and probably the network. Probably one of these pieces (say, the 4x100base-t instead of a single gigabit ether) is really sub-optimal and replacing it would probably yield incredible results, but you've still got the additional handicaps.
But the Linux community has walked right into this one, thinking they can tune it out. Probably short of re-writing some kernel and driver codes, this piece of hardware isn't going to fly on anything *but* NT. (*BSD will also compare unfavorably.)
It probably is too late now to point this out -- they'll have the claim of "sour grapes" to use.
Looks like Linux did not come out on top in their second test, and Mindcraft is willing to bring in Linux experts to verify the results.
My biggest fear with this is not that Mindcraft will try some underhanded trickery, it's that the Linux community will not accept the results unless linux comes out on top. That is just as bad as if Mindcraft tailored its benchmarks to favor NT.
This open benchmark is a wonderful opportunity for the Linux community to benchmark itself. AFAIK, no controlled experiments on high-end servers have been done to see how Linux stacks up against the heavyweights. If Linux comes out on top, great. But if not, we as a community have to accept that and learn from the process.
This is best chance yet to discover the bottlenecks in the kernel and several critical pieces of software. IMHO, the Linux experts should not go in with the goal of beating NT. They should go in with the goal of squeezing every last bit of performance out of the machine and using the resulting data to fix the problems.
Mindcraft and Microsoft (!) are donating resources to the Linux community in an effort to help us improve the OS. Let's grab the opportunity!
...that "benchmarking" product A and product B, while getting paid from product A's manufacturer, using product A's consultants, and using product A's own testing fascilities...IS INHERENTLY BIASED!!! Regardless if product A is actually better than product B or not. My God, its so simple yet they can't figure it out!
Mindcraft has no credibility left. Worse, now they have an axe to grind because of how badly they were beat up in the mainstream press (ABCnews.com, Salon, Slashdot, etc). Any "benchmarking" they do from here on in should be ignored completely. If we can learn something from this, such as the need for better documentation, then at least thats a positive for us.
Sig (appended to the end of comments you post, 120 chars)
If that happens we should point out that the setup is tailored to NT's strengths and Linux's weaknesses. Ask people to consider whether the hardware might have been chosen with that in mind.
Also ask them if they're going to be running these 4 processor, 4GB, 4nic machines in their businesses and point out that these results say nothing about how NT and Linux compare on more mundane machines.
Ask people to consider the following: Tests like these are expensive. MS has money, the Linux community doesn't. If Linux beats NT on some machines and NT beats Linux on others, you're never going to see test results for the systems where Linux is superior.
Last but not least we need to show that we don't sweep problems under the rug, we fix them.
--
Fuck the system? Nah, you might catch something.
Alan Cox brought up a few points about the tests here:
http://linuxtoday.com/stories/5631.html
He mentions that we really should be benchmarking Zeus or a faster web server under Linux versus IIS if we want to find out how fast the OS can serve static web pages. IIS is the fastest web server for NT, so we should be able to use the fastest web server for Linux for the tests. If you want to compare NT versus Linux, then get the fastest web server for both.
If you want to compare capabilities, then use Apache, etc. Use the best product for the job.
Additionally, he does mention that we should use NT clients or at least a mixture of both, for the tests. At the company I did some consulting for, they are standardizing on NT. Microsoft's roadmap is all NT for the future. Why would you want to benchmark against a dead-end technology like Win9X?
Somehow I think these tests are rigged. Notice, Mindcraft only offered to rerun the tests after they ran a second one, which they didn't release the results of.
I think we need another party, that's neutral, to do some benchmarks, and take Mindcraft out of the picture altogether. They already admitted they fouled up, and they shouldn't be trusted to do the benchmarks again, because we will be giving them credibility they don't deserve.
Just my $.02
Ben
We don't exactly have millions or even thousands of dollars to be throwing around to cook books, forge benchmarks, and buy off reporters.
We react so violently to Mindcraft, not because we think Linux will always beat NT, but because the tests were so obviously rigged to produce marketing data as opposed to objective decision-support data. We are concerned that some people might believe this tripe because MS has a bigger marketing machine than we do.
We are justified in our wrath because we know MS has tried to do this before, with their disastorous Astroturf campaign, and their all-but-outright-fake demonstrations in the antitrust courtroom. And, as in a trial, if you throw enough money and time at a problem, you can prove (or disprove!) any criminal charge.
To say that the Linux community spreads FUD the way that Microsoft does is baloney. We spread polemic, yes. We p*ss of reporters when they p*ss on our OS, yes. We don't lie to people and try to mislead the gullible.
It doesn't matter that NT might be better in one contrived set of circumstances, because we can as easily contrive a test where Linux does better - why don't we repeat the Mindcraft test on a couple of 486/66's from the PC graveyard, and see what happens?
Mindcraft's credibility is gone in any case, because the press has very neatly seen to it that they shall always be perceived as a cog in MS's marketing machine. Whether this is correct or incorrect, it is a likely scenario, knowing the prevailing winds in Redmond, and that is why we righteously rant. Take it or leave it.
Mindcraft is participating in this third round of testing at their own expense, as they point out in bold text on the invitation page. They are doing this, I believe, to recover some of the credibility they lost by conducting their first benchmark in an extremely sloppy and biased manner.
And why are they doing this? Because for a company like Mindcraft their credibility is their cash cow -- if their test results can't be believed, no one's going to pay for them. So before we linux advocates get ourselves all worked up over the opportunity to prove what linux can do, we must ask ourselves: "what's the ultimate goal of this test?". Or perhaps that should be phrased "who is the ultimate audience of this test?".
The answer, I believe, is that the ultimate audience, the target, of this 3rd benchmark is Mindcraft's collective future customers, including, me must presume, those customers from which Mindcraft might expect repeat business. In a word, Microsoft.
So, while I'm encouraged by the news that linux will get another run at the benchmark, I'm not entirely satisfied that this will be a completely unbiased test. Although it's encouraging that Mindcraft has opened the test to tuning by linux experts, they still have dictated the structure of the test, and it seems to me that there's room there for bias.
BTW, I visited the Mindcraft web site shortly after the publication of the initial test results. Their home page included some text that read something like (paraphrasing here) we work with the customer to identify their test goals, then design a test to produce the desired results. In other words, Microsoft got what they paid for. It seems interesting now, in the aftermath of the Mindcrap Affair, that those rather damning words seem to have disappeared from their site.
--JT
There is a lot of talk about the Hardware RAID controller, and the drivers for it.
What about setting a $$ limit on the HW and let both "sides" choose the best HW for it's platform?
This is to produce a Linux test on the same Mindcraft problem space. That is, solve the same problem (same Web site, same clients, same network). But do it on a Linux venue, and with Linux-friendly hardware. Invite Microsoft, invite Mindcraft, invite the press.
Don't duplicate the hardware. Use inferior hardware, where "inferior" means "posts smaller numbers or costs less". But an "inferior" part may have better Linux drivers. Some people have complained that the RAID controller isn't Linux friendly. Replace it with a cheaper unit that is Linux friendly. Others say that SMP is irrelevant to this problem for Linux, since the processor isn't the limiting factor. Go for a single processor, the same speed as one of the SMP processors.
The point here is not to do a head-to-head, but to post better numbers with cheaper hardware. Then we challenge Microsoft--not Mindcraft--to beat the Linux benchmarks on the Mindcraft problem and their hardware or less. We replace the "fair benchmark" with a "competitive benchmark". We replace the (weak) assumption that the test team is trying to be fair with the (strong) assumption that each side will pull out all stops to post high numbers. This is how sports teams compete (rather than having the refs measure how hard you can throw or how well you can catch, they let you do your best and simply maintain a fair venue). IMHO, Linux engineers can put together a Linux server that is faster than any NT server that NT engineers can create.
People have noted that this is an expensive machine to put together. For you and I, that is correct. For a company with a vested interest in the results, it's an investment. It's a better investment when the company can use the machine for real world applications such as Web service. Relatively sizeable guns such as Red Hat or VA Research can stand to turn an indirect profit from pulling this sort of thing off.
A Linux/NT bakeoff would actually benefit both sides (!). Once we blow their doors off with our numbers, MS will probably find a way to post bigger numbers. Then we tweak Linux, and post bigger Linux numbers. If MS posts bigger NT numbers by making NT a faster server, we have at least forced MS to improve their product. If MS falls down the trap of optimizing NT for this bakeoff and pessimizing it for the real world, we introduce another bakeoff test. OTOH, MS can keep us from optimizing Linux for this bakeoff by making new bakeoffs at any time--that way, they help keep us honest.
--The basis of all love is respect
Linux, on the other hand, keeps more of its services in user space. This means it may suffer a little in this type of benchmark but it makes Linux an all around better application server. We are talking apples and oranges here.
To *win* Mindraft 3 we don't need to beat NT totally, just come close (which should be possible on a properly configured Linux box). This will show that the original test was a crock, and that Linux is still a much better price/performance buy.
Then we should go on to do REAL tests that show how Linux smokes NT's butt when you throw in CGI support and a mix of other net services.
Thad
The Bolachek Journals
This is nothing more than Mindcraft going into butt-covering mode...
Mindcraft isn't interested in an honest test, they just want to show
(to the media) that they know what they're doing.. it's a PR game
plain and simple.. they want Linux people to put their stamp of
approval on something that they have no real control over...
Why the time restrictions? the Linux experts aren't allowed to use
any patch that came out after April 20th... One of the main points
about the original test was the unsupported RAID card used... so if
someone were to magically release a patch tomorrow that made that card
run 3x as fast, they wouldn't be able to use it.
By the terms in the paper, as soon as someone from the Linux camp
joins, they're bound to put their name on the PR sheet. (which is
essentially just a confirmation of their second test, which they have
already run and won't show to anybody.) Since Mindcraft has STILL not
levelled the playing field, I strongly urge a boycott of this 'test'.
If Mindcraft REALLY wanted to have an unbiased test, they would invite
Redhat and Miscrosoft to sit down and draw up a mutually agreed-upon
hardware list; that way no side is at a disadvantage.
When you're at war, you don't allow your opponent to choose the
battlefield unless you have no other option. We have another option,
which is not to fight. By allowing someone else to choose the
hardware, the Linux side is at a disadvantage. Don't give them more
ammunition against us.