Slashdot Mirror
Mindcraft Fun Continues
LinuxOnEveryDesktop sent us Mindcraft's comments on a
third benchmark that
will be open to a wider array of Linux Experts (the second
benchmark took tips from Linus which raised a lot of eyebrows: my
favorite being 'did they require Bill to be involved too?')
There are quite a few restrictions, but overall it seems
like a solid chance to show what Linux can really do. Check
it out.
Based on the confidence Mindcraft is displaying you could guess that maybe NT beat out Linux in the second test and they at Mindcraft are sure that it will in the third test as well. It will be very interesting to see the results. It should lay to rest the bickering that goes on between the MS and Linux community on performance. Anyone know when the test results should hit the streets?
I would be intertested in seeing a test done on the Alpha PC164 platform as well since they both have native support for it too. Anyone know of such an effort or willing to do your own?
I'm all for this grudge match, though I would also like to see some benchmarks/shootouts between less expensive systems. I could be wrong, but I'm almost positive that the majority of Linux-based servers out there only have one processor and less than 1 GB of RAM. Also, the price of NT Server is a pretty important consideration on a system that has less than $2,000 worth of hardware...
As I understand it; NT handles quad processors better than Linux. Is this true?
Don't give them ideas. They're bound to troll Slashdot for 'em.
Only mostly offtopic, I saw somewhere in here a mention of a 1.0 version of the ami megaraid driver, does anyone know where i can aquire a copy of this? All i've been able to find is .96 in the kernel and on ami's site the ancient .92 i think.
Thanks
Will match their system as closely as possible, using any tweaks that Linus, Jeremy, etc perform. This is your chance to have the "ultimately" tweaked Linux system.
Mark
Linus et al had input on test two according to Mindcraft. I don't see this as an unfair test. If you want to take a product like Linux and compare it to a product like NT (tens of billions of dollars on research and production) on high-end hardware then the results are going to be less than favorable for the Linux community (regardless of who's hardware it is running on). Truth is that Linux is not scalable to the big boxes yet. NT is in part (but not totally). Of course NT will win just as Linux would win the same kind of competition if the test were to be run on a 486 with 16MB of RAM. These two OSes have different strengths and weaknesses. Linux doesn't do big SMP boxes well at all. Test #2 was test #1 with SOME input from Linux experts. Obviously Mindcraft saw the results of test #2 and thought they would totally redeem themsleves by allowing anybody from anywhere to do anything they could to get Linux to surpass NT on big iron. It's very clear what the results are expected to be in general but I would still like to know how big the margin is.
Sad, but what do we expect? Probably true for a lot of non-commercial ones too.
Actually the test should run better for NT Server if the clients were NT. And most desktops are still Win3.1/Win95. I don't think there will be any way a person is going to be able to call test #3 unfair or bias if it conducted as they say.
Yep. It could be honest... when pigs fly and I inherit the Brooklyn toll bridge.
The Linux and NT experts both will be there to watch. They will have a voice to be heard.
NT seems GREAT. It beat linux. It beat Netware. It beat Solaris. All for only marginal cost.
... according to the other tests on their front page.
... at least
Admittedly, i only read the summary results - but still, does this seem POSSIBLE? i've never seen NT running faster than linux (literally, NEVER) - but that could easily be me...
So, is NT vastly superior and i just missed it?
z.
Why can't we see the results of the 2nd test?
Something has obviously given Mindcraft the confidence for this showdown.
If browser elections are causing you grief simply turn off the feature. NT is lightyears ahead of Win95 and other versions of DOS.
Read the article for more details.
I can't believe this. I have worked with NT for 5 years and I can tell you that I have never heard of a claim like this. NT is vastly superior in networking (and every other aspect except gaming) than w95. NT can keep the server busier because of this. if the server can keep up with the beefier clients, then performance should go up not down.
Don't fall for the test. It will not be a fair match --- too little input from the Linux community and too much from Micrsofts tuning engineer Will.
The media has been talking about how RedHat 6 scales linearly to 4 processors.
Apparently the 2.2.x kernal used has a linear performance increase up to 4 processors no problem, after that it is probably still substantial.
Besides you get much better performance and reliability per buck using mid to low end servers in clusters or farms.
Aaron.(TheJackal)
Evil bill is probably behind this. Testing an earth quake machine or something.
This point has been raised in several posts.
Actually, there may be two reasons for this new test.
1- testing Linux versus NT as a server
2- testing Mindcraft capacities to run honests tests.
If the aim is #1, then there should not be imposed configuration, but an imposed budget.
Besides, there should not be points as 'no patches anterior to April, 20th'. Noone is interested in using an old OS version to run a server.
This leads me to think they don't want to benchmark Linux vs NT, but to prove that they are profesionally qualified to run benchmarks.
But, if this is what they aim at, why new tests should be run on NT. I did not see any critics about their tuning of NT. What they should test is their first Linux tests again new ones. The restrictions about configurations and patches make sense in this situation.
Merging the two tests is not a good idea. The results cannot be interpretated clearly. Two tests should be performed, one on their credibility, the other beeing whether :
- Linux vs NT as server with a budget of $ XXX (harware and tuning costs).
- Linux vs NT on 4 processors configurations (but no specification about the number of network cards).
Reading the comments scored 2 or more, I realise many people say Linux would not do better than NT. I personally have no opinion, but I think some conditions (or specifications) in the test are not honests.
I am the same Anonymous Coward who posted the first message... It also surprises me that NT server is optimized to better serve Win 9x than NT clients, but I have never tested this myself, so I have to trust other people's tests: Jeremy Allison and the Samba team have performed these tests, as Jeremy himself mentions in one of his articles (http://www.linuxworld.com/linuxworld/lw-1999-04/l w-04-mindcraft.html). He clearly states that this is the behavior of the NT server, while on the other hand Samba is optimized to be a better server for NT than for Windows 9x clients.
Of course there is the issue of whether you believe -or trust- Jeremy Allison and the rest of the Samba Team. I certainly do, especially after following his postings (on this an other issues) for some months. He has a well-earned reputation. Please notice that he is not saying anything like "Samba is the best" or "Linux is superior", but "Linux/Samba work more efficiently with this client configuration, NT does it better with this other". He in fact admits that NT can be faster than Linux/Samba in one configuration (and slower in the other). Doesn't sound like propaganda to me... I can understand why he thinks that the Mindcraft test configuration should be modified, Mindcraft is simply using the type of client configuration NT is best suited for... which is another way to bias a benchmark. Don't you think MS would know about all this NT/9x serving thing, and that the benchmark configuration is tailored to make NT look better? You will agree that Bruce Weiner's argument of "NT serving NT is not a real world configuration" to dismiss the possibility of using NT clients also sounds a little suspicious. You say that you use NT clients. Well, according to Bruce Weiner you don't exist, or more accurately, your case is so unfrequent that is negligible.
Draw your own conclusions...
Okay I got the feeling M$ tweak the internal kernel for the one they use in Test.
so to be perfectly fair, M$ must use off the shelf shrinkwrap NT, and all public service release.
which is the same requirement they impose for Linux.
Otherwise, there is no way on earth we can know if they changed the NT internal with different built which is not available to public.
the open test only goes open for Linux, than M$ will process the result.
ever got the feeling they will tweak NT kernel behind the door?
the only fair test will be if people can inspect what goes on in NT too, so we all can now if they build a special purpose NT just for this benchmark.
Yes -- how many NT admins actually use things
such as CPU binding? (p.s. If we really want
some fun -- for the last benchmark, we should
get Cox,Riel et al. to sort out the VM patches
(and allow for binding NICs to CPUs for whatever
card is used (and generalise it later)).
It would, however, be a good idea to suggest
a basic install vs basic install shotout.
Microsoft will built a special version of NT tweak, that's why they did not permit open inspection on their side of the kernel.
True -- we'd need to get some kernel patching
done for sake of sprucing up the Linux score.
One trick we'll need to do is the network card
to CPU thing that affinity does on NT.
we have to keep an eye for them so they only instal off the shelf NT software from randomly slected shop.
otherwise they will cheat like hell with special NT built.
they lie. part of the point of this whole benchmark fuss is that there has _never_ been any benchmark showing linux scaling better than any other os. every indication is that nt, solaris, hpux, irix, and even beos scale better than linux.
easy answer for the M$ part, they will tweak the win95 for the client part. so it performs better, but slightly different from the off the shelf OS.
how can we know they are not doing this dirty trick?
easy answer, M$ is hiding all the test number and configuration from public for the second test.
after they got the 3rd test, they will cook up the second test with special brew of NT built and win95 client built, so it will zoom fly past Linux.
but the problem is...this built are not available for public, and all tweaks cannot be inspected.
liar, liar pants on fire.
Stability was not tested by Mindcraft. If frequency of crashes are a problem, NT is obviously not a choice. However, it wil be a good idea if you make a Linux server to build it on well-supported hardware instead of finding some specific RAID controller for which you know the Linuxsupport to be poor.
So not only is NT itself finely tuned by Microsoft experts, but the hardware is finely tuned by Microsoft experts to provide the worst possible results on Linux?
If they get to tune the hardware to their liking for their benchmark, then the Linux test system should be given the opportunity for hardware selection.
You can't claim a fair open/public benchmark if you've done multiple internal benchmarks and select the worst possible combination for your competitor to be put in the spotlight.
I'm willing to bet that the particular hardware configuration was chosen not because it gave the best NT numbers, but the worst Linux numbers, and still looks like a reasonable configuration.
And it would be interesting to see the different if you take away or add a fast ether card, or go to standard f/w SCSI disks instead of RAID arrays.
I'd say we know WHY the results were what they were, and if we point it out to microsoft/mindcraft, they're not going to give an inch of ground on this one.
All M$ has to do is modify the TCP/IP stack in the client OS to recognize the server OS and provide an appropriate delay.
Nothing does SMP better tan BeOS. And I've used them all!
I have supported NT clients but far more cases have been with Win95 clients to an NT server. It's been my experience that NT clients are more competent in a networked environment and are more capable of saturating a network segment than win95 due to the higher performance (NT clients are really NT servers after all).
We need some raw data to look at on the box when it is at the heavy end of the benchmark... iostat, vmstat, netstat. A hands-on approach is really needed to find the exact problem.
And I'll vote with you... I don't think the processor is what is being taxed here.
This is NOT a good thing. If they change the client OS so that it delays transactions from a Linux server, they can get any result that they want.
My read on this:
Windows NT server does as good a job as possible on 95.
NT clients still will perform better than 95 but it just means that the gap is narrower than it would be if NT server were optimized for NT clients.
I think Mindcraft is somewhat innocent in all of this and, in fact, has been set up by Microsoft to take the heat!
They were given the test case by Micrsoft. It is very reasonable to speculate that Microsoft already ran the test internally and knew the results before handing it over and paying for it to be done by a third party.
Does anyone REALLY think that Microsoft didn't have any solid idea of the results before handing it over to an independent party? Remember that the test was done in Microsoft's test lab, and the conditions were spelled out in advance by Microsoft how the test was to be run. Anyone want to bet money that Microsoft didn't try it out on the equipment (and probably multiple other configurations which Linux showed much more favorable results) before handing access over to Mindcraft?
So, Microsoft played the role of Jokey Smurf and gives Mindcraft a loaded package. To mix the metaphors, Mindcraft open it up and BOOM, they've got egg on their face.
I'm willing to bet that Mindcraft really does want to preserve their credibility in all of this. But they've got some desire to serve Microsoft.
I already see a big mistake (regardless of what you think of Microsoft Internet Information Server 4.0):
Under Web-Server Performance Analysis, you'll see that the Linux machine is using Apache vs. the Windows NT box running Internet Information Server (IIS). Why not run the Win32 port of Apache, instead of an entirely different piece of software? How stupid!!
Secondly, if they want to conduct this benchmark with all the most recent software, why are they still using Windows 95 as their client platform? Yet again I say, "How stupid"!!
Does anyone disagree with me?
Brian
This should be televised on pay-per-view.
Reading through the new test guidelines, it looks as if they are going to test the machines in single processor mode, as well as quad. Are they going to allow different setups for the two tests? Will a SMP compiled kernel run slower than a non-SMP compiled kernel on a single processor machine?
Also, why not pull 3 gigs of the RAM instead of just disabling it in configuration, just in case there's some little tweak that could take advantage of it.
Lastly, I liked the fact that they claim that Novel didn't have anything against their test... (see their ABC rebutal, and then look at Novel's home page)
To avoid the "no publishing benchmarks" clause in the license, they must already have MS's explicit, full support for Mindcraft v3.
And, seeing as Mindcrap is STILL publishing their v1 results on their main page, they're obviously still in bed with Bill.
lets get penguin to let Linus use one of their 8 way Xeon machines, with 128MB L3 cache for the test. Oh ya, NT cannot do 8 way multiprocessing, sorry.
dusty lloyd lloydu00@usfca.edu
If I were them, I'd be Mift to!
Some good points have been made about high-end intel servers like the one tested in this benchmark - so even if we lose we have the opportunity to build in scalability at module or even kernel level.
Say kernel modules for handling high-throughput, the kind of thing people are saying NT can do with this tailored hardware.
Fortunately the vendors are now prepered to work with the Open Source community to make their hardware work at its best so Linux should be able to leverage even high end intel and adaptec peripherals as the NT box is in this.
The evidence also shows that the test is using hardware that is deliberately tied to NT. Soon enough that will be a thing of the past. The benchmark is meaningless but at least we have learnt some new tricks that can be implemented with any luck.
Aaron. (TheJackal)
Given the above, here's a very simple situation which would match the data. The data showed Linux very favorable for a low number of connections, but as the number of connections increase, it hit a ceiling and stayed there.
Could it possibly be that the clients are hitting the Linux box with requests on all four of its NIC cards, but that outgoing data is only being channeled out its primary NIC, which saturates?
It seems to me that this is a very simple explanation for the poor (1/4 of NT) results... because it is really only taking advantage of one out of the four NIC cards.
Its a simple network configuration issue that a non-UNIX/network oriented person would make, and it is something that would not easily be caught when soliciting outside support if you didn't supply statistics from netstat or such to show you've balanced your load correctly.
Actually, I don't remember the 4-NIC issue being mentioned with any prominence until now. But Mindcraft insists that a 4-NIC setup be part of any future test.
-jcm
The problem with this benchmark is that it is more a measure of the efficiency of the network card drivers than it is of the speed of the TCP/IP stack. I'm also somewhat concerned about the test results exceeding the "theoretical optimal" values.
In general, however, I concur -- the Linux TCP/IP stack is more efficient, faster, and much easier to develop software with than Microsoft's TCP/IP.
Ooooh there's a good argument. Here's my rebuttal:
PowerPC boxes from IBM are pretty 'high end', and since NT doesn't even install on them, we can claim Linux is infinitely faster than NT on high end hardware.
Q.E.D.
Are we all prepared to accept the possible conclusion that NT4.0 really is better tuned for Quad Xenon processors than Linux is? Do ANY of the Linux kernel developers even have one of these boxes to play with?
It will be interesting to see how the numbers change for NT5.0 and Linux 2.3...
This seems the most important point, no one would have this kind of machine for this purpose. One would buy many machines to do file serving and many machines to do WEB serving. One only needs large machines like the test machines to do large scale computation I have not heard of anyone ever choosing a 4 way processor machine using NT over an 8way processor board from penguin running Linux for the job. Hell most people would build a beowulf cluster of low speed pentiums.
Dusty lloyd lloydu00@usfca.edu
Slashdot, biased??? hahaha Slashdot is the epitome of bullshit bias. Slashdot is for Linux people, period. There will never be a balanced, intelligent discussion here. Hell, why am I here, then...? I'm gone.
Standard Dell server - doesn't Dell support Linux or is this FUD?
If you're talking about PC servers, most don't have that many CPUs, etc. If you're talking about real servers, then the situation is a little bit different.
Ok, we should now accept this as a lose-lose situation. (I'm not going to say why, 199+ people before me just did that) The only way we are going to save face now, is to pull out all together. Claiming that we are not going to deal with biased (blatantly setup) benchmarks. For when a company decides to construct a service, or a presence on the web, they don't tell their department heads that "ok, we magically have these certain computers here, set them up"... they tell them what they need, and ask if it could be done in X amount of dollars, and the department goes out and makes it so. What a more meaningful benchmark would be, is that Linux outperforms NT for any given amount of money. We all know that every OS has it's strong points and it's weak. One of Linux's major ones being SMP.
(funny, how you can have a fully functioning linux system before you even have the money to buy NT, of course this is just a silly aside, a real network solution is going to cost alot on either side)
Mindcraft be damned...
Has it ever occurred to you that maybe NT _can_ outperform Linux on the high end, no matter who is doing the testing? That Microsoft has a _right_ to claim that NT is a better choice for SMP servers because it's supported by facts? Don't lose your faith in the Linux cult, sheep.
This would be another benchmark altogether. Right now, the rules say what and how we are to test. We can either accept, and show Linux not to be as bad as depicted, or refuse to participate.
Also, the rules seem pretty fair.
This test is a mute point. On average Linux is faster then NT, but the biggest advantage of all is linux doesn't crash like NT, or have memory leaks, and whole bunch of other problems.
- -- - --
Chances are NT will beat Linux, but one has to wonder how many blue screens of death will pop up. I think this test will end up biting Microsoft in the ass. It will make the Linux community want to perform better on these kind of tests, and support raid better, etc, and still be STABLE.
Anyway most of you probly know of this page, but if you don't go to: www.unix-vs-nt.org
and you'll feel real good about linux
my 2cents
-----------------------------------------------
bang the head that doesn't bang --Rich Badger
-----------------------------------------------
I see. Now you want to dictate the test conditions so that your favorite makeshift OS can look reasonably good. Well that's not the way it would work in a corporate environment. You'd have to deal with Win95 clients, you'd need the flexibility provided by Apache etc.
Linux experts are already clasping at straws. If the tests are not required to have any practical significance at all then let's drop the high-end stuff altogether. I'm sure Linux boots faster than NT and may even beat it at formatting a floppy. Happy now?
Good points. The same should go for Linux. However, if you want to test uptime, try running TPC. Can Linux even get throug a TPC run without crashing?
Does anyone know if MindCraft is going to continue to take Web Benchmarks only against Apache/Linux or if they will also be testing the performance of thttpd or mathop. I get the feeling that Mindcraft only wants Apache used to match their goal of producing the negative results for Linux.
Hey, this is the scenario of ESR's Halloween 4 - go take a look and see if it isn't identical.
I think ESR should update Halloween 4 to include a reference to Mindcraft's benchmarks.
Funny post. I'm an Oracle/ASP guy for all NT boxes, and we don't have and problems. SlashDot, however, is frequently unreachable. Not too impressive.
What happens if the tuned linux results are the same as what mindcraft saw?
Hi clueless person. The point was that you often cannot dictate the external conditions for a server environment. If Linux is any good it will perform well compared to NT no matter what the external conditions are.
You cannot deny Microsoft the right to pick a favorable setup and at the same time clamour that the test should be rigged in favour of Linux.
Hellooo? How does this support your argument that Linux is superior? Out of the box Linux performs worse - just look at the Mindcraft study. The only difference I can see is that Microsoft has the Knowledge Base whereas with Linux it's close to impossible to get the information required to tune your system (unless you bribe a kernel hacker).
So are you trying to say that Linux was running like a dog two weeks ago but thanks to some ingenious patches can now outperform the very best of them? I wouldn't want to invest in a system where you need to apply patches every hour or so to fix performance bottlenecks.
Pigs will fly.
So your all worried that NT is gonna beat the pants of Linux? Yeah? And in Mindcrafts, Microsofts and the Medias eye it's not unlikely that it will. It's not the first time and it's not the last time the linux will be beaten. What people can't seem to grasp is the fact that Linux is not a product it's a project. A project that an entire community of people has worked on and will continue to work on for years to come. Linux is all about evolving, if your worried that Linux is failing in certain areas then do what no NT user can do. Sort it Out!! Work through some source code, report bugs, hassle companies for hardware information. We are ALL responsible for Linux's developement so quit booming doom and gloom and join the ride. And, for the love of all that is good. ENJOY IT!
in the case of the AMI megaraid, there is a driver written, it's just still somewhat beta.. so wait until it's matured and try it again
However, I figure this Bruce guy knows this, & would have posted a few by now if they existed.
:-(
He posted a few on his site. It's pretty tame....
http://www.mindcraft.com/linux-net-rage.html
BluSkreen
a few hours and still no password....
Maybe this already occurred : Linux performed better in test 2, and test 3 is a PC damage limitation exercise (sure, linux met nt's performance, but it took a linux.god to do it).
-adrian
In an open-source operating system, the code is available for tuning as well as any system parameters. So ANY change can be made : not just binary patches.
-adrian
yes of course it occurs,
BUT.. it is contrary to most of our experiences, and its not in our interest to have to rely on cooked figures or fantasy benchmarks.
Nobody is arguing that NT can't do what they say it can, they are arguing that the test isn't fair to linux so it isn't much use to developers or administrators who many of us are.
From a quick glance its plain to see that the hardware is designed to work best with NT, Its dell's corporate policy.
Aaron (TheJackal)
NT runs on Alpha too you know. And Linux still doesn't scale well past one processor. Oh, sure it will run on a quad box but how much return do you get for each additional processor? 50% maybe. While Solaris is getting about 90% and NT is getting around 80%. While I agree that NT isn't a big iron OS, neither is Linux.
Yes there will be several Linux tuning experts at the test site watching every detail of the testing procedure. This was pointed out in the article.
The environment that the benchmark is set in so heavily weighted in NT's favour that Mindcraft and Microsoft (this of course wouldn't happen without their approval), you can be pretty certain that linux will look bad.
The original NT result has been validated externally but I suspect they'll move the goalposts and hack NT beyond recognition so that even if Linux matches the NT score in test 1 it looks bad in the new test.
The whole thing is set up to save the reputation of both mindcraft and microsoft and has nothing to do with giving Linux or Apache a fair deal.
The benchmark is only open in that tuning can be done but the hardware is deliberately chosen to run NT quickly, as are the clients and the benchmark itself - the football pitch is definately sloped and at quite an angle.
It can safely be assumed that the benchmark was designed after some investigation from microsoft on NT's strengths and Linux' acilles heel.
Get ready for some damage limitation. we need some benchmarks of dynamic content, database (they were published briefly), and on more appropriate hardware and environments.
We also need to see a test that recogises stability, security, etc.
Mindcraft's test is pure hype, I don't even think that it will provide any helpful insights for developers because any problems or limitations won't apply or affect any but the rarest and contrived situations.
These tests should also be run over TCP, rather than just m$ proprietary SMB (Samba).
TCP is also common in industry, in fact a 'standard'.
But then, clients on 56K dialup lines would add realism.
Anyone else try copying large files (~12Meg+) from NT Server to 98 client over ethernet/10Mpbs? I've noticed that smb barfs, where ftp/tcp works.
Does anyone have some back issues of "Sm@rt Reseller"?
There was an article in that magazine not two maybe three months back on NT/Linux server comparison. Unfortunately, I didn't keep the issue, and I can't remember the server specs, but I do remember that they said that linux (Red Hat with Samba) was a better windows server than windows itself. I believe it provided 5x the performance at high loads.
Does anyone have this? Can we post it? Or post a link to it? I've not seen it mentioned, but it was a solid unbiased test that proved what linux could do on a modest machine.
Oops, forgot to mention, i'm speaking about the original test that is.
/Micke
Instead of wining about a company who are clearly hand-in-hand with MS doing benchmark Linux vs NT comparisons, why doesn't somebody organise an independent test? It seems that the Linux experts are willing to turn out for this test, would the NT experts turn out for something that was independent? It would allow us the opportunity to do bencmark comparisons on real world systems. i.e. from a lowly 386 to a 4 cpu SMP box. Not every desktop in the world is a 4 way pentium III xeon with 2Mb L2 caches, with 2Gb RAM and a 50Gb RAID array. My 0.02 ukp.
The obvious problem with the third 'honest' tests is with the choice :)
of hardware. Will an i386-Linux machine outperform a Cray vector
supercomputer? Most likely yes!, if Linus tailors the hardware and
the software to our liking, we will easily win.
So forget the Mindcraft challenge, with tests like these we should go
against the Crays
thttpd is not a threaded web server. In fact, due to its design (as I understand it), the thread implementation makes no difference to thttpd. If you're running a multi-processor system, Apache will probably be better because it will take advantage of the multiple processors; thttpd, because it is only a single process, will not.
ned
Please read the article "openbenchmarks.html" at the Mindcraft which is the basis of this story.
Mindcraft accuses Linux and others who posted critical comments, and the press, of besmirching its good name and honesty. A threat of legal action is implied. Such threats, implied or otherwise, are much worse than the original decption. Mincraft has no credibility or honor.
That is enough. Who are the idiots from the Linux community who are cooperating with Mindcraft in this scam ? Mindcraft has already lost its good
name by not prominently identifying the original test as Microsoft funded, and now tries to intimidate individuals, magazines and IT companies that exercise their right to free speech and engage in peer review.
Clearly Mindcraft has its own axe to grind. Any Linux guru who cooperates with such a discredited
venture does great damage to us all. The world will forget about Mindcraft, except these media hungry Linux "experts" take the bait, and lend credibility to a discredited and dishonourable commercial company which, to add icing to the cake, makes threats of implied legal action when these same individuals exercise their rights to free speech and peer review of Mindcraft's methods.
This situation is grave, only because Linux gurus who should know better have taken this bait. The proper response would be legal action against Mindcraft for false and misleading reporting of its initial tests and failure to adequately disclose its funding. In other words, misrepresentation and slander of Linux and Linux related software and service vendors. It is time to start playing hardball with these dishonourable people who can never gain respect in the IT community no matter how many repeat tests they perform.
It would be better for Mindcraft to just close shop, and best for Linux gurus to stop playing MIcrosoft's game and use some common sense in such matters. It may well be that NT beats Linux in certain tests with certain hardware configurations. So what. Nobody would care except that these people who may be respected in the Linux community are so eager for publicity that they fail to see they are just being used by Mindcraft and MS to their own disadvantage.
Part of the problem with the results being generated is that what they are doing is creating a system for getting results. Not for true functionality. Also there is an assumption that like systems should compete alike, that everyone needs max performace, and that all systems should scale performance wise accross the spectrum.
For a more complete idea of benchmarking what should be done is they should design systems based on what would really happen say in a small, medium, large, and possibly max performance areas. Then setup budgets that should be used for each level, except for the top which could be whatever anyone would want to spend. Of course Microsoft could probably win that one since they could spend the most.
This way you could show more realistic situations, where the equipement would be set up for certain purposes, and they could be designed to show the true strengths of the platforms, say NT with 4 processors, and Linux Clustering for an example. However other platforms should be allowed also as I would love to see OSX compete head to head in a open test with NT and Linux.
An important thing also to note is that NT will crash, often. This is one of the reasons Linux is slower, is because it makes sure that the system will not crash. A friend of mine wrote a program that allows any user to access any part of memory in NT. This causes a system to crash. Linux will always be more stable.
Question: Who has servers where 2% of this much traffic really matters at this speed. People doing VR or building Beowulf clusters?
Another Question: Is speed more important than reliability?
Dusty Lloyd lloydu00@usfca.edu
Realistically, two things can happen from all this. While I'm not completely convinced that NT will win, I'm very skeptical about any tests that aren't 3rd party (ie: payment is not coming from one of the manufacturers)
Scenario 1: We get our butts kicked. NT beats linux into a pulp in front of everyone and it's a huge embaressment. Microsoft toots its own horn even louder than usual, and all that positive press we've been getting will essentially be negated. The only good that can come of it is that critical components are finally improved and perhaps, one day, will outperform NT even though it will have cost us dearly.
Scenario 2: We win, or tie (as much as a "tie" can be defined in benchmarks). Either way, it proves to our *slight* advantage since we can shut Microsoft up for awhile about their being better by fighting fire with fire. But the problem is M$ already has what they're looking for... they continue to have the first benchmarks bashing linux into the ground.
Either scenario ends with M$ giving linux the shaft. We have to remember that complacency will kill us, and overconfidence will turn us into a bunch of horn-tooting, advocating Linuxserfs. The only way we can win at this is improving the system (both the Linux system components and our way of doing things). Both will have to evolve if we ever expect to reduce Microsoft to a mere corporation instead of dictatorship. M$ will always have the upper hand when it comes to marketing. No one's ever beat them. Let's use the fact that they SUCK at making OS'es to our advantage buy beating the living *TAR* out of them, even *UNTUNED*.
Other than Mindcraft's complete lack of integrity, the issue at hand seems to be exactly with what purpose are we trying to optimize the systems for, and what is the cost of achieving that? Instead of structuring this a reasonable, abstract test ("Build a system that can devliver 1000 8k web pages a second and report the cost.") they seem to be trying to keep it a biased, restricted test ("Use this hardware and software, and we'll see if we can't find some way our system still beats yours."). If this sort of underhanded "independent testing" continues, I will have to stop buying products from companies that use Mindcraft studies to support their product.
In the Mindcraft report comparing NT with Netware, price/performance appeared at the very top of the report in the executive summery. Likewise, in the comparison between NT and Solaris, Mindcraft begins with an executive summery going over the price/performance. In fact, a spot on Mindcraft web site will be dedicated to price/performance since it is such a critical measurement.
So, when it comes to comparing NT and Linux, where is the price/performance figures? The Linux report also has an executive summery like the last two comparisons that Mindcraft published. But the key issue of price/performance never apppears. It is missing from the executive summery and missing from the report complettely. The reason for this is very clear. While the NT and Netware comparison put NT Server 4.0 at a software cost of $4,949 while RedHat Linux 5.2 with a 3-incident support pack both direct from RedHat comes to $249. (By the way, the 3-incident support pack also provides support for Apache & Samba issues.) The end result is that NT Server 4.0 would have to outperform a Linux server by nearly *twenty* times to achieve a perferable price/performance measurement for NT Server. Even misconfigured to produce bad results, Linux continued to FAR under-cut NT Server 4.0 in the price/performance.
The Open Benchmark Invitation will also product even more damning price/performance comparisons. The reason is that single CPU systems will be included in the tests. The Mindcraft report was based on the Dell PowerEdge 6300. This system comes with a base cost of about $15,000. One you add the Mindcraft specs of 4 processors, 1MB cache, 4GB memory, a 10,000 RPM 9GB hard drive, eight 7,200 RPM 4GB hard drives, the Dell PowerEdge 6300 system has over a $35,000 price tag to it. But it isn't the PowerEdge 6300 that Dell pushes with RedHat pre-installed on. It is the PowerEdge 1300 with a base price of $3,000. Once you kick the specs on the PowerEdge 1300 to having 768MB and four 9GB drives (matching the Mindcraft 41GB storage on their four processor system), the price is still down at $8,000. Hence, four PowerEdge 1300's can be had at $32,000, over $3,000 below the MindCraft PowerEdge 6300 configuration. But, if four uni-processor systems can ourperform the price/performance of a quad-processor NT server 4.0, do you think that statistic will make it on Mindcraft's price/performance page?
It is too bad that Linux can not be configured to perform badly enough to provide Mindcraft with price/performance they can publish. It is too bad that Mindcraft can not be "unbias" when choosing what price/performance figures too publish.
According to Mindcraft's website, they are experts in evaluating
computer systems and they have been in business for 14 years. And yet,
they do not own a lab in which they evaluate computers and conduct tests. Why?
Their server configuration was a dual PII-333, 512MB. They report system Price/performance.
Mindcraft's NT vs. IntraNetware benchmarks:
Server Config: Compaq ProLiant 5500 single 500Mhz Xeon. They report software Price/performance. (see Novell's rebuttal)
And then, sometime between last fall and now, servers needed 3 more Xeons and another 512 Megs of ram, and companies stopped caring about price/performance.
Only now, on the eve of a possibly disastrous test result, are you willing to admit that Linux may be inferior to NT. Why didn't we see such rational comments after the first test instead of the mindless bashing of Mindcraft and Microsoft? As a neutral observer I must say that the Linux community seems to be at least on par with Microsoft when it comes to hyping your system and spreading FUD.
How about an even more "realistic" benchmark that would reflect the outside world: Set a budget and get let the hardware/software fall where it may. Benchmarks are useful only to those in the market to buy and need guidelines for purchases.
Am I wrong on this? What percentage of people out there have similar setups _currently or ordered) in shop that match the Mindcraft study?
> NT supports zero copy transmits out of the disk cache or the web server cache.
Windows2000 supposedly does. Not NT4.
> This means that the data is directly DMAed by the NIC card out of the disk cache or user-space memory buffer out to the wire without having to be copied to kernel space.
But guess what, you still have to touch every byte of data to do the TCP checksums. Zero-copy won't help you much there. (read today's kernel-list archives for more info)
> 2. NT (with Intel cards) supports Fast EtherChannel if connected to a Cisco switch.
So does Linux. (I believe you can do this with the Beowulf patches if you really want to) And Mindcraft won't be using this configuration, anyway.
> 3. Intel adapters on NT can offload the TCP checksum calculation to hardware.
Adapters which are used in about 0.1% of all servers out there, due to the fact that they cost as much as an entry-level server in and of themselves.
> I don't know if SP4 added that support to NT4 or not. It is definitely in NT5
It's not in NT4, which is what Mindcraft is supposedly testing.
> 4. If NT uses NetBEUI it will beat Samba. NetBEUI is much faster than NetBIOS over IP.
It also sucks. Can you say non-routable over the Internet and a general pain in the ass to set up over any sort of large area network.
Again it shouldn't be any part of this test.
Why do people find it necessary to send Bruce rude
and vulgar email on account of these tests? That
can only encourage him NOT to help the linux people.
It is fine to be upset about the test results, to
disagree with them, but sending offensive email
doesn't help anyone, and certainly makes Linux
look less professional, which is not what we want.
If the tests come out in favor of Linux, do you
think Microsoft employees will start sending Bruce
nasty emails?
To compare vanilla Linux vs vanilla NT? Most of the servers out there are not tuned by experts.
Mmmm... This is getting interesting. Jeremy Allison, of the Samba Team, has already answered to the Open Benchmark invitation (http://linuxtoday.com/stories/5630.html) suggesting a modification in the benchmark configuration, namely that the clients used in the tests should also include NT clients and not only Windows 9x. Apparently the performance of an NT server goes down quite a bit when serving NT clients, and both the Samba Team and Mindcraft (or Microsoft) know this well. In one of his previous rebuttals, also available at Linux Today (http://linuxtoday.com/stories/5628.html), Bruce Weiner of Mindcraft tried to justify the use of their setup not because it gave the NT server any advantage, but because there were far more Windows 9x than NT clients out there, and testing the system whit NT clients wouldn't be a "real world configuration". Interesting, if you think that NT is supposed to be the way to go in the near future according to Microsoft and their flagship product... I work at a university. Should we dump the pile of NT workstation CDs we bought some months ago for our departamental network and go back to 98?
I believe there is a clear chance of NT beating Linux -even properly tuned- in these benchmarks, but all the details surrounding the tests, including the hardware and the configuration chosen, don't smell very well... Too biased.
http://www.microsoft.com/mi sc/backstage/column_T2_1.htm
... which hide server failure from end users. The result: Up to 100 percent availability"
Quote: "How microsoft.com finally achieved 100% web site availability"
"Wanke: We needed to figure out why our servers were crashing, which requires them to be taken offline for debugging. For every machine, you have to have an IP address. This problem, for years, has been unsolvable."
"Weeks: Just running CHKDSK on a 36GB server, which is standard after a crash, can take up to three hours. We couldn't have a server fail, or do maintenance on a server, or be able to leave a server down and try to figure out why it went down without it affecting customers."
The solution? Surely it must be to get one of these huge mindcraft servers that are so fast? No?
"The microsoft.com Web site consists of several clusters of servers, each with a number of segments consisting of 4-6 servers containing mirrored content.
Up to 100 percent availability... Impressive.
Paul
Come on.. its just a worthless benchmark. No benchmark can be 100% accurate. Benchmarks are nothing but a device to sell software. Benchmark programs themselves can be skewed by running faster/slower on different OSes.
/.'ers make it out to be.
People who are fighting to get a better benchmark test are nothing but Linux bigots. I remember a time when Linux was used for fun and hacking (not cracking). Who cares if NT shows better "results"? I sure don't. (Don't give me that "my company will look at these benchmarks".. you shouldn't work for a FUD driven company in the first place)
Lets face it.. you (/.'ers) are all anti-Microsoft Linux bigots who want free (as in beer) software and do not give a damn about free (as in speech) software. Anyone with a brain knows Linux is crap in many ways and is not the holy grail
Oh well.. go ahead and moderate this down. The truth hurts.. and it seems many moderaters can't handle the real truth.
If IIS with these configs can't beat the "swiss army knife" Apache then IIS is really bad ! I mean look at what those configs actually meant
p p05_5xtf.htm
:)
These were some that i bothered to investigate.
ObjectCacheTTL = 0xffffffff.
This it to be used if all contents fit into memory and is static (hmm, and guess what the benchmark measures...yes, static performance.) It supposedly turns of the cache "garbage colllector". See http://msdn.microsoft.com/library/sdkdoc/bdg/bdga
OpenFileInCache = 0x5000 = 20480 (max number of cached open file handles) Well caching up 20 k open filehandles should speed up serving static pages
ListenBackLog = 200 (couldn't find it on MSDN), presumably means that listen() can have 200 pending connections.
And what is my point ? well IIS was totally geared up to handle a website that fit in it's primary memory (anyone how big is the "site" in webbench?)
but,pardon me if'you've got a quad xeon running some kind of "enterprise" website wouldn't the amount of data be _much_ more than 1(or 4) GB ? And would hardly be static either, but generated from scripts,databases and whatnot ?
In humble respect of all computer gurus out there, i hope i haven't got this totally wrong
/Micke
The fact that Linux has done this well is interesting, but really folks, take a look at the test -- it is specifically designed to be a test that Linux cannot win at this current time. 4-way processing. RAID support. And with third party utilities which were created for their usefulness, not their ability to shine on a benchmark.
I don't think we're going to find a 4x boost in the results. One has a good argument saying that Linux is *not tunable* for the reason that it has a decent design with reasonable defaults and there's not much that needs to be tuned! [If someone does a partial quote of that last line and puts it in a magazne, they are vermin.]
BTW, a few semi-related notes. If I was going for performance on a web server with Linux, I would chose an architecture far different from a monolithic 4-processor, 4-network box with RAID storage. I would go with multiple simple systems, which would allow me a path to grow that is not tied to upgrading to expensive bleeding edge technology.
If I wanted to do a performance based web server, I'd put all the files in a memory based filesystem. (Or have sufficient memory to make sure that I'm going to get a high cache hit rate.)
If I had to use drives, I'd use solid state drives (the kind with built in battery and automatic disk backup) for ultimate performance.
I would spread the documents (images, html, etc) across several machines. Additionally, I might have "www.myhost.com" assigned to several IP addresses so that each new connection is semi-randomly distributed to a new machine.
There would be no need for 4 x 100Base-T connections for each host. (BTW... are these going to the same network or different networks? Meaning... is NT doing *IP trunking* with these (all four 100base connectors "striping" output on the same network), which Linux won't? That could explain a 4x difference and wouldn't it be funny if one simple feature is all that this test demonstrated?)
The design given for the "web server" and "file server" aren't really good hardware choices for the job. BTW... two different services on the same box? Aside from the fact that it is bad design on the NT side, the two functions really require different configurations.
And unlike in the previous hatchet job, we are going to be right there watching as they tromple over our OS and publicly defame it.
We are going to be standing there with our mouths open saying, "What happened?"
Folks, this is a battle we can only lose. Bruce has his LIVELIHOOD at stake here. NT is not going to lose no matter how open it may seem. Whatever it was that happened in that second test Mindcraft did gave them the confidence to do this test so they could appear to the public as "fairminded" and "open".
It ain't so. What can we do to prevent this? I hope noone important falls into this trap. It could be a black day for us all as the press trumpets "NT prevails in open tests where linux gurus try their best!"
I'd like to point out a few issues with this: 1) Motivation; in point #1, Mindcraft says the purpose of the test is to verify the results of the second tests. No Linux experts ever even had the opportunity to view the second test's results. Why are we rerunning test #2 which no one has had the opportunity to see or critique? Why not retest #1, the one that everyone says is false and inaccurate? 2) Machine used; this machine will not be the Dell PowerEdge that was used for the first test. It may be a machine that was specifically chosen for it's weak Linux drivers or other reasons. 3) Mindcraft configuration; why must the Linux experts use the configuration that Mindcraft used? 4) Microsoft's ability to use the results in press releases; I admit, this part is the reason why the tests are run in the first place; but why is this starting to look like a press campaign? Microsoft has incredibly tricky attorneys; why should we play their game? 5) Will the Linux experts have the opportunity to edit the 'joint' press release that includes quotations, possibly out of context, from the Linux experts? --Curious in Atlanta
It's a little obvious when you think about it. Mindcraft says that the hardware configuration is "set in stone". And, in fact, when they posted messages on the newsgroup, people told them that they needed to change their hardware configuration... but they won't. The key to this entire test IS the hardware.
Where did this specific hardware configuration come from? Microsoft. And Microsoft didn't just pull this configuration out of thin air. They've been doing all sorts of internal benchmarks with Linux systems to see what kind of numbers they could get. In their tests, I'm sure they've come across many favorable Linux comparisons.
But, as expected, they've found a few sub-optimal configurations for Linux which NT does well with. Microsoft has run many tests, and found the hardware that works the worst with Linux but good with NT. This is what explains the email from a Microsoft email address regarding the Xeon configuration... the email that Mindcraft said they did not send. It was done by Microsoft's internal testing.
Once they have the bad configuration, they need to send it off to another third party for "independent verification". Which, I believe, it isn't Mindcraft that was responsible for the low numbers... they were handed a benchmark that Microsoft ran ahead of time and already knew the results for. They just needed an outside party to go discover it for themselves.
This creates a simple smoke-and-mirror effect. Microsoft isn't blamed for this... Mindcraft is. And it disguises the real issue... it isn't software/OS tuning. It is hardware de-tuning done in advance by microsoft.
Consider that each piece of the system is pretty much a non-optimal configuration for linux. CPU, RAID disk, and probably the network. Probably one of these pieces (say, the 4x100base-t instead of a single gigabit ether) is really sub-optimal and replacing it would probably yield incredible results, but you've still got the additional handicaps.
But the Linux community has walked right into this one, thinking they can tune it out. Probably short of re-writing some kernel and driver codes, this piece of hardware isn't going to fly on anything *but* NT. (*BSD will also compare unfavorably.)
It probably is too late now to point this out -- they'll have the claim of "sour grapes" to use.
If you think anyone had any real input into the second set of Mindcraft benchmarks, go have a look at AC's commentary, entitled "Bruce Weiner: Lies, Damned Lies, Statistics":
http://linuxtoday.com/stories/5631.html
slashdot broke my sig
It's still there. Click on the Services link (http://www.mindcraft.com/company/services.html), second paragraph under Performance Testing & Capacity Planning.
Who will pay for it? Are the various commercial Linux vendors willing to pool resources and commission Mindcraft or some other third party to run the tests?
All of these tests should be run, not to prove anything about Linux vs. NT, but to make Linux better. That should be our goal. NT should be an afterthought.
Right on.
The Linux team could set up clusters of workstations, etc., but i don't think that gets us any useful information. The more I think about it, the more I like the idea that the hardware may favor NT. If it does, it indicates a problem with Linux and we need to fix it.
Why doesn't samba work well with Win95? These are the questions we need to answer.
On the other hand, there is some merit to letting a third party pick the hardware. In this case, we will have the opportunity of finding problems with drivers rather than simply using things we already know work well.
If poorly supported hardware is used, however, it must be made absolutely crystal clear in the press release. Nothing less is acceptable.
If the Linux community is serious about doing this, it's going to take a lot of planning. Just because you may not be taking the trip doesn't mean you can't contribute to the effort. After all, the community has always been trumpeted as a strength of Linux. We ought to set up a web site, mailing list or something to coordinate all of this.
To those who make the trip: take good notes!
Looks like Linux did not come out on top in their second test, and Mindcraft is willing to bring in Linux experts to verify the results.
My biggest fear with this is not that Mindcraft will try some underhanded trickery, it's that the Linux community will not accept the results unless linux comes out on top. That is just as bad as if Mindcraft tailored its benchmarks to favor NT.
This open benchmark is a wonderful opportunity for the Linux community to benchmark itself. AFAIK, no controlled experiments on high-end servers have been done to see how Linux stacks up against the heavyweights. If Linux comes out on top, great. But if not, we as a community have to accept that and learn from the process.
This is best chance yet to discover the bottlenecks in the kernel and several critical pieces of software. IMHO, the Linux experts should not go in with the goal of beating NT. They should go in with the goal of squeezing every last bit of performance out of the machine and using the resulting data to fix the problems.
Mindcraft and Microsoft (!) are donating resources to the Linux community in an effort to help us improve the OS. Let's grab the opportunity!
Yep, hardware compatibility is an issue. My first thought upon seeing the original Mindcraft benchmark was "Man, that Dell SUCKS, I wonder how much Mindcraft would charge to compare it against one of our Linux Hardware Solutions machines?". My second thought, of course, was that if we wanted to do such a comparison it would be better to choose a more professional organization than Mindcraft (I was NOT impressed by the unprofessional conduct of Mindcraft -- both the test itself and the reporting of the results were conducted in an extremely unprofessional manner).
We weren't told what brand of network card is being used (if it is a PNIC-based one the PNIC has transmitter lockup problems under Linux), but the AMI MegaRaid is still pretty aweful under Linux. The latest version at least will do concurrent requests (the last version serialized requests -- i.e., you could not issue a new SCSI request until the previous one had returned its results), but it is still very immature compared to the ICP-Vortex or Mylex drivers used by most Linux hardware vendors.
In reality, people wanting this class of machine are going to buy it from someone with a track record, like VA Research, they're not going to buy an off-the-shelf box and install Linux on it themselves.
Send mail here if you want to reach me.
Linus has a quad Xeon as his home computer!
But: It's an Intel box (sold by VA Research, the former Linux Hardware Solutions, Micron, and others), rather than a Dill.
Send mail here if you want to reach me.
I agree that the real reason for this is because Mindcraft's reputation is in tatters.
However, as long as they are using a RAID card which is known to perform poorly with Linux, it can hardly be considered to be a "fair" test. The Mylex and ICP-Vortex cards used by VA Research and others are known to be fast and stable with both Linux and NT, and would undoubtedly be what is installed in any Linux server of this size.
The network configuration may be a problem too, but we don't know enough about the network cards used to be able to tell.
An interesting thing is to see whether mod_mmap_static (a new module for Apache which pre-maps files into memory) would greately increase Apache's ability to serve files swiftly. Otherwise we already know that IIS trounces Apache soundly in serving files.
Remember, Linux does not have to beat NT in order to win. All Linux has to do is get numbers high enough so that Linux can be considered credible. If Linux is within 10% of NT in Samba performance, and within 20% of NT in Apache performance, Microsoft can tout those results all they want -- all it will be is free advertising for Linux. Remember, performance is only one reason for choosing Linux. The most important reason, FREEDOM, freedom from onerous licensing restrictions, freedom from being at the mercy of one monopolistic company, that reason is one that Microsoft will never "get" because Microsoft is about everything BUT freedom.
Send mail here if you want to reach me.
You can't say that NT beats Linux or Linux beats NT based on such a small sampling of situations! Mindcraft appears to be saying as much though...
I expect that if you were to take the CD's and the supplied boot disks and tested NT vs Linux on uniprocessor machines, then Linux would do quite well! Then install and compile the latest 2.2 kernel and all the appropriate NT patches and test uni and multiprocessor configurations. And do the tests on more than one hardware configuration.
Evidently, we are going to need to have a project dedicated to testing and put out a Hardware Compatability List! Hardware that is either supported by the manufacturer or the manufacturer provides detailed specifications to third-party driver writers would likely appear prominently on the list.
Codifex Maximus ~ In search of... a shorter sig.
I needed to read just what you said. I think we all did.
Stay on target.... Just a little longer... Staaaay on target!
Codifex Maximus ~ In search of... a shorter sig.
but Mindcraft is playing poker. They have the unreleased results of Test #2 and those tests may or may not be favorable to NT. These unreleased results are Mindcrafts hole cards.
:)
Now, let's say we refuse the test and Mindcraft releases the results of the second test showing favorable results for NT. They can say that they offered to retest but we refused. They win. Mindcraft's stock is going up.
If we refuse to do the tests and Mindcraft doesn't release the results of Test #2 because they are unfavorable to NT, Mindcraft will simply change their name and go on spinning for their customers. They lose a round.
Let's say we do the test and it is so narrow in scope that the hardware is prime terra firma for NT but less than savory for Linux. Also, the tests must be done using software current as of April 1999 meaning that there may be broken drivers. Our experts tune Linux to it's utmost yet Linux still doesn't perform as well as NT. They win. Mindcraft is back in business.
Let's say we do the tests and Linux wins. Someone already said that Microsoft could spin that too! "Linux is faster than NT if you can get Linus Torvalds to install it for you!" Linux wins but they still look good. Mindcraft's rep is restored.
Based on these scenarios... I say we should participate in the tests. They should be held at a neutral site or have them at a major trade show with the world looking on and installed from a clean machine. That way, if Linux loses, it'll be in a fair fight! I can't imagine Linux losing by much though if at all. I am confident.
Codifex Maximus ~ In search of... a shorter sig.
I would suggest then that they are in the wrong line of work. Benchmarking requires honestly, diligence, and impartiality. If they willingly sell their "fixed" benchmarks to the highest bidder then they really shouldn't expect anyone to believe their work.
Seriously, this is a scam. Step back for a minute and you will see it clear as day.
Sig (appended to the end of comments you post, 120 chars)
I was thinking this myself. While its not true that everyone runs stock settings on their production servers, not everyone has Linus Torvalds and Alan Cox at their desposal either.
Computer Reseller did a test with stock software, albeit much less hardware horsepower than the Mindcraft machines, and Linux came away the winner by miles.
Sig (appended to the end of comments you post, 120 chars)
...that "benchmarking" product A and product B, while getting paid from product A's manufacturer, using product A's consultants, and using product A's own testing fascilities...IS INHERENTLY BIASED!!! Regardless if product A is actually better than product B or not. My God, its so simple yet they can't figure it out!
Mindcraft has no credibility left. Worse, now they have an axe to grind because of how badly they were beat up in the mainstream press (ABCnews.com, Salon, Slashdot, etc). Any "benchmarking" they do from here on in should be ignored completely. If we can learn something from this, such as the need for better documentation, then at least thats a positive for us.
Sig (appended to the end of comments you post, 120 chars)
There have been some excelent points made here, and I agree that the whols thing is quite biased, but we can only learn from it, no matter what the end result is.
Here is an evalutaion of Linux 2.0, Linux 2.2 and NT4 TCP stacks. http://www.marko.net/neteval.
Evaluation was done with Netperf network performance tool. A small perl script was used to generate a series of test message sizes, beginning at 10 bytes, and with each additional packet being 10% larger than the previous until a maximum of 65000 bytes was reached. Each test was executed for 20 seconds, with a minimum of one second pause between the tests, in case there was still data in any outgoing or incoming data caches. The resulting files were plotted with gnuplot.
The tests were done by Mark Spencer, and questions regarding it should be directed to him, markster@marko.net
Anyone read Mindcraft's rebuttals of the reports about them? I found them very interesting...for what they did *not* contain.
In particular, one accusation unrefuted is that Mindcraft's claims of having gone to the Linux community for help are simple lies. Any appropriate forum in which they could have looked for help is archived: a URL or two from Mindcraft showing that they looked for help would set the issue to rest. No such URL is given.
So if Mindcraft are honest they're doing a damn good job of making it look otherwise.
Just to emphasise the other key points here: (1) the hardware has already been chosen by Microsoft as that which best favours NT, and would not be chosen by someone depolying Linux. (2) Apache isn't designed to be fast at serving static HTML because static HTML servers are bandwidth-bound except during stupid unrealistic benchmarks. For such benchmarks Zeus is a better choice, also available for Linux. (3) It's very suspicious that it's the unpublished second test that we're being asked to reproduce.
We already know how these sorts of results look if a less biased organisation does them, because ZDNet did them. Linux blew NT away on every count, by every measure.
--
Xenu loves you!
He stated this after the first Mindcraft test results were published. I think he also helped set up the Linux box in the second test.
It comes down to facts in the end.
Either Mindcraft is using hardware to favor NT or it isn't. Either they are allowing NT to be tuned by experts from Microsoft or they aren't. They are obviously using Microsoft's own testing facilities. I don't know that this is necessarily a bias. What equipment are they using? Does the site favor NT in any way? Aside from personnel on-hand, I don't know of any other benefit that could be gained. As long as Linux has its own experts and witnesses present to inspect the hardware, install and tune the software, and witness the testing of both machines, I don't see any problem with the test. The only sticking point for me is their ridiculous choice of hardware.
There are much more realistic and fair ways of conducting such a test. Perhaps each side should be allowed to build its own machine, given the same amount of money to do it with and the same source(s) to buy from. Any money not spent could be considered in the results. (It would have to be given appropriate weight in the results though, which in most cases wouldn't be worth a whole lot.) Then each side could tune its machine to meet basic real world requirements for a web server or whatever else they decide to test it as. This would be a real contest I think.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
If the results come out the same, it will be for different reasons in a fair test so that we know the problems with Linux are valid.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
He said we would get beaten. He didn't say that the results of the first test were correct. There were many mistakes made in the first test. Whether NT would still win or not doesn't matter. We want a fair test. Microsoft knew about the test. They let Microsoft experts tune the NT machine. Why didn't they tell anyone in the Linux community about the test? Why weren't Linux experts allowed to tune the Linux machine in either of the first 2 tests? Don't try to pretend it was a fair test. It wasn't. Maybe we'll lose, but at least it will be a fair test so that we can see the real reasons we lost so that something can be done about them.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
I'm almost ashamed to say it, but I have to agree with you. I enjoyed Linux more back when it came on 5.25" floppies (3 of them) and the installation instructions said "use fdisk and mke2fs to format your hard disk, then cp this floppy onto it".
Sure, it wasn't easy to use. Sure, it wasn't even that impressive: it was slow, buggy, didn't have half the features my copy of Interactive had, and even my SCO box made Linux look like a cheap and nasty toy.
But I enjoyed Linux because it was FREE. Proper free. Free speech. It was liberating. I didn't care that Linux was crap. I was happy to see the rekindled interest in free software, proof that software didn't have to hide behind locked doors.
Now Linux has become just a face among many. So many people don't realise that Linux would be the better OS even if it isn't as fast. Even if more users choose MacOS or BeOS or Windows. Even if it was harder to use, or slower, or crashed more.
RMS recently gave a speech at the ANU where he expressed the same concerns. People using Linux should be exalted by the realisation that they have experienced something that was well on it's way to becoming extinct. Instead too many users think only of destroying Microsoft, of having an impressive number of users, or having elite new features and all the latest buzzwords. Don't get so impressed by the glitz that you lose sight of the goal.
31 comments already and no one has asked "Who are the Linux experts?" Mindcraft got busted hard and now they're trying to make up for it. But don't expect Linux to win. Remember: No one in the Linux community will ever be a customer of Mindcraft, but Microsoft will be, and we know the pockets of Microsoft. Mindcraft has no reason not to botch the test again and post false, misleading, or biased results.
Mindcraft is a business, and as a business they only want the almighty buck. I seriously doubt they really care about publishing correct results.
Before the original slashdot.org story, I'd never heard of Mindcraft. Now I see why...
Linus himself has been invited to help tune the system, and witness the tests himself, along with other Experts of Samba and Apache...
-- I'm the root of all that's evil, but you can call me cookie..
I've got a hunch that a whole month after the tests, if they where to fail, would be devoted to getting Linux to outperform NT in these cases.. It would indeed be a blow to the 'movement', and a WHOLE lot of reasons for people that are on the border to go with NT..
I'm hoping that what I suspect will come about, aka, Linux outperforming NT by a wide margin, this type of test has never been done to my knowledge before, and if it turns out that they CAN get NT to work better then Linux, it could be the end of a bad first few chapters in 'The Book of Linux'
-- I'm the root of all that's evil, but you can call me cookie..
Apart from the client issues of NT/95/98, the other major concern for this test seems to be the choice of hardware. Mindcruft would be in a very difficult position if they turned down the offer by VA Research (or another vendor) to provide a machine of roughly equivilant value and similar spec that was known to have existing optimized drivers available.
This would be equivilant to the first machine being hand picked for NT.
The other serious concern is the stability - Rumors I have heard from a production enviroment indicate that NT will Never stay up longer than 49 days - due to bugs that still remain unfixed. Unfortunatly 49 days is a long time to wait for a reliability test, but with a sustained load, or repeated tests, I would be interested in seeing if NT could survive.
Hong Kong Linux Center home of squidblock, and other cool stuff
On our box we switched in thttpd to see how much of a difference it would make, and it maxed out at 1200 hps (compared to apache's 1800+). Not sure why. I expected it to be faster too. Perhaps because of Linux's weak threads implementation.
,hacker Perl another Just)'
Matt.
perl -e 'print scalar reverse q(\)-:
Matt. Want XML + Apache + Stylesheets? Get AxKit.
(eg: If Mindcraft insists the expert uses GCC 2.6.3, you're not going to get the same results as if you used PGCC, regardless of how expert the expert is.)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
What most people want is something that works (doesn't crash), give decent performance for the lowest cost... In real life if you want a server to run with Linux you probably won't use exactly the same hardware, than with NT (and maybe a completly different setup)... Most people are happy if something run with a minimum of maintenance, no crashes (reliability is much more important than speed) and is not too slow...
Linux does have many weakness, but so does NT... What a particular benchmark show is how a particular setup with particular result perform, it's not for all cases.
Benchmark could prove anything, but like someone said 90% of statistics are useless...
This seems to point at the fact that there will be "representatives" from all three sides present.
jaraxle
It seems that Micros~1 and Mindcraft knew that this was going to happen. However, by denying Linux the use of these drivers clearly demonstrates how tenuous the NT advantage must be.
This clearly shows how long Mindcraft has been looking for a platform that gives NT enough of and advantage. Perhaps these new drivers eliminate the advantage, but we will never know...
Is this a rhetorical question?
This looks biased already. Most corporate desktops are NT, and this is what Samba serves best. It doesn't do as well with 9X.
If they can't win on speed with the Web server I presume they will want to use ASP. Expect a whole lot of MS "features" that Linux can't support, or only supports poorly.
Linux is not a toy, but it suits Microsoft to make it look that way, and the more we moan about how this isn't fair, or that isn't fair, the more we play into their hands.
What we're missing here is that Microsoft is throwing us a huge opportunity to demonstrate how Open Source really works. Let them come up with the most disadvantageous set of circumstances for the test, the configuration that makes Linux looks the worst that it possibly can. We then say 'thank you very much, you've contributed hugely to the open source effort by testing our code for us!' The great strength of Linux is not that it is the best, but that it is agile enough to be able to get pretty damn close in any set of circumstances. It's designed to get the job done, not to look good and sell well. That's Microsoft's territory.
They did the first test badly. This gets them rapped in the press and they sit down and wonder if htey really did mess it up. They then do a second test, where Linux does indeed come out much better than in the first, although it may not have beaten NT.
They decide that to salvage their image they should have a crew of Linux Experts tune linux and rerun this second test that no-one has seen, hoping that they will not be able to do much better.
Possibly linux will come out on top in the third test and they can turn around and show the second test where they got similar results and everyone will know that they could in fact configure the linux systems, at least in the second test. If, on the other hand, NT comes out on top, their first test is validated and they are even happier.
Of course, this is so far-fetched that it isn't ever going to happen :-)
Logi
Logi - I can do anything, but not everything.
I am no hardware device programer but If NT can perform otpimally by writing the drivers for the hardware being used, why can't use do the same for linux or freebsd.
"Never stick an electrical appliance down your pants." -Tim Allen
1) "are we prepared ...?" - i personaly am (after reading posts here :).
2) "how many have it ...?" - do not know. and i would like to know. and i would be even happier when enought devlopers have such.
few days ago my friend forwarded me some article from www.winntmag.com (can't read it whole, IP is unreachable) and we take little argument about whether NT is better in "enterprice". as i'm taking word "enterprice" as compromited (main stram media use it without thinking, what does it mean and are using it with word "linux" too) i argue with him, that linux is better in enterprice. my friend told me, that in enterprice NT with well written app + enterprise HW is realy better (he uses arguments from author of article at winntmag which published some disadvantages in linux kernel too). he told me something like "NT is fairly good kernel with very buggy and shitty shell". after clearing what "enterprise" mean i agreed with my friend. linux (for now) is not designed and tested for enterprise projects (enterprice - BIG load, BIG server, BIG hardware, ...) so NT acts better in such environments (for now).
hany
If that happens we should point out that the setup is tailored to NT's strengths and Linux's weaknesses. Ask people to consider whether the hardware might have been chosen with that in mind.
Also ask them if they're going to be running these 4 processor, 4GB, 4nic machines in their businesses and point out that these results say nothing about how NT and Linux compare on more mundane machines.
Ask people to consider the following: Tests like these are expensive. MS has money, the Linux community doesn't. If Linux beats NT on some machines and NT beats Linux on others, you're never going to see test results for the systems where Linux is superior.
Last but not least we need to show that we don't sweep problems under the rug, we fix them.
--
Fuck the system? Nah, you might catch something.
CGI's are slower than ASP or ISAPI simply because of the overhead of creating a process rather than just a thread. Apache needs a similar threaded equivalent to ISAPI to really get the speed differential down.
Other net services, such as? On the web servers I've deployed, for security reasons they only do http servicing. You don't even want them to look up DNS - just log the IP address and get out of there.
Andrew van der Stock
I mean really. There can't be too many people doing this.
Most file servers I've seen are single CPU machines with less than 1GB RAM and huge amounts of disk.
support gun control: take guns from cops
"Slashdot on NT" Day
One day a year, slashdot runs on NT, just to see the difference - on the same machine it runs linux. Maybe VA could donate an identical box just for marketing purposes. Of course CT would have to stomach figuring out how to convert slashdot to run on NT (no small task I imagine - does MySQL even run on win32?)
support gun control: take guns from cops
The fairness of this is dubious ... Linux probably has some advantages on some hardware as well. Pardon me, but is this a driver issue (just making sure)? Is this more a capability of the intel cards, or NT?
Last I heard Nt was terrible with gigibit ethernet, and linux, Netware, Solaris, et. al. kicked ass on the tests, getting over 800Mb/sec throughput - NT IIRC was stuck at less than 350Mb/sec. So to even out why not stick a gigabit ethernet adpter in the linux box and a Gb ethernet switch? If you don't think this is fair, then how is it fair that they can do that little trick with NT?
This definitely sounds like a driver issue. Is it?
This is definitely true and should not be allowed. I don't know anyone who uses NT over NETBEUI as a file server ... everyone uses IP. Well, maybe some folks do, but, NETBEUI is definitely "trending" towards non-exisitence (for reasons other than speed).
support gun control: take guns from cops
The Mylex raid controllers are the ones for linux, not the AMI MegaRAID. Not to mention the Mylex is a great performer for NT as well. Everyone knows that; I'm surprised Dell ships the AMI controller with linux. VA uses Mylex.
support gun control: take guns from cops
That story's available here:
http://www.zdnet.com/sr/s tories/issue/0,4537,387506,00.html
They mention Samba and Apache stats. To get to the graphs, go to the related links item on the right side, and click on one of the items.
Ben
Unfortunately, they specifically picked hardware where Linux did worse than NT, they specifically tuned the Linux system down, and the NT system up, and they employed any number of underhanded methods to get the results they got.
When I see a fair benchmark, then we can start worrying about "fixing" Linux.
Ben
Alan Cox brought up a few points about the tests here:
http://linuxtoday.com/stories/5631.html
He mentions that we really should be benchmarking Zeus or a faster web server under Linux versus IIS if we want to find out how fast the OS can serve static web pages. IIS is the fastest web server for NT, so we should be able to use the fastest web server for Linux for the tests. If you want to compare NT versus Linux, then get the fastest web server for both.
If you want to compare capabilities, then use Apache, etc. Use the best product for the job.
Additionally, he does mention that we should use NT clients or at least a mixture of both, for the tests. At the company I did some consulting for, they are standardizing on NT. Microsoft's roadmap is all NT for the future. Why would you want to benchmark against a dead-end technology like Win9X?
Somehow I think these tests are rigged. Notice, Mindcraft only offered to rerun the tests after they ran a second one, which they didn't release the results of.
I think we need another party, that's neutral, to do some benchmarks, and take Mindcraft out of the picture altogether. They already admitted they fouled up, and they shouldn't be trusted to do the benchmarks again, because we will be giving them credibility they don't deserve.
Just my $.02
Ben
The Mindquest retest is a trap. They are using hardware that was designed for NT and nothing else. A reasonable solution is to let a company like VA Research loan 2 boxes known to work well with both Linux and NT. Do the tests on those. Agreeing to do the retest with their current hardware is the trap they have set for Linux. Don't been pulled into this trap. Should the numbers show that NT out performs Linux on the hardware provided by VA, then so be it. At least we will know it was not a hardware issue, or that the test was rigged in some way. I don't trust any company having dealings with Microsoft. Mindcraft is just dangling the cheese in front of the trap door. If Mindcraft will not let us spec which hardware to use with Linux then we should just Ignore this whole retest issue.
You have been assimilated.
Why not run the tests for a week or so straight? I guarantee the NT server will leak memory, bog down, start swapping and eventually crash if left chugging along for more than a week under a high load. The linux box will win that battle hands down... our $20k NT web servers need rebooting about once a week... lets see, our $1000 RedHat 5.1 box... 5 months and counting...
Key Labs appears to be trying to do this. I withhold any opinions or endorsements of the project, but there it is.
Well, I don't know about the rest of you folks, but if NT beats Linux, I'll be switching to NT. I cannot afford to use a "Slow" or "Unstable" operating system for my fileservers & webservers. If my clients find out that I use a lesser system, I'll be eviscerated for sure.
This whole thing is flawed. The MS folks simply cherry-picked a specific set of not-way-ordinary conditions which their own tests found NT to be superior on, so they can say "well what about on THIS junk? Look how well NT blah blah." Do a *real-world* test- To wit: "Here is your budget of X dollars; you guys BUY/use MS-whatever and pick your system, same-same for you Open-Source guys. Compare the performance on what you got for your money- no other strings." This is what the REAL market has been doing, which is why Bill is purchasing rigged comparions.
Nothing is *always* the best- never bring a knife to a gunfight.
DC
"Gentlemen.. START YOUR ENGINES!"
Your Servant, B. Baggins
We don't exactly have millions or even thousands of dollars to be throwing around to cook books, forge benchmarks, and buy off reporters.
We react so violently to Mindcraft, not because we think Linux will always beat NT, but because the tests were so obviously rigged to produce marketing data as opposed to objective decision-support data. We are concerned that some people might believe this tripe because MS has a bigger marketing machine than we do.
We are justified in our wrath because we know MS has tried to do this before, with their disastorous Astroturf campaign, and their all-but-outright-fake demonstrations in the antitrust courtroom. And, as in a trial, if you throw enough money and time at a problem, you can prove (or disprove!) any criminal charge.
To say that the Linux community spreads FUD the way that Microsoft does is baloney. We spread polemic, yes. We p*ss of reporters when they p*ss on our OS, yes. We don't lie to people and try to mislead the gullible.
It doesn't matter that NT might be better in one contrived set of circumstances, because we can as easily contrive a test where Linux does better - why don't we repeat the Mindcraft test on a couple of 486/66's from the PC graveyard, and see what happens?
Mindcraft's credibility is gone in any case, because the press has very neatly seen to it that they shall always be perceived as a cog in MS's marketing machine. Whether this is correct or incorrect, it is a likely scenario, knowing the prevailing winds in Redmond, and that is why we righteously rant. Take it or leave it.
If SP5 has the same effect as SP4 does (50% drop in disk speed) then I've no problems with that..
By the very fact of re-running the tests, they admit the first one was invalid.
By the very fact of running the tests at all, they admit Linux matters.
By publicizing any part of this fiasco, they publicize the entire sordid history of it.
--
Infuriate left and right
Is it only me or doesn't it seem strange that Mindcraft insists on .
1) Use the Setup from Test #2
2) Use the tuning parameters from Test #2
After all, they were being critisised for not giving enough information to be helped.
If they were realy interested In comparing the two product they would allow any patches to the NT/Linux kernels/drivers, after all, that it what customers are going to use, the latest and best.
They seem only interested in the old results, not the right results.
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
Oh... boy...
American Megatrends today released a new multithreaded driver for their MegaRAID controllers.
Check out www.linuxtoday.com or look here.
Sigh... The irony...
The Internet: Most successful software development company in history.
----------------------
There is no K5 cabal.
I am not the real rusty.
One famous quote comes to my mind...
or something along those lines. I don't have a copy of ANH handy right now...
Fairer, but still in an MS "emulation" mode with Win9X clients and Samba. I wonder if the results would be different if instead of MS networking/Samba the tests just used 'open' internet protocols such as http and ftp?
I have to admit that I'm getting a little suspicious about this constant claim that this guy at Mindcraft is receiving ``rude and vulgar emails".
First, if you are going to make a strong statement about *anything* on the Internet, you are going to get flamed. Especially if you defend Micro$oft -- do you think every email Gates receives is a love note? The potty-mouths are out there, & the best thing to do is to publish their subliterate ravings for all to laugh at. Or ignore them.
However, I figure this Bruce guy knows this, & would have posted a few by now if they existed. (It would be the kind of evidence M$ needs to prove their claim that Linux is an OS written by 16-year-olds in their bedroom on their big brother's PC.) But all he has done is whine about receiving them.
In short, methinks this AC smells like a troll.
Geoff
I think I see a trend here. Maybe for them it really would be easier to muzzle the entire internet than to produce p
Their reputation is stained because they weren't objective in testing. This kind of company does not belong in business. By refusing the test, nothing Mindcraft publishes will be taken serious and they will become the next DataQuest.
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
That is what should happen. Let each camp choose a system and its components then see who is faster. Phase 2 is to then take the loser and run it on the winners hardware just to see if it can get a boost from the hardware combination. Specifying something like # of Intel CPUs, Max NIC speed, Max memory size, and no over clocking. Businesses can look at the results and say that today, system components x,y,z are the fastest when used with the x OS. Not this mock trial that Mindcraft is proposing.
I'd still like to see that NT is really the production NT with just a service patch and not some tuned version just for this test hardware. That is kinda why I don't like seeing the tests run at Microsoft. Too much of a chance that there is a 'man behind a curtain' blowing smoke and moving mirrors.
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
IIRC it was NT clients that Linux worked best with and Microsoft had tuned NT Server for Windows 9x clients. Surprising because Micros~1 states that NT is the corporate client. Why has Micros~1 not been able to merge the file/print client systems yet is beyond me. It looks like this test will favor NT just by virtue of the cleint choice. Hey, make them OS/2 clients and watch NT serve those. Micros~1 won't go for that because they most likely cripple serving OS/2 and the advantage will go to the more neutral Linux.
IMHO
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
Set a $ limit for combined hardware, OS, and software.
Withous Micros~1's money, Mindcraft is sunk. M$ won't pay a company that produces benchmarks that make NT look bad. The Mindcraft guys know that the future of their [pathetic] company likely rides on the outcome of this test. Logically, it follows that there is no way in Hell they will release a benchmark that shows anything but NT soundly trouncing Linux. If Linux comes close they'll doctor the data, if Linux wins, the results will never see the light of day. If Linux loses, you can bet that a copy of their report will be on every CIO's desk within a week. Linux will be publicly trashed in front of the IT types, and it will likely take many years to get the pointy-hairs to listen to Linux advocates again. Linux has very little to gain here, and a lot to lose.
0 1 - just my two bits
It looks like the hardware for the server will be the same as the original test. Correct me if I am wrong, but wasn't one of the problems the fact that some of the hardware was not fully supported on Linux but well supported on NT? A test using completely compatible hardware for both seems to be the proper thing to use.
Mindcraft is participating in this third round of testing at their own expense, as they point out in bold text on the invitation page. They are doing this, I believe, to recover some of the credibility they lost by conducting their first benchmark in an extremely sloppy and biased manner.
And why are they doing this? Because for a company like Mindcraft their credibility is their cash cow -- if their test results can't be believed, no one's going to pay for them. So before we linux advocates get ourselves all worked up over the opportunity to prove what linux can do, we must ask ourselves: "what's the ultimate goal of this test?". Or perhaps that should be phrased "who is the ultimate audience of this test?".
The answer, I believe, is that the ultimate audience, the target, of this 3rd benchmark is Mindcraft's collective future customers, including, me must presume, those customers from which Mindcraft might expect repeat business. In a word, Microsoft.
So, while I'm encouraged by the news that linux will get another run at the benchmark, I'm not entirely satisfied that this will be a completely unbiased test. Although it's encouraging that Mindcraft has opened the test to tuning by linux experts, they still have dictated the structure of the test, and it seems to me that there's room there for bias.
BTW, I visited the Mindcraft web site shortly after the publication of the initial test results. Their home page included some text that read something like (paraphrasing here) we work with the customer to identify their test goals, then design a test to produce the desired results. In other words, Microsoft got what they paid for. It seems interesting now, in the aftermath of the Mindcrap Affair, that those rather damning words seem to have disappeared from their site.
--JT
As stated in a previous Slashdot article on the Mindcraft fiaso (I believe by a Samba employee) he stated that the test although obviously biased based on credibility was a usefull source for NT administrators for tuning optimally. The truth is many administrators of NT servers don't have one good source for tuning tips to increase uptime, speed, or whatever.
We in the Linux and Unix community should not approach this test as necessary to defend the honor of our favored OS but as a way to get information on how to tune our own servers.
My guess is NT will come out ahead because of its RAID support and ability to bind network cards to acheive very large pipes as the four network cards in the Dell PowerEdge can acheive 300-350Mps. For running application servers which are not used for web-serving or external applications probably a quad-Xeon with NT although bloated is extremely useful. However Solaris-based Office servers although equally expensive and semi-counterintuitive appear to handle both intranet and internet tasks with less burden.
We should appreciate that Mindcraft for free is providing us with valuable information on tuning Linux servers. However if you are running a massively parallel system probably Linux is not the right choice as an OS. Even a microsoft "win" should not be considered a loss for the community. I applaud Mindcraft for attempting to retry their tests with the tuning suggestions of us in the Linux Community.
Linux tuning information can be difficult to find and the Mindcraft retest may allow us to improve the usage of more mainstream servers and clients. Remember that losing a battle does not always imply that the war will be lost.
Disclaimer: This is not an apology for MS or Mindcraft
Please remember that this is a benchmark regarding SERVER performance. Who cares what's on the average desktop. I've never worked for a company that set up an application server from someone's desktop. (OK, it might have been ``set up'' on someone's desk but when it went ``live'' it went into the data center.)
Of course, the average small business ain't gonna buy a quad Xeon system for their intranet server. But that doesn't make this benchmark's HW configuration any less valid, important, whatever.
I would like someone to do a study of how Walnut Creek was able to pump out so much data with a mere dual PPro (until their recent upgrade). IMHO, a quad processor machine isn't really that necessary unless you're doing something like running Oracle Parallel Server or similar application. Just how many business applications written for the Intel platform are really able to take advantage of SMP? Damn few I'll bet. The PC community (maybe this is Intel marketing machine at work) has convinced everyone that you need faster and faster CPUs all the time. Many of the applications that people use their servers for are I/O bound. Spend your money on faster disk drives, caching controllers, etc. But you sure won't hear that from Intel unless they decide to get into the disk/disk-controller business, that is.
CUR ALLOC 20195.....5804M
Agreed. The disallows one of the main features of OSS: Software patches come out at a much greater rate than they can (or at least do) than one sees from a Cathedral-type development model. Is it possible that MS has had a say in the testing format because they don't want end-users to get the idea that receiving patches in this timely a manner is a Good Thing? That wold only serve to strengthen the arguments in favor of the OSS development model. I smell more than just FUD directed at Linux here; it's also directed at OSS in general.
Does anyone really buy software like this and not apply patches on a fairly regular basis? Have I spent too much time in the VMS/DEC UNIX/HP-UX world (i.e. traditional ``industrial-strength'' OSs) that I'm missing something.
Jeezy Pete! If I found out that I could get a patch that ran my HSZs 3X faster or I could make my backups run 3X faster and I didn't apply it my boss would (and should) be bitching me out no end.
IMHO, this 3rd test is an attempt by Mindcraft to regain some bit of respectability after Microsoft left them twisting in the wind following the first ``test''. How long do you think it'll be before Mindcraft either changes the part of their Services web page that says:
doesn't apply to manufacturers any more or that they vow never to do testing for vendors again.
I get this feeling that Mr. Weiner is still smarting from the experience of working with Microsoft on the NT-vs-Linux ``test''.
CUR ALLOC 20195.....5804M
It looks an awful lot like the hardware for this third test is slanted to favor NT. Another consideration is that Mindcraft insists on holding down the software revs to the date of the second unpublished test. To me, this rule eliminates one of the greatest strengths of Linux: that it enjoys a far faster software upgrade lifecycle than a commercial product can ever hope to achieve.
Everyone knows that Linux performs far better on lower-end hardware than NT. For instance, my own web server runs on a P133 with 64MB of RAM, and would stomp all over a similarly-configured NT system, even without tuning.
In my opinion, the fact that it takes three tests before the two operating system communities agree that the tests were unbiased puts MindCraft in a no-win position. If I were looking for an independent lab to do tests for my company, MindCraft would never be considered, based on their performance regarding this issue.
I really don't care if Linux 'loses' this third test, because I find it much easier to compete with clueless Internet service companies who choose NT as their solution. If Linux even comes close to NT's performance, then what does that say about the technical merits of commercially developed software from a company with hundreds of programmers and millions of budgeted dollars?
If Linux 'loses', then you can bet the legions of programmers will tune the code to beat the next benchmark. Result: even faster service on high-end hardware, as well as middle- and low-end real world hardware.
If Linux wins, well then... *chuckle*, we knew that, anyway.
i really don't think that with the amount of dogma surrounding these benchmarks much good can really come out of the tests anymore.
Uh... DUH! Benchmarks very very very rarely have anything to with anything apart from benchmarking.
--
The Internet is the Suppository of All Knowledge. You get it in the end.
Its clear that setting up an OS bench mark for PC's with pre-choosen hardware has too many variables. OS X may support this NIC better than OS Y supportes this NIC...the fun can go on and on. Benchmarks have always had too many variables that lead to misleading results. I propose this: Why not have a benchmark that truly represents a real world situation based on cost/benefit. For example, allow $5K to be spent on hardware and software for each OS configuratoin. Of course, the $5k would have to represend real market values (ie vendors could donate software/hardware, but the market value of these donations would be subtraced from the $5K). This would allow each OS to use the hardware that works best for the drivers avaliable. Sure, OS X may support better hardware that is cheaper than the hardware for OS Y, but isn't that what customers should really be interesed in. Further, if OS Y ran better as a cluster, it doesn't matter as long as you don't exceed the $5K limit. In all, I think this kind of benchmark would be the most accurate and it would be the most benificial to customers that have budgets (that would be everyone).
just my 2 cents
\forall code \in C, \frac{\Delta readability(code)}{\Delta t} < 0
Are they going to use BETA hardware (AMI RAID) again?
There has to be a catch. If I was to do the configs I would not accept these conditions without knowing the hardware configs ahead.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
We do not have a single 95/98 in the company (all is NT or unixes).
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
The third will not be fair.
They select the clients - these should be selected based on market stats.
They select the hardware - this should be selected based on being supported by both systems.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
In this case the Linux team can simply put a Microway dual CPU alpha at 500MHz with 8MB L1 cache for the same amount of money. Which obviously wins. And this is not what Mind$/M$ whant.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
I think it is a good opportunity, but I agree with some of the other posters in that this "Open Benchmark" is probably a trap that will play into the hands of Linux detractors.
I don't see any problem with the restrictions on software and software tuning. I don't see any intrinsic problem with using a high end machine either. If Linux doesn't do SMP as well, this will be a good incentive to improve it.
The things I do see problems with are the clients, and the machine hardware. Ideally you would test samba performance with both windows NT and 9x clients, not just the configuration that favors NT. Also, an "Open Benchmark" should be composed of hardware that is comparably supported on both machines. I think that there should be some discussion over the hardware to be used. The hardware used on the first test, like every other part of the first test, is suspect.
I remember a previous thread of a previous article introduced the saying "Chase the dream, not the competition". I think it definitely applies here. I want to see linux stressed so that it will be improved. It doesn't matter to me if NT beats it, so long as the test is fair.
-OT
if I had to use drives, I'd use solid state drives (the kind with built in battery and automatic disk backup) for ultimate performance.
I don't think you would want to do that for NT systems. I've had direct quotes from Microsoft employees that the EMC drives are not "reliable" under NT and he recommended NOT using them.
BTW... two different services on the same box? Aside from the fact that it is bad design on the NT side, the two functions really require different configurations.
Well, that brings up a point. It would be interesting to run both tests at the same time under NT and Linux and see how they come out then!
If you develop your product or site for NT, you are stuck on systems that NT runs on, and those are pretty pitiful compared to the "big iron" available in the UNIX world.
And there is no indication that that is changing--the upcoming 64bit and enterprise versions of NT are still years behind current high-end UNIX systems.
Similarly, the concern about SMP support also has much less relevance for Linux than for NT. Unlike NT, Linux can be scaled easily and cheaply by using multiple single processor machines.
Another crucial difference is that Linux has a growth path: you can get started with a small, single processor machine, and if your business takes off, you can get a 256 processor IBM system. That's because Linux uses POSIX standard APIs. NT, on the other hand, is, for practical purposes, stuck with the Win32 APIs, so you better like the four processor performance, because that's all you are likely to get for now.
And Mindcraft's survey doesn't take into account cost. How much does a four processor SMP NT machine cost vs. four single processor Linux machines? What about all the software licenses? What about all those other little bits of software NT machines need, you know, the ones that cost $50-$100 a piece? Even if NT were faster, what matters is cost.
Only fools make decisions on server platforms based on benchmarks like Mindcraft's. The best that a test like Mindcraft's might be good for is to weed out obviously bad apples. But given that both Linux and NT are already used widely as servers, they clearly pass that test.
I have no doubt that for bursts of time, NT is quite fast, and can match any other OS for RAW speed for short periods of time.
The horrible stability, the very poor multitasking when many services and applications are running, and the lack of good remote administration of Windows NT is why I use *nix on all of my servers, and advise others to do the same.
Regardless of what the benchmarks show, I will continue doing the same. This raw benchmarking is only a part of an entire OS equation. NT just falls far too short in too many other areas to be used by any serious enterprise.
Unless the speed difference is huge, stability and robustness is *far* more important to a heavily loaded server than raw speed.
When a machine crashes, it's speed becomes zero.
How about a 6 hour test? That would certainly disable the ability of programs to hide performance hits over short periods of time. Does anyone really care how many hits their webserver can take in 6 minutes and then stop serving?
I remember hearing something about NT being configured to do all logging after the test, not during, while apache was configured to log during the test (with host names -> DNS lookup).
Not to mention, the chances of NT failing under a really heavy load in 6 hours is much greater than in 6 minutes, really helping things out to be more accurate. Noone runs their webserver for 6 minutes.
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
Do even believe what you're saying? Do you really mean that Linux should be able to beat NT at utterly contrived situations that have little to no bearing on real life? Can you point me to one sight where people are requiring thousands of pages per second of static content for only six minutes?
I don't particularly care about samba/NT fileshare performance. That whole network neighborhood is so damn slow on the client end that a slow server wouldn't really be noticed. That thing is a pain to deal with when there are ten compuers on the network. I couldn't imagine how slow it would be with 100. Or the thousands that this test is trying to simulate. Anyhow, I don't really know much about file sharing, nor do I care much about file server performance. Other people can worry about that.
What part of this mindcraft test has any bearing on reality? It's a benchmark contest, pure and simple. It's an NT machine tuned for this benchmark versus what was a Linux machine tuned for bad performance on this benchmark. Now maybe it will be an NT machine tuned for this benchmark versus a Linux machine tuned for this benchmark.
To quote penguin computing's FAQ, "Now, this is a complicated question, and one that benchmarks can't really answer (unless of course, your intended application of the computer is to run benchmarks, in which case benchmarks will be a great indication of the performance you will get).
To quote someone else, "Disraeli was pretty close: actually, there are Lies, Damn lies, Statistics, Benchmarks, and Delivery dates." (found in fortune.)
Benchmarks are virtually useless, especially highly tuned benchmarks. What's the point in saying that if two OS's are configured in such a way as to achieve optimal speed for a useless application one can do that useless operation faster than another.
Static pages server to as many people as are being simulated using hardware with beta drivers has nothing to do with the real world. I defy you to come up with one example that has the sort of usuage pattern that was tested in the mindcraft test (i.e. >2000 page views per second of static pages for only 6 minutes).
So in a sense you are right, you can't dictate the external conditions for a server environment when making a server, one builds the server to those conditions. That's exactly what didn't happen here, the conditions were build for the NT server.
I congratulate the windows people that they can make an OS which can do so well on pointless benchmarks against crippled products. That's really something to be prowd of. Now pardon me while I go challenge my grandmother to a game of basketball...
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
It's good to know that in a corporate environment one only serves static web pages for 6 minutes at a time. And that one picks hardware that one knows is only supported by beta drivers. And that one doesn't tell the people who are trying to help you to optimize your system what's really on it, or, heaven forbid, let them actually see the box. That's what really happens in a corporate environment.
And no corporate environment ever moves over to NT on the client side, nor do they use win98 clients. They immediately reformat their hard drives and install win95 on all those new boxes that they get. Thanks for that wonderful description of what happens in the "corporate environment".
Don't ask me why I fed the troll. I really don't know. I just can't let that ever-so-slightly credible argument go unanswered.
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
Ahahah =)
Its all a big plan to assassinate Linus right?
Conceived by Bill Gates, Dr.Evil and wierd Green Bad Guy from that that one flintstones.
"Computers will never truly be free until the last windows user is strangled with the entrails of the last mac user."
Give each side $X, say $5000, with which to buy hardware and software for the test. Each side can use whatever hardware and software they want, as long as they pay retail for it. _Then_ see who comes out on top.
This more accurately mimics real-life situations: my boss doesn't say, "Here's a $20,000 platform, now put the best OS on it", he says, "I need a really fast rock-solid fileserver. How much will it cost?"
--joe
"It sure was strange to see something on Usenet about me that didn't involve Klingon gang rape." -- Wil Wheaton
I was checking out this weeks ComputerWorld mag. and on the back cover is an add for kingston ram claiming that it will improve your NT server performance by up to 540%. Thinking this to be a rather odd claim, I checked out the fine print - guess who conducted the testing that proved that kingston ram would speed up an nt machine by 540%? You got it! Mindcraft!
sigh. I'm going to pay them to "prove" that my 1993 Subaru sedan is faster than a ferrari.
There was a time when these "benchmarks" did not matter. Now they do and here is why.
In the time of innocence when Linus and others put the fruits of their labours out onto the net and invited anyone who was interested to see what they thought of it and do what they liked with it. There was no great launch or fanfare or hype. The code spoke for itself and under the freedom of the GPL and other licences has spread like wild fire.
Under those circumstances anyone could try this stuff and decide it was rubbish and maybe tell whoever they liked that it was rubbish. Of course most thinking people who come into contact with it have come to take the opposite view, for reasons that I hope all slashdotter's are intimately familiar with, and will do all they can to propel it forward.
There was a chance that Microsoft or whoever could now be running any kind of "benchmark" they like and publishing any results that they dream up and it really would not matter. As the Mindcraft test shows nobody is taken in by this nonsense and it only makes the perpetrators look stupid or desperate or evil or all of those things.
Unfortunately things have gotten a little out of hand. When Linus is quoted as wanting "world domination" and using phrases like "we will crush them" etc. then he is throwing down the challenge, not Mindcraft or Microsoft. And this gets everyone into this mess. As I say in the past, before becoming famous for is rhetoric, these tests could have been simply ignored.
So now Linus and co. have to be man enough to live up to their words. Some how they have to "keep face" no matter what the outcome. It may have been wiser to have kept quite and let the code speak for itself , which it does so effectively.
Now I'm sounding a bit negative and I can't imagine how they should handle this but I have every confidence that they will come out OK.
When Eric Raymond wrote his piece about Linus and his merry men in Sherwood Forest going to challenge the Sheriff of Nottingham a few months ago I would never have believed that it would become reality so quickly !!
I agree with you. And just add this :
what if the test shows Linux is not far below NT ? Then Mindcraft _and_ Microsoft are shamed. For people at Mindcraft that would not only mean the corp dies and they loose their job, but also they'll probably not find any job in that field. Therefore the last test _has_ to support first tests.
And that result might be much more dramatic to Linux. And after the test show bad figures for Linux it will be too late to say it was biased.
But just look at it. Maybe it is true that Linux is not at the same level of an highly tuned NT box ? This is not a shame to be weaker than a competitor in specific conditions, but it is shamefull if noone does anything to change this...
The world belongs to those who get up early. - I'm far from being the king of Earth then
with your conclusion ;-)
The world belongs to those who get up early. - I'm far from being the king of Earth then
A few months ago a first miracle stroke upon M$. Before the M$ trial started everyone (or nearly) though nothing would get out of this. But if Linux is so hype these days, that because M$ people have shown to be very stupid (Gate's dumb testimony where he hardly reminded things was the stronger argument _against_ M$ ever).
;-)
Mindcraft test seems totally biased, and at first glance we cannot expect much thing from it (exactly like the trial). But maybe we'll get some nice surprise ?
However I will not bet a cent on it (unless you give me one
The world belongs to those who get up early. - I'm far from being the king of Earth then
not necessarily, because very few companies tune servers M$ did. In fact the action of M$ proved that it is not easy to have a high end NT server.
Here at work we just buy a PC, install NT4 + SP3 and that's all. There is no customization and sometimes we even forget to stop some services that we don't need.
The world belongs to those who get up early. - I'm far from being the king of Earth then
Then it sounds fair. Let's see the results. I hope they got good linux experts. And I hope they will not cheat with hardware. Like take some HW that is known to lose performance under linux.
Everybody Lies. But it doesn't matter since nobody listens.
Silmaril the clueless wrote:
>Their benchmarks let anyone see (and >REPRODUCE) the relative performance of >NT+IIS or Linux+Apache for a given >high-end hardware configuration
Seems like another M$ fanatic who likes to play with words. FYI, Jemmy reproduced NT numbers, not Linux numbers.
How about this?
Broken drivers (disk & LAN card drivers) on a sabotaged Linux+Apache server only led 50% performance drop when compared against similiar NT machine?
I think you hit is on the nose. This is Mindcraft trying to save their butt's. Microsoft doesn't like negative publicity and someone from Microsoft told them to "try and make it look honest or your history" or something along that line.
Mindcraft is nothing but a PR company pure and simple.
In benchmarks as in research that
follows the scientific method, the top
priority must always be REPRODUCIBILITY.
(_Computer Architecture, A Quantitative
Approach_, by Patterson & Hennessy)
Mindcraft follows this principle
scrupulously. They publish, in minute
detail, the hardware and software used,
and the configuration and tunes
applied. Too bad we can't say the same
for PC Week!
Their benchmarks let anyone see (and
REPRODUCE) the relative performance of
NT+IIS or Linux+Apache for a given
high-end hardware configuration. They
also answer the question of whether NT
or Linux+Samba is a faster file server
for Win9x clients on that kind of
hardware. This is valuable research!
It's a shame that so many people would
rather complain about the 'unfairness' of
Mindcraft's benchmarks rather than
accomplish something useful. Like
actually FIXING Linux so it can perform
as well on NT under the kind of
hardware used in the benchmarks.
All this will be a fair enough. After all, who would seriously recommend a 4-way Xeon with 4 high-performance netcards to run Linux??? Linux, as I see it, does not yet have its strength here. If you would like to use Linux, tailor some other high-end solution, e.g. with multiple servers. If not, then buy expensive hardware from Sun and run Solaris on it! Just don't think that NT is your only alternative.
Others have pointed out that the NT Server configuration is highly optimised to the particular test. Taking advantage of the fact that a fairly small amount of static-only pages will be served, the NT is configured to keep it all in its cache.
While, as Alan cox, among many other things points out, using the (apparently commercial) Zeus webserver might up the Linux webserver performance, but virtually anyone installing a Linux webserver are going to use Apache, so I find find it reasonable to tune Apache instead. But it must be possible for the experts to similarly tune Apache/Linux to take advantage of the nature of the test.
The first test was completely bogus. They won't allow us to see the second. Why should we think the third will be fair? As many people have mentioned, Microsoft's image and Mindcraft's image are at stake here. They're going to make damn sure that the third test shows NT is better than Linux w/ Apache.
I will never believe anything they say again. Hell, even if Linux/Apache/Samba wins I'll be suspicious, because maybe they'll be shooting for a chance in the future to have someone using Linux as a customer.
I'm sorry. But I won't believe anything coming from their labs, even if Linus, Alan, et al, say it's good.
I'm wondering how much tests like these cost from a place like Mindcraft. Would RedHat, VA, the SlashDot community, or others, be willing to pay for WinNT vs. Linux tests on all the stuff we know linux is better at?
What if we had uptime measurements? Or CGI site tests like slashdot (I'd like to see NT handle slashdot half as well as a real OS does!) What if we did CPU intensive Oracle database generated pages? How about other, non-web tests? It seems a shame that this community doesn't have the right to do anything but scream in newsgroups, just because we don't have the money.
Are there Linux experts out there who would put together lists of what would be best to test in our favor, and is there funding or a company willing to do such tests? I wonder...
I think they should test all the brands and style of CPU's..
Like the Intel stuff
PPro
P-II
Xeon
Celeron
P-III
Amd:
K6
K6-2
K6-3
Cyrix
What ever they have!
IDT
Now that a wierd chip
And I know there are couple of odd ball chips out there......
"Windows 98 Second Edition works and players better than ever." -Microsoft's Home page on Win98SE.
I ate my tag line.
-=Ellis (D)25=-
There is alot of tunning you can do w/ 98.. TCP stack tunning and a program called PowerTweak (www.powertweak.com) which can optimize your hardware.. Got a whole 5% increase on an overclocked 350(401 now with optimizing should act like a 421 =P) K6-2.
"Windows 98 Second Edition works and players better than ever." -Microsoft's Home page on Win98SE.
I ate my tag line.
-=Ellis (D)25=-
Dude, I'd radther use 98 as a server than NT...
"Windows 98 Second Edition works and players better than ever." -Microsoft's Home page on Win98SE.
I ate my tag line.
-=Ellis (D)25=-
There is a lot of talk about the Hardware RAID controller, and the drivers for it.
What about setting a $$ limit on the HW and let both "sides" choose the best HW for it's platform?
Put up or shut up! I like it!
This just proves you can fight FUD with facts. Whether Linux "wins" or not, the facts will be on the table. Should the optional tests include being able to swap Apache for one the threaded httpd servers?
I am not your blowing wind, I am the lightning.
Obviously NT will "win" this sham. That doesn't mean a thing. I want to see a long running, real world test. Give $3,000 to the two parties, to run a file server and a web server for 6 months. A mix of Linux, *nix, Mac, Win9x, WinNT, Win31 clients, running automated tests. Reboots of the servers are not allowed. Lets see which gets the highest average bandwidth over the 6 months. Ho ho
>the Linux experts aren't allowed to use
any patch that came out after April 20th...
Yeah, and they'll probably install the new SP5 on the NT system too.
Lowmag.net
If it loses then we lose on a big stage with a lot of important people there to get egg on their faces, if they let their pride get in the way.
The upside is that if we lose and we have the right people there, we can diagnose the situation and learn to make it better. Hopefully there would then be a fix within a couple months and linux would truly match or outperform NT (assuming it doesn't) I don't like how MS is so willing to let this test happen at their lab.
It's all going to come down to spin in the end and it will be a good test of who's more the market darling, MS who has worked the media for years and years or Linux who is the new kid on the block with all the cool toys.
This is my signature. There are many signatures like it but this one is mine..
I've done many benchmarks in the past 15 years and this one appears to be a setup as proposed. I would propose: 1) The benchmark not be conducted at Microsoft labs but at a more neutral benchmark site . 2). Both OS's should start from their latest 'shrink wrapped' version and then apply any patches that are available to the PUBLIC prior to the benchmark. 3). Linux and NT workstations should be included in the mix of Win'95/98 W.S's - most enterprises are not pure Win'95/98! - volume should cover the complete range to saturation/failure. Right now it appears that Linux up against a uniquely tuned NT 4.0 Server. The NT Server itself having been stripped of all unnecessary code modules, tailored to the particular hardware, and processing tailored to handle the proposed load.
Yeah, and they'll probably install the new SP5 on the NT system too.
I dual-boot with NT4 (for 3dsmax) and i looked on Micro$haft's webiste, but I couldn't find anything about SP5. Is it out yet?
"Software is like sex- the best is for free"
Now I am no network expert, but isn't there some way to set a router to share a load acrost two system 50/50? Why not let Microsoft choose a system to be tested as an enterprise server and let the linux experts choose another system(or network of systems). Attach them to to said router and then the router to the internet.
Since these are supposed to be an enterprise systems, let's put them as close to the real thing that we can. The internet would generate the trafic and the internet would not know which system they are hitting. In the end we should see which system handles the load generated by the internet the best.
As for financing, mindcraft should set a budjet, or purchace all the hardware and software so that we are on a level playing field. This would be the constraning factor, (so microsoft could not come back with 50,000 quad PIIIxeon servers). This way there is no hardware de-tunning or anything else that can be claimed to offset the tests.
Basically a straight linux vs. microsoft challange.
As for the required data necessary for the test you can check the logs of the two servers(if they have not been tunned out) or check the log for the router.
If I was an enterprise manager I would not by a NT system an then install linux. I would base my harware on the system that I was going to implement.
That's real life, not a lab.
M$ are devious and malicious enough to do this, and there's no way a Micro$haft-funded test is going to proclaim Linux the victor, it's just not going to happen.
Perhaps if they actually let Linux gurus in to configure things, then they wouldn't be able to fix things, but they won't. They will only accept advice, why won't they let them in if all is to be fair???
Bugs come in through open Windows
~ Artificial intelligence is no match for natural stupidity ~
Is a linux expert actually going to be present at the tests, or help the mindcraft testers the tuning tips?
dont you?
-- your knees hurt, don't they?
Mindcraft HAVE to do it to attempt to correct its own credibility. Personally, I think both Mindcraft and Microsoft should be publicly shamed. Microsoft even more so. They should be made to run television commercials of apologies (a months worth? ;P) for misleading consumers.
Do your best, hope for the best, suspect the worst.
Geez, I wish I could totally hose a benchmark, get nailed for it, then invite outside professionals to do a better job. All the while, Mindcraft is probably still getting paid to run these benchmarks, and I doubt they're going to share the proceeds with anyone who signs up to do the "Open Benchmark" performance tuning.
If a company claims to be able to perform benchmarking studies but can't supply the expertise, it's either an advertising agency or it's defrauding it's customer, IMHO. (Since Microsoft doesn't seem to be upset with the results so far, my money is on the former.)
Another way of looking at it: This is a new kind of Slashdot Effect. The tone of the postings on the Mindcraft Web site shows absolute panic, because the Open Source community has succeeded in undermining their reputation.
Technology reporters are so jaded by decades of Microsoft FUD that many are more inclined to believe posts on Slashdot than press releases from Redmond (or Redmond's surrogates).
The lesson to learn is to stick to the facts. If Open Source advocates stoop to fuzzification of facts, reporters will discount their assertions, too.
When Linus et al... are to agree to this, they should (in order to agree to this trial) force mindcraft to also do the benchmarking on the most common pc used for web servers today.
;)
At a guess, that would be the 486dx2/50 with 8 meg of ram. Good luck to the windows NT team
This all seems a little bad for linux in a lot of ways. Ill touch a couple no one else has I dont beleive. Mindcraft is developing Hardware to enhance NT, thats great NT runs faster taking advantage of some cool stuff. ( yes it is cool even if NT uses it ) So instead of saying.. Gee these guys are ripping the Linux community off. Ask yourself, Why has'nt anyone developed hardware that takes advantage of really cool stuff in Linux that NT does not have. Or ask yourself Why does'nt Linux support some of this really cool stuff, Why doesnt someone just start writing the code now instead of complaning about bad publicity. In the end good media or bad media it does not matter... People will go to what is best, easiest, and most cost effective. So all these tests dont mean anything compared to personal experience of individuals's. Sure its coming from important people, butt it also must make you ask why they are trying so hard to do this and How they truly have an edge, its apparent they have some things that Linux does not and everyones all upset about it or else the posts wouldnt be like they are.
Jeremy Allen
knights@hom.net
Just to bring up a nice point that has been covered by a lot of posts here. Benchmarks are IRRELEVANT. They mean little to the true Measure of an Operating Systems capability. Yet, If the community allows them to seem important then what does that really do to us..?
The proposed Mindcraft comparison is obviously a stacked deck, and the conditions are unacceptable. The only way to dodge this sucker punch without refusing to cooperate is to offer up an equally ridiculous proposal. Say, NT40sp5 vs RedHat 6.0 on single processor PII400s with 128mb of ram, 40GB of disk space, and one ethernet card configured by factory reps to be identical, and with software installed by randomly chosen users and tuned according to written specifications provided by test advisors - one team for MS and the other for Linux - on a 33-33-33 mix of 95, NT and Linux clients. Oh, and lets do the test at Networld/Interop, in front of an audience, using tests developed by an independent organization (by definition not Mindcraft) to be platform neutral, and representative of what real users do - login, access files, save files, read and respond to email, view & publish web pages, and logout.
Oh, and let's run the tests continuously for the length of the show, comparing performance at the outset vs performance at the close.
That might prove interesting...
-- "Never underestimate the power of human stupidity." - R.A.H.
This is to produce a Linux test on the same Mindcraft problem space. That is, solve the same problem (same Web site, same clients, same network). But do it on a Linux venue, and with Linux-friendly hardware. Invite Microsoft, invite Mindcraft, invite the press.
Don't duplicate the hardware. Use inferior hardware, where "inferior" means "posts smaller numbers or costs less". But an "inferior" part may have better Linux drivers. Some people have complained that the RAID controller isn't Linux friendly. Replace it with a cheaper unit that is Linux friendly. Others say that SMP is irrelevant to this problem for Linux, since the processor isn't the limiting factor. Go for a single processor, the same speed as one of the SMP processors.
The point here is not to do a head-to-head, but to post better numbers with cheaper hardware. Then we challenge Microsoft--not Mindcraft--to beat the Linux benchmarks on the Mindcraft problem and their hardware or less. We replace the "fair benchmark" with a "competitive benchmark". We replace the (weak) assumption that the test team is trying to be fair with the (strong) assumption that each side will pull out all stops to post high numbers. This is how sports teams compete (rather than having the refs measure how hard you can throw or how well you can catch, they let you do your best and simply maintain a fair venue). IMHO, Linux engineers can put together a Linux server that is faster than any NT server that NT engineers can create.
People have noted that this is an expensive machine to put together. For you and I, that is correct. For a company with a vested interest in the results, it's an investment. It's a better investment when the company can use the machine for real world applications such as Web service. Relatively sizeable guns such as Red Hat or VA Research can stand to turn an indirect profit from pulling this sort of thing off.
A Linux/NT bakeoff would actually benefit both sides (!). Once we blow their doors off with our numbers, MS will probably find a way to post bigger numbers. Then we tweak Linux, and post bigger Linux numbers. If MS posts bigger NT numbers by making NT a faster server, we have at least forced MS to improve their product. If MS falls down the trap of optimizing NT for this bakeoff and pessimizing it for the real world, we introduce another bakeoff test. OTOH, MS can keep us from optimizing Linux for this bakeoff by making new bakeoffs at any time--that way, they help keep us honest.
--The basis of all love is respect
Linux, on the other hand, keeps more of its services in user space. This means it may suffer a little in this type of benchmark but it makes Linux an all around better application server. We are talking apples and oranges here.
To *win* Mindraft 3 we don't need to beat NT totally, just come close (which should be possible on a properly configured Linux box). This will show that the original test was a crock, and that Linux is still a much better price/performance buy.
Then we should go on to do REAL tests that show how Linux smokes NT's butt when you throw in CGI support and a mix of other net services.
Thad
The Bolachek Journals
I do.
.5Gb memory, 5Gb raided harddisks, multiple NICs, multiple power supplies, and another server to mirror it.
Seriously, when in a banking environment (as I was) there is no way I'm going use a shiity little pentium for a server.
Minimum spec (and this was 2 years ago) was dual proc,
There is no way I would ever *ever* recommend going cheap on critical file servers, or web servers, or application servers.
You're viweing this wrong. Look I'll agree Linux is great for older PCs, it's great for a cheap web server, but until it mirrors, clusters, runs SQL and pushes network traffic as fast as a real world mission critical NT box, forget it.
And no, I've had a server running sweet for 6 months, no crashes, no reboots, other than scheduled downtime.
>"There would be no need for 4 x 100Base-T connections for each host"
But there is! Apart from security against network failure, when serving speed is critical more *is* better, lets face it, network speed is a bigger bottle neck these days than anything else.
Right lets talk about the hardware.
.5Gb) and of course mirrored servers.
Fair enough, for a lazy web server, sticking it on a "normal" speced machine, running IIS or Apache is fine.
Now lets talk large companies, lets choose any sort of database access, banks, insurance companies, anything interative.
What is the use in static HTML these days? Even slashdot isn't static.
So lets forget about the OS or the HTTPD.
In a mission critical environment (that of my customers) I would recommend a serious specced up machine, dual proc (or quad if you want to add SQL to it), raided drive, minimum 2 NICs (for fall over more than anything else),a shit load of memory (current spec I'm throwing these days is
Now if IIS serves faster on that use it. I'm not a bigot over this (although I did work at Microsoft last year *grin*).
I've used Apache for development of static pages, Netscape Fasttrack under Irix for serving static pages (and that crashed weekly - Netscape server software sucks) and IIS for database intensive, SQL bashing call centre apps.
It's tools for the jobs folks, and the sooner you realise that the better. If Linux can't hack it on a mission critical server, I'm not gonna recommend it. And don't give me grief about using NT for mission critical stuff, I've had it running stock market transfers grossing over $300 million a day, and it never crashed once. Ever. I haven't even seen IIS flip out in a production environment.
Of course, there is the possibility that this could be used against
the Linux community as well. They could say somehting like "No one
wanted to participate in the third test because they knew NT would
still come out on top. They prefer to sit back and criticize and cast
doubt than to actually accept the challenge and be shown
inferior."
Yes, this it true - which is why any boycott would have to be
publicized with a press release (published before Mindcraft published
theirs) stating the reasons (that the hardware used for testing is
biased towards NT) - cc'ing Mindcraft would ensure that they couldn't
use this excuse.
paper.
This is nothing more than Mindcraft going into butt-covering mode...
Mindcraft isn't interested in an honest test, they just want to show
(to the media) that they know what they're doing.. it's a PR game
plain and simple.. they want Linux people to put their stamp of
approval on something that they have no real control over...
Why the time restrictions? the Linux experts aren't allowed to use
any patch that came out after April 20th... One of the main points
about the original test was the unsupported RAID card used... so if
someone were to magically release a patch tomorrow that made that card
run 3x as fast, they wouldn't be able to use it.
By the terms in the paper, as soon as someone from the Linux camp
joins, they're bound to put their name on the PR sheet. (which is
essentially just a confirmation of their second test, which they have
already run and won't show to anybody.) Since Mindcraft has STILL not
levelled the playing field, I strongly urge a boycott of this 'test'.
If Mindcraft REALLY wanted to have an unbiased test, they would invite
Redhat and Miscrosoft to sit down and draw up a mutually agreed-upon
hardware list; that way no side is at a disadvantage.
When you're at war, you don't allow your opponent to choose the
battlefield unless you have no other option. We have another option,
which is not to fight. By allowing someone else to choose the
hardware, the Linux side is at a disadvantage. Don't give them more
ammunition against us.
They should test on a 386, then a 486, Pentium, Pentium II, and then Alpha. That would show off what linux really has to offer.
this is a good point.
if linux performs better the HUGE microsoft press machine will be like "but look how much effort was required for the top linux geeks around, whereas joe lunchbox simply had to double-click his way through a couple of installs and he had Windows NT performing almost as well".
that is a best case scenario i reckon.
if linux doesn't perform as well it's bad bad press. hmmm, gotta think of some contigencies here... how about "Mindcraft optimised the AC power supply for Windows NT!" or "Mindcraft put
all the Linux boxes down the hot end of the room".
hmmm, better give my bookie a call -- he'll be interested in this one. whaddya reckon the odds'd be?
--
Rare Window - free your photos
i didn't say the benchmarks themselves would be affected by "dogma"-force. merely that the situation has escalated beyond science. whatever the results, the losing side will not accept them as valid. hence, the test's are useless as they won't prove anything that will be accepted as conclusive or at least they'll always be obscured by a cloud of intrigue. ;)
as you should have seen in the first test -- the benchmark wasn't just a benchmark. biases, alterior motives, and foul play will always spice the situation when the stakes are as high as Linux vs Windows.
--
Rare Window - free your photos
what a showdown! and i thought kasparov vs. big blue was intense.
;) oh well, hopefully projects like linuxtune will start spreading the word and provide a glimpse in getting the most outta one's setup for the average linux punter.
i really don't think that with the amount of dogma surrounding these benchmarks much good can really come out of the tests anymore. the amount of tweaking that'll be going on will remove these benchmarks from the realms on anything meaningful.
sure, i'd love to have Linus come install our web servers but that ain't gonna happen!
--
Rare Window - free your photos
As my friend once sugested to me, give Microsoft and Gnu/Linux $10,000 each to buy hardware and SOFTWARE.
Already Gnu/Linux is going to be ahead saving quite a ton on software here and each could buy the hardware that is best for their system.
It's turtles all the way down.
Looks like Mindcraft is serious aout a neutral benchmark. Finally we can actually have a solid understanding of Linux capabilities. Right now most of Linux credibility comes from experience and Word of mouth. The Mindcraft test will show how much of a lead Linux has. If it so happens that Linux doesnot have a lead, it can always be improved. The third benchmark is a Win-Win situation for Linux. A lot is at stake for MS though. THey must e biting their nails wishing Linus and Redhat wouldn't accept the offer. Wish more benchmarks were like this. Especially Apple's. No one seems to take the honest approach to benchmarking. Remember the Sun/Solaris/Java fiasco?
I liked this little snippet:
We will be glad to conduct this Open Benchmark at any mutually agreeable test site. As one option, Microsoft has volunteered their test lab where Mindcraft conducted its previous tests.
So not only did MS pay for this study and offer their highest level NT gurus for tweaking, they gave them their LAB to do it in. I can see the proposal now:
TO: Mindcraft
FROM: Bill Gates
SUBJECT: We need NT vs Linux test results
Hello, I'm writing because we are taking a lot of heat over this "Linux" thing and we'd like to shed some light on the fact that NT is still the best choice for file sharing and web serving. Here is what we propose:
- We will pay for your time and "trouble" to produce this study
- We will give you unfetter access to our best NT setup and config experts
- We will give you full and total access to our testing lab for your testing
All we ask in return are favorable results for NT which should not be too hard considering what we offer above.
Regards,
Bill Gates
Personally, I think ANY testing lab that 1- takes money from the publisher of the product they are testing and 2- utilizes resources of that company (both personnel AND hardware/lab) is FAR from unbiased. Give me a friggin break!
So,
What I see is our community that has grown up enough to see all this from a mature point of vue.
Beside all this I was just wondering what is going to happen if the new benchmarks are not in the favor of a linux based system.
This is not really new, but as I see it, the choice of a platform is part of the tuning system isn't it?
Besides all hardware and software considerations, we know the first benchmarks were really insulting and a good thing for us.
Still I believe the third benchmarks aren't going to please us enough. So what?
Anyway the benefits to our community are already there so we should continue on this way and not bother too much about the results of this benchmark.
The Linux community should let it be know that they do not accept this challenge from Mindcraft because that is what it is. (It's even being couched as such by Bruce on their website).
Mindcraft, it is clear are not, nor can they possibly be, an impartial third party for two reasons.
1. They are now a known Microsoft front. Their ability to be impartial and unbiased is completely compromised and now also, the public is aware of that.
2. In their view their credibility has been adversely impacted by the Linux community. They are in dispute of the facts of what has transpired. In a court of Law, for obvious reasons, one of the parties in a dispute, cannot also be the judge!
Any interaction with Mindcraft on this acts to validate them. This should not happen because simply, the reason their credibility is shot is because they have acted unprofessionally and in a biased fashion, misrepresented the very movement that they are now trying to court in order to regain some credibility.
It would be better, if some *trusted* parties engaged Microsoft and (say for example) RedHat labs to prepare some systems to go head to head... Leave MindCraft out of the loop entirely.
This would give us benchmarks to compare and in truth, I do think it's time to move from anecdotal to proper benchmark evidence to prove (or perhaps disprove) that Linux is a better performer. Should the results not favour Linux (unlikely in my mind) at least I'd be more trusting of them and they'd be easier to swallow.
I'm not afraid to know which is better, I'd just not trust MindCraft to truthfully tell me.
remember braveheart?! Don't go in the barn linus!
My cube. My friend. My solace. My prison.
How do we know that the version of WinNT being
used is even an off the shelf version? Yes they
have the right to do performance tweaks, but
maybe they are using a totally modified OS that
is not avaliable to the public to run their
test with. I say that the linux testing folks
should make sure that its an off the shelf copy..
maybe bring a few NT cd's with them just to be
sure. I am sure mindcraft is gonna find a way
to doctor this thing.. Maybe this is how they
are going to do it.
Malice
Let's assume that the test will be "rigged" so that NT wins. The reason I say we assume this is because of the Mindcraft credibitily issues that many people have pointed out in previous posts.
:)
The Linux community has to come back and provide patches within the next month or so to improve Linux in the area's it lost in. Then we go out and say, "Look at us!! In but a month we've improved the flaws/bugs/Bad Things in our OS. Let's see NT do that." I belive that we can do this. I've been lead to believe that this is the strength of Linux and Open Source in general..
Basically, if Linux loses, we have to do our best to let the general public know that it lost, but here's why it lost. And here's how we fixed it.
Refusing to cooperate with this test would be DISASTEROUS.. We don't even have a *chance* of winning in that case.
These, of course, are just my opnions
-- Ace
Although I think NT will almost certainly beat linux in this test (due to chosen hardware), this test will probably be good for linux in general as it will give incentive for a lot more development to be done on stuff like SMP, and > 2 GB memory.
I really hate it when dads boss (a linux hater) sends me newspaper clippings with headlines like "Linux fails the High End Test".
2GB memory max, and 2GB file max seem to generate the most flak, poor SMP performance also. (if i ever did any kernel hacking (i wish) the 2GB limit would probably be the first priority)
After saying that, I have to admit I will feel pretty depressed when the results of the test come out, being a linux supporter.
1. Get someone that is impartial to judge and oversee the tests (ie not mindcraft). The independant reviewer should really set up the test conditions, oversea the tests and release the results.
2. Review tests (testing on a range of performance levels seems nice). something simpler that the folks at home can emulate, without selling the house and the 3 kids. The "specific" hardware setup chosen seems a bit bleeding edge, and could have been chosen because of specific stuff put into NT to make it work well.
3. Why can't a linux expert add there own "special" patch, get latest versions etc. Thats what linux is all about, and really if you put that much money into the hardware to squeeze all the performance out of it, You'd likely spend a bit of time tinkering with the software, build a reduced kernel with only the required stuff, that sort of thing. This being another advantage of open source software.
4. Spread it over several sessions over several weeks, release all the intermediate results and get feedback over the web etc, to apply to the later tests. Many people would love to participate in (an attempting to) rub MS nose in something.
5. Make it an annual event.
Shame on you!
There's nothing wrong with bieing over 50, bald, being an ex-programmer or selling PCs.
One day, I hope to be over 55, bald and with all the physical detoriation that comes from excess drinking.
Paddy Irishman
"Captain, I cannot believe my ears!" - Spock
You could always mangle your sentence to prevent writers from cutting and pasting your sentence segments...they would have to use a lot of elipses. Curse every third word, or use a bizarre numbering scheme.
e.g.
One has a good 445 argument saying that 445 Linux is *not tunable* for the 445 reason that it has a decent 445 design with reasonable 445 defaults and there's not much that needs to be 445 tuned!
Just a crazy idea...
Jesse