Slashdot Mirror
Mindcraft Fun Continues
LinuxOnEveryDesktop sent us Mindcraft's comments on a
third benchmark that
will be open to a wider array of Linux Experts (the second
benchmark took tips from Linus which raised a lot of eyebrows: my
favorite being 'did they require Bill to be involved too?')
There are quite a few restrictions, but overall it seems
like a solid chance to show what Linux can really do. Check
it out.
According to Mindcraft's website, they are experts in evaluating
computer systems and they have been in business for 14 years. And yet,
they do not own a lab in which they evaluate computers and conduct tests. Why?
Their server configuration was a dual PII-333, 512MB. They report system Price/performance.
Mindcraft's NT vs. IntraNetware benchmarks:
Server Config: Compaq ProLiant 5500 single 500Mhz Xeon. They report software Price/performance. (see Novell's rebuttal)
And then, sometime between last fall and now, servers needed 3 more Xeons and another 512 Megs of ram, and companies stopped caring about price/performance.
Only now, on the eve of a possibly disastrous test result, are you willing to admit that Linux may be inferior to NT. Why didn't we see such rational comments after the first test instead of the mindless bashing of Mindcraft and Microsoft? As a neutral observer I must say that the Linux community seems to be at least on par with Microsoft when it comes to hyping your system and spreading FUD.
How about an even more "realistic" benchmark that would reflect the outside world: Set a budget and get let the hardware/software fall where it may. Benchmarks are useful only to those in the market to buy and need guidelines for purchases.
Am I wrong on this? What percentage of people out there have similar setups _currently or ordered) in shop that match the Mindcraft study?
> NT supports zero copy transmits out of the disk cache or the web server cache.
Windows2000 supposedly does. Not NT4.
> This means that the data is directly DMAed by the NIC card out of the disk cache or user-space memory buffer out to the wire without having to be copied to kernel space.
But guess what, you still have to touch every byte of data to do the TCP checksums. Zero-copy won't help you much there. (read today's kernel-list archives for more info)
> 2. NT (with Intel cards) supports Fast EtherChannel if connected to a Cisco switch.
So does Linux. (I believe you can do this with the Beowulf patches if you really want to) And Mindcraft won't be using this configuration, anyway.
> 3. Intel adapters on NT can offload the TCP checksum calculation to hardware.
Adapters which are used in about 0.1% of all servers out there, due to the fact that they cost as much as an entry-level server in and of themselves.
> I don't know if SP4 added that support to NT4 or not. It is definitely in NT5
It's not in NT4, which is what Mindcraft is supposedly testing.
> 4. If NT uses NetBEUI it will beat Samba. NetBEUI is much faster than NetBIOS over IP.
It also sucks. Can you say non-routable over the Internet and a general pain in the ass to set up over any sort of large area network.
Again it shouldn't be any part of this test.
Why do people find it necessary to send Bruce rude
and vulgar email on account of these tests? That
can only encourage him NOT to help the linux people.
It is fine to be upset about the test results, to
disagree with them, but sending offensive email
doesn't help anyone, and certainly makes Linux
look less professional, which is not what we want.
If the tests come out in favor of Linux, do you
think Microsoft employees will start sending Bruce
nasty emails?
To compare vanilla Linux vs vanilla NT? Most of the servers out there are not tuned by experts.
Mmmm... This is getting interesting. Jeremy Allison, of the Samba Team, has already answered to the Open Benchmark invitation (http://linuxtoday.com/stories/5630.html) suggesting a modification in the benchmark configuration, namely that the clients used in the tests should also include NT clients and not only Windows 9x. Apparently the performance of an NT server goes down quite a bit when serving NT clients, and both the Samba Team and Mindcraft (or Microsoft) know this well. In one of his previous rebuttals, also available at Linux Today (http://linuxtoday.com/stories/5628.html), Bruce Weiner of Mindcraft tried to justify the use of their setup not because it gave the NT server any advantage, but because there were far more Windows 9x than NT clients out there, and testing the system whit NT clients wouldn't be a "real world configuration". Interesting, if you think that NT is supposed to be the way to go in the near future according to Microsoft and their flagship product... I work at a university. Should we dump the pile of NT workstation CDs we bought some months ago for our departamental network and go back to 98?
I believe there is a clear chance of NT beating Linux -even properly tuned- in these benchmarks, but all the details surrounding the tests, including the hardware and the configuration chosen, don't smell very well... Too biased.
http://www.microsoft.com/mi sc/backstage/column_T2_1.htm
... which hide server failure from end users. The result: Up to 100 percent availability"
Quote: "How microsoft.com finally achieved 100% web site availability"
"Wanke: We needed to figure out why our servers were crashing, which requires them to be taken offline for debugging. For every machine, you have to have an IP address. This problem, for years, has been unsolvable."
"Weeks: Just running CHKDSK on a 36GB server, which is standard after a crash, can take up to three hours. We couldn't have a server fail, or do maintenance on a server, or be able to leave a server down and try to figure out why it went down without it affecting customers."
The solution? Surely it must be to get one of these huge mindcraft servers that are so fast? No?
"The microsoft.com Web site consists of several clusters of servers, each with a number of segments consisting of 4-6 servers containing mirrored content.
Up to 100 percent availability... Impressive.
Paul
Come on.. its just a worthless benchmark. No benchmark can be 100% accurate. Benchmarks are nothing but a device to sell software. Benchmark programs themselves can be skewed by running faster/slower on different OSes.
/.'ers make it out to be.
People who are fighting to get a better benchmark test are nothing but Linux bigots. I remember a time when Linux was used for fun and hacking (not cracking). Who cares if NT shows better "results"? I sure don't. (Don't give me that "my company will look at these benchmarks".. you shouldn't work for a FUD driven company in the first place)
Lets face it.. you (/.'ers) are all anti-Microsoft Linux bigots who want free (as in beer) software and do not give a damn about free (as in speech) software. Anyone with a brain knows Linux is crap in many ways and is not the holy grail
Oh well.. go ahead and moderate this down. The truth hurts.. and it seems many moderaters can't handle the real truth.
If IIS with these configs can't beat the "swiss army knife" Apache then IIS is really bad ! I mean look at what those configs actually meant
p p05_5xtf.htm
:)
These were some that i bothered to investigate.
ObjectCacheTTL = 0xffffffff.
This it to be used if all contents fit into memory and is static (hmm, and guess what the benchmark measures...yes, static performance.) It supposedly turns of the cache "garbage colllector". See http://msdn.microsoft.com/library/sdkdoc/bdg/bdga
OpenFileInCache = 0x5000 = 20480 (max number of cached open file handles) Well caching up 20 k open filehandles should speed up serving static pages
ListenBackLog = 200 (couldn't find it on MSDN), presumably means that listen() can have 200 pending connections.
And what is my point ? well IIS was totally geared up to handle a website that fit in it's primary memory (anyone how big is the "site" in webbench?)
but,pardon me if'you've got a quad xeon running some kind of "enterprise" website wouldn't the amount of data be _much_ more than 1(or 4) GB ? And would hardly be static either, but generated from scripts,databases and whatnot ?
In humble respect of all computer gurus out there, i hope i haven't got this totally wrong
/Micke
The fact that Linux has done this well is interesting, but really folks, take a look at the test -- it is specifically designed to be a test that Linux cannot win at this current time. 4-way processing. RAID support. And with third party utilities which were created for their usefulness, not their ability to shine on a benchmark.
I don't think we're going to find a 4x boost in the results. One has a good argument saying that Linux is *not tunable* for the reason that it has a decent design with reasonable defaults and there's not much that needs to be tuned! [If someone does a partial quote of that last line and puts it in a magazne, they are vermin.]
BTW, a few semi-related notes. If I was going for performance on a web server with Linux, I would chose an architecture far different from a monolithic 4-processor, 4-network box with RAID storage. I would go with multiple simple systems, which would allow me a path to grow that is not tied to upgrading to expensive bleeding edge technology.
If I wanted to do a performance based web server, I'd put all the files in a memory based filesystem. (Or have sufficient memory to make sure that I'm going to get a high cache hit rate.)
If I had to use drives, I'd use solid state drives (the kind with built in battery and automatic disk backup) for ultimate performance.
I would spread the documents (images, html, etc) across several machines. Additionally, I might have "www.myhost.com" assigned to several IP addresses so that each new connection is semi-randomly distributed to a new machine.
There would be no need for 4 x 100Base-T connections for each host. (BTW... are these going to the same network or different networks? Meaning... is NT doing *IP trunking* with these (all four 100base connectors "striping" output on the same network), which Linux won't? That could explain a 4x difference and wouldn't it be funny if one simple feature is all that this test demonstrated?)
The design given for the "web server" and "file server" aren't really good hardware choices for the job. BTW... two different services on the same box? Aside from the fact that it is bad design on the NT side, the two functions really require different configurations.
And unlike in the previous hatchet job, we are going to be right there watching as they tromple over our OS and publicly defame it.
We are going to be standing there with our mouths open saying, "What happened?"
Folks, this is a battle we can only lose. Bruce has his LIVELIHOOD at stake here. NT is not going to lose no matter how open it may seem. Whatever it was that happened in that second test Mindcraft did gave them the confidence to do this test so they could appear to the public as "fairminded" and "open".
It ain't so. What can we do to prevent this? I hope noone important falls into this trap. It could be a black day for us all as the press trumpets "NT prevails in open tests where linux gurus try their best!"
I'd like to point out a few issues with this: 1) Motivation; in point #1, Mindcraft says the purpose of the test is to verify the results of the second tests. No Linux experts ever even had the opportunity to view the second test's results. Why are we rerunning test #2 which no one has had the opportunity to see or critique? Why not retest #1, the one that everyone says is false and inaccurate? 2) Machine used; this machine will not be the Dell PowerEdge that was used for the first test. It may be a machine that was specifically chosen for it's weak Linux drivers or other reasons. 3) Mindcraft configuration; why must the Linux experts use the configuration that Mindcraft used? 4) Microsoft's ability to use the results in press releases; I admit, this part is the reason why the tests are run in the first place; but why is this starting to look like a press campaign? Microsoft has incredibly tricky attorneys; why should we play their game? 5) Will the Linux experts have the opportunity to edit the 'joint' press release that includes quotations, possibly out of context, from the Linux experts? --Curious in Atlanta
It's a little obvious when you think about it. Mindcraft says that the hardware configuration is "set in stone". And, in fact, when they posted messages on the newsgroup, people told them that they needed to change their hardware configuration... but they won't. The key to this entire test IS the hardware.
Where did this specific hardware configuration come from? Microsoft. And Microsoft didn't just pull this configuration out of thin air. They've been doing all sorts of internal benchmarks with Linux systems to see what kind of numbers they could get. In their tests, I'm sure they've come across many favorable Linux comparisons.
But, as expected, they've found a few sub-optimal configurations for Linux which NT does well with. Microsoft has run many tests, and found the hardware that works the worst with Linux but good with NT. This is what explains the email from a Microsoft email address regarding the Xeon configuration... the email that Mindcraft said they did not send. It was done by Microsoft's internal testing.
Once they have the bad configuration, they need to send it off to another third party for "independent verification". Which, I believe, it isn't Mindcraft that was responsible for the low numbers... they were handed a benchmark that Microsoft ran ahead of time and already knew the results for. They just needed an outside party to go discover it for themselves.
This creates a simple smoke-and-mirror effect. Microsoft isn't blamed for this... Mindcraft is. And it disguises the real issue... it isn't software/OS tuning. It is hardware de-tuning done in advance by microsoft.
Consider that each piece of the system is pretty much a non-optimal configuration for linux. CPU, RAID disk, and probably the network. Probably one of these pieces (say, the 4x100base-t instead of a single gigabit ether) is really sub-optimal and replacing it would probably yield incredible results, but you've still got the additional handicaps.
But the Linux community has walked right into this one, thinking they can tune it out. Probably short of re-writing some kernel and driver codes, this piece of hardware isn't going to fly on anything *but* NT. (*BSD will also compare unfavorably.)
It probably is too late now to point this out -- they'll have the claim of "sour grapes" to use.
If you think anyone had any real input into the second set of Mindcraft benchmarks, go have a look at AC's commentary, entitled "Bruce Weiner: Lies, Damned Lies, Statistics":
http://linuxtoday.com/stories/5631.html
slashdot broke my sig
On the other hand, there is some merit to letting a third party pick the hardware. In this case, we will have the opportunity of finding problems with drivers rather than simply using things we already know work well.
If poorly supported hardware is used, however, it must be made absolutely crystal clear in the press release. Nothing less is acceptable.
If the Linux community is serious about doing this, it's going to take a lot of planning. Just because you may not be taking the trip doesn't mean you can't contribute to the effort. After all, the community has always been trumpeted as a strength of Linux. We ought to set up a web site, mailing list or something to coordinate all of this.
To those who make the trip: take good notes!
Looks like Linux did not come out on top in their second test, and Mindcraft is willing to bring in Linux experts to verify the results.
My biggest fear with this is not that Mindcraft will try some underhanded trickery, it's that the Linux community will not accept the results unless linux comes out on top. That is just as bad as if Mindcraft tailored its benchmarks to favor NT.
This open benchmark is a wonderful opportunity for the Linux community to benchmark itself. AFAIK, no controlled experiments on high-end servers have been done to see how Linux stacks up against the heavyweights. If Linux comes out on top, great. But if not, we as a community have to accept that and learn from the process.
This is best chance yet to discover the bottlenecks in the kernel and several critical pieces of software. IMHO, the Linux experts should not go in with the goal of beating NT. They should go in with the goal of squeezing every last bit of performance out of the machine and using the resulting data to fix the problems.
Mindcraft and Microsoft (!) are donating resources to the Linux community in an effort to help us improve the OS. Let's grab the opportunity!
I agree that the real reason for this is because Mindcraft's reputation is in tatters.
However, as long as they are using a RAID card which is known to perform poorly with Linux, it can hardly be considered to be a "fair" test. The Mylex and ICP-Vortex cards used by VA Research and others are known to be fast and stable with both Linux and NT, and would undoubtedly be what is installed in any Linux server of this size.
The network configuration may be a problem too, but we don't know enough about the network cards used to be able to tell.
An interesting thing is to see whether mod_mmap_static (a new module for Apache which pre-maps files into memory) would greately increase Apache's ability to serve files swiftly. Otherwise we already know that IIS trounces Apache soundly in serving files.
Remember, Linux does not have to beat NT in order to win. All Linux has to do is get numbers high enough so that Linux can be considered credible. If Linux is within 10% of NT in Samba performance, and within 20% of NT in Apache performance, Microsoft can tout those results all they want -- all it will be is free advertising for Linux. Remember, performance is only one reason for choosing Linux. The most important reason, FREEDOM, freedom from onerous licensing restrictions, freedom from being at the mercy of one monopolistic company, that reason is one that Microsoft will never "get" because Microsoft is about everything BUT freedom.
Send mail here if you want to reach me.
...that "benchmarking" product A and product B, while getting paid from product A's manufacturer, using product A's consultants, and using product A's own testing fascilities...IS INHERENTLY BIASED!!! Regardless if product A is actually better than product B or not. My God, its so simple yet they can't figure it out!
Mindcraft has no credibility left. Worse, now they have an axe to grind because of how badly they were beat up in the mainstream press (ABCnews.com, Salon, Slashdot, etc). Any "benchmarking" they do from here on in should be ignored completely. If we can learn something from this, such as the need for better documentation, then at least thats a positive for us.
Sig (appended to the end of comments you post, 120 chars)
Anyone read Mindcraft's rebuttals of the reports about them? I found them very interesting...for what they did *not* contain.
In particular, one accusation unrefuted is that Mindcraft's claims of having gone to the Linux community for help are simple lies. Any appropriate forum in which they could have looked for help is archived: a URL or two from Mindcraft showing that they looked for help would set the issue to rest. No such URL is given.
So if Mindcraft are honest they're doing a damn good job of making it look otherwise.
Just to emphasise the other key points here: (1) the hardware has already been chosen by Microsoft as that which best favours NT, and would not be chosen by someone depolying Linux. (2) Apache isn't designed to be fast at serving static HTML because static HTML servers are bandwidth-bound except during stupid unrealistic benchmarks. For such benchmarks Zeus is a better choice, also available for Linux. (3) It's very suspicious that it's the unpublished second test that we're being asked to reproduce.
We already know how these sorts of results look if a less biased organisation does them, because ZDNet did them. Linux blew NT away on every count, by every measure.
--
Xenu loves you!
He said we would get beaten. He didn't say that the results of the first test were correct. There were many mistakes made in the first test. Whether NT would still win or not doesn't matter. We want a fair test. Microsoft knew about the test. They let Microsoft experts tune the NT machine. Why didn't they tell anyone in the Linux community about the test? Why weren't Linux experts allowed to tune the Linux machine in either of the first 2 tests? Don't try to pretend it was a fair test. It wasn't. Maybe we'll lose, but at least it will be a fair test so that we can see the real reasons we lost so that something can be done about them.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
(eg: If Mindcraft insists the expert uses GCC 2.6.3, you're not going to get the same results as if you used PGCC, regardless of how expert the expert is.)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
They did the first test badly. This gets them rapped in the press and they sit down and wonder if htey really did mess it up. They then do a second test, where Linux does indeed come out much better than in the first, although it may not have beaten NT.
They decide that to salvage their image they should have a crew of Linux Experts tune linux and rerun this second test that no-one has seen, hoping that they will not be able to do much better.
Possibly linux will come out on top in the third test and they can turn around and show the second test where they got similar results and everyone will know that they could in fact configure the linux systems, at least in the second test. If, on the other hand, NT comes out on top, their first test is validated and they are even happier.
Of course, this is so far-fetched that it isn't ever going to happen :-)
Logi
Logi - I can do anything, but not everything.
If that happens we should point out that the setup is tailored to NT's strengths and Linux's weaknesses. Ask people to consider whether the hardware might have been chosen with that in mind.
Also ask them if they're going to be running these 4 processor, 4GB, 4nic machines in their businesses and point out that these results say nothing about how NT and Linux compare on more mundane machines.
Ask people to consider the following: Tests like these are expensive. MS has money, the Linux community doesn't. If Linux beats NT on some machines and NT beats Linux on others, you're never going to see test results for the systems where Linux is superior.
Last but not least we need to show that we don't sweep problems under the rug, we fix them.
--
Fuck the system? Nah, you might catch something.
The Mylex raid controllers are the ones for linux, not the AMI MegaRAID. Not to mention the Mylex is a great performer for NT as well. Everyone knows that; I'm surprised Dell ships the AMI controller with linux. VA uses Mylex.
support gun control: take guns from cops
Alan Cox brought up a few points about the tests here:
http://linuxtoday.com/stories/5631.html
He mentions that we really should be benchmarking Zeus or a faster web server under Linux versus IIS if we want to find out how fast the OS can serve static web pages. IIS is the fastest web server for NT, so we should be able to use the fastest web server for Linux for the tests. If you want to compare NT versus Linux, then get the fastest web server for both.
If you want to compare capabilities, then use Apache, etc. Use the best product for the job.
Additionally, he does mention that we should use NT clients or at least a mixture of both, for the tests. At the company I did some consulting for, they are standardizing on NT. Microsoft's roadmap is all NT for the future. Why would you want to benchmark against a dead-end technology like Win9X?
Somehow I think these tests are rigged. Notice, Mindcraft only offered to rerun the tests after they ran a second one, which they didn't release the results of.
I think we need another party, that's neutral, to do some benchmarks, and take Mindcraft out of the picture altogether. They already admitted they fouled up, and they shouldn't be trusted to do the benchmarks again, because we will be giving them credibility they don't deserve.
Just my $.02
Ben
Why not run the tests for a week or so straight? I guarantee the NT server will leak memory, bog down, start swapping and eventually crash if left chugging along for more than a week under a high load. The linux box will win that battle hands down... our $20k NT web servers need rebooting about once a week... lets see, our $1000 RedHat 5.1 box... 5 months and counting...
We don't exactly have millions or even thousands of dollars to be throwing around to cook books, forge benchmarks, and buy off reporters.
We react so violently to Mindcraft, not because we think Linux will always beat NT, but because the tests were so obviously rigged to produce marketing data as opposed to objective decision-support data. We are concerned that some people might believe this tripe because MS has a bigger marketing machine than we do.
We are justified in our wrath because we know MS has tried to do this before, with their disastorous Astroturf campaign, and their all-but-outright-fake demonstrations in the antitrust courtroom. And, as in a trial, if you throw enough money and time at a problem, you can prove (or disprove!) any criminal charge.
To say that the Linux community spreads FUD the way that Microsoft does is baloney. We spread polemic, yes. We p*ss of reporters when they p*ss on our OS, yes. We don't lie to people and try to mislead the gullible.
It doesn't matter that NT might be better in one contrived set of circumstances, because we can as easily contrive a test where Linux does better - why don't we repeat the Mindcraft test on a couple of 486/66's from the PC graveyard, and see what happens?
Mindcraft's credibility is gone in any case, because the press has very neatly seen to it that they shall always be perceived as a cog in MS's marketing machine. Whether this is correct or incorrect, it is a likely scenario, knowing the prevailing winds in Redmond, and that is why we righteously rant. Take it or leave it.
By the very fact of re-running the tests, they admit the first one was invalid.
By the very fact of running the tests at all, they admit Linux matters.
By publicizing any part of this fiasco, they publicize the entire sordid history of it.
--
Infuriate left and right
Withous Micros~1's money, Mindcraft is sunk. M$ won't pay a company that produces benchmarks that make NT look bad. The Mindcraft guys know that the future of their [pathetic] company likely rides on the outcome of this test. Logically, it follows that there is no way in Hell they will release a benchmark that shows anything but NT soundly trouncing Linux. If Linux comes close they'll doctor the data, if Linux wins, the results will never see the light of day. If Linux loses, you can bet that a copy of their report will be on every CIO's desk within a week. Linux will be publicly trashed in front of the IT types, and it will likely take many years to get the pointy-hairs to listen to Linux advocates again. Linux has very little to gain here, and a lot to lose.
0 1 - just my two bits
It looks like the hardware for the server will be the same as the original test. Correct me if I am wrong, but wasn't one of the problems the fact that some of the hardware was not fully supported on Linux but well supported on NT? A test using completely compatible hardware for both seems to be the proper thing to use.
Mindcraft is participating in this third round of testing at their own expense, as they point out in bold text on the invitation page. They are doing this, I believe, to recover some of the credibility they lost by conducting their first benchmark in an extremely sloppy and biased manner.
And why are they doing this? Because for a company like Mindcraft their credibility is their cash cow -- if their test results can't be believed, no one's going to pay for them. So before we linux advocates get ourselves all worked up over the opportunity to prove what linux can do, we must ask ourselves: "what's the ultimate goal of this test?". Or perhaps that should be phrased "who is the ultimate audience of this test?".
The answer, I believe, is that the ultimate audience, the target, of this 3rd benchmark is Mindcraft's collective future customers, including, me must presume, those customers from which Mindcraft might expect repeat business. In a word, Microsoft.
So, while I'm encouraged by the news that linux will get another run at the benchmark, I'm not entirely satisfied that this will be a completely unbiased test. Although it's encouraging that Mindcraft has opened the test to tuning by linux experts, they still have dictated the structure of the test, and it seems to me that there's room there for bias.
BTW, I visited the Mindcraft web site shortly after the publication of the initial test results. Their home page included some text that read something like (paraphrasing here) we work with the customer to identify their test goals, then design a test to produce the desired results. In other words, Microsoft got what they paid for. It seems interesting now, in the aftermath of the Mindcrap Affair, that those rather damning words seem to have disappeared from their site.
--JT
Agreed. The disallows one of the main features of OSS: Software patches come out at a much greater rate than they can (or at least do) than one sees from a Cathedral-type development model. Is it possible that MS has had a say in the testing format because they don't want end-users to get the idea that receiving patches in this timely a manner is a Good Thing? That wold only serve to strengthen the arguments in favor of the OSS development model. I smell more than just FUD directed at Linux here; it's also directed at OSS in general.
Does anyone really buy software like this and not apply patches on a fairly regular basis? Have I spent too much time in the VMS/DEC UNIX/HP-UX world (i.e. traditional ``industrial-strength'' OSs) that I'm missing something.
Jeezy Pete! If I found out that I could get a patch that ran my HSZs 3X faster or I could make my backups run 3X faster and I didn't apply it my boss would (and should) be bitching me out no end.
IMHO, this 3rd test is an attempt by Mindcraft to regain some bit of respectability after Microsoft left them twisting in the wind following the first ``test''. How long do you think it'll be before Mindcraft either changes the part of their Services web page that says:
doesn't apply to manufacturers any more or that they vow never to do testing for vendors again.
I get this feeling that Mr. Weiner is still smarting from the experience of working with Microsoft on the NT-vs-Linux ``test''.
CUR ALLOC 20195.....5804M
I think it is a good opportunity, but I agree with some of the other posters in that this "Open Benchmark" is probably a trap that will play into the hands of Linux detractors.
I don't see any problem with the restrictions on software and software tuning. I don't see any intrinsic problem with using a high end machine either. If Linux doesn't do SMP as well, this will be a good incentive to improve it.
The things I do see problems with are the clients, and the machine hardware. Ideally you would test samba performance with both windows NT and 9x clients, not just the configuration that favors NT. Also, an "Open Benchmark" should be composed of hardware that is comparably supported on both machines. I think that there should be some discussion over the hardware to be used. The hardware used on the first test, like every other part of the first test, is suspect.
I remember a previous thread of a previous article introduced the saying "Chase the dream, not the competition". I think it definitely applies here. I want to see linux stressed so that it will be improved. It doesn't matter to me if NT beats it, so long as the test is fair.
-OT
Similarly, the concern about SMP support also has much less relevance for Linux than for NT. Unlike NT, Linux can be scaled easily and cheaply by using multiple single processor machines.
Another crucial difference is that Linux has a growth path: you can get started with a small, single processor machine, and if your business takes off, you can get a 256 processor IBM system. That's because Linux uses POSIX standard APIs. NT, on the other hand, is, for practical purposes, stuck with the Win32 APIs, so you better like the four processor performance, because that's all you are likely to get for now.
And Mindcraft's survey doesn't take into account cost. How much does a four processor SMP NT machine cost vs. four single processor Linux machines? What about all the software licenses? What about all those other little bits of software NT machines need, you know, the ones that cost $50-$100 a piece? Even if NT were faster, what matters is cost.
Only fools make decisions on server platforms based on benchmarks like Mindcraft's. The best that a test like Mindcraft's might be good for is to weed out obviously bad apples. But given that both Linux and NT are already used widely as servers, they clearly pass that test.
It's good to know that in a corporate environment one only serves static web pages for 6 minutes at a time. And that one picks hardware that one knows is only supported by beta drivers. And that one doesn't tell the people who are trying to help you to optimize your system what's really on it, or, heaven forbid, let them actually see the box. That's what really happens in a corporate environment.
And no corporate environment ever moves over to NT on the client side, nor do they use win98 clients. They immediately reformat their hard drives and install win95 on all those new boxes that they get. Thanks for that wonderful description of what happens in the "corporate environment".
Don't ask me why I fed the troll. I really don't know. I just can't let that ever-so-slightly credible argument go unanswered.
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
I'm wondering how much tests like these cost from a place like Mindcraft. Would RedHat, VA, the SlashDot community, or others, be willing to pay for WinNT vs. Linux tests on all the stuff we know linux is better at?
What if we had uptime measurements? Or CGI site tests like slashdot (I'd like to see NT handle slashdot half as well as a real OS does!) What if we did CPU intensive Oracle database generated pages? How about other, non-web tests? It seems a shame that this community doesn't have the right to do anything but scream in newsgroups, just because we don't have the money.
Are there Linux experts out there who would put together lists of what would be best to test in our favor, and is there funding or a company willing to do such tests? I wonder...
There is a lot of talk about the Hardware RAID controller, and the drivers for it.
What about setting a $$ limit on the HW and let both "sides" choose the best HW for it's platform?
Obviously NT will "win" this sham. That doesn't mean a thing. I want to see a long running, real world test. Give $3,000 to the two parties, to run a file server and a web server for 6 months. A mix of Linux, *nix, Mac, Win9x, WinNT, Win31 clients, running automated tests. Reboots of the servers are not allowed. Lets see which gets the highest average bandwidth over the 6 months. Ho ho
Now I am no network expert, but isn't there some way to set a router to share a load acrost two system 50/50? Why not let Microsoft choose a system to be tested as an enterprise server and let the linux experts choose another system(or network of systems). Attach them to to said router and then the router to the internet.
Since these are supposed to be an enterprise systems, let's put them as close to the real thing that we can. The internet would generate the trafic and the internet would not know which system they are hitting. In the end we should see which system handles the load generated by the internet the best.
As for financing, mindcraft should set a budjet, or purchace all the hardware and software so that we are on a level playing field. This would be the constraning factor, (so microsoft could not come back with 50,000 quad PIIIxeon servers). This way there is no hardware de-tunning or anything else that can be claimed to offset the tests.
Basically a straight linux vs. microsoft challange.
As for the required data necessary for the test you can check the logs of the two servers(if they have not been tunned out) or check the log for the router.
If I was an enterprise manager I would not by a NT system an then install linux. I would base my harware on the system that I was going to implement.
That's real life, not a lab.
Another way of looking at it: This is a new kind of Slashdot Effect. The tone of the postings on the Mindcraft Web site shows absolute panic, because the Open Source community has succeeded in undermining their reputation.
Technology reporters are so jaded by decades of Microsoft FUD that many are more inclined to believe posts on Slashdot than press releases from Redmond (or Redmond's surrogates).
The lesson to learn is to stick to the facts. If Open Source advocates stoop to fuzzification of facts, reporters will discount their assertions, too.
When Linus et al... are to agree to this, they should (in order to agree to this trial) force mindcraft to also do the benchmarking on the most common pc used for web servers today.
;)
At a guess, that would be the 486dx2/50 with 8 meg of ram. Good luck to the windows NT team
This is to produce a Linux test on the same Mindcraft problem space. That is, solve the same problem (same Web site, same clients, same network). But do it on a Linux venue, and with Linux-friendly hardware. Invite Microsoft, invite Mindcraft, invite the press.
Don't duplicate the hardware. Use inferior hardware, where "inferior" means "posts smaller numbers or costs less". But an "inferior" part may have better Linux drivers. Some people have complained that the RAID controller isn't Linux friendly. Replace it with a cheaper unit that is Linux friendly. Others say that SMP is irrelevant to this problem for Linux, since the processor isn't the limiting factor. Go for a single processor, the same speed as one of the SMP processors.
The point here is not to do a head-to-head, but to post better numbers with cheaper hardware. Then we challenge Microsoft--not Mindcraft--to beat the Linux benchmarks on the Mindcraft problem and their hardware or less. We replace the "fair benchmark" with a "competitive benchmark". We replace the (weak) assumption that the test team is trying to be fair with the (strong) assumption that each side will pull out all stops to post high numbers. This is how sports teams compete (rather than having the refs measure how hard you can throw or how well you can catch, they let you do your best and simply maintain a fair venue). IMHO, Linux engineers can put together a Linux server that is faster than any NT server that NT engineers can create.
People have noted that this is an expensive machine to put together. For you and I, that is correct. For a company with a vested interest in the results, it's an investment. It's a better investment when the company can use the machine for real world applications such as Web service. Relatively sizeable guns such as Red Hat or VA Research can stand to turn an indirect profit from pulling this sort of thing off.
A Linux/NT bakeoff would actually benefit both sides (!). Once we blow their doors off with our numbers, MS will probably find a way to post bigger numbers. Then we tweak Linux, and post bigger Linux numbers. If MS posts bigger NT numbers by making NT a faster server, we have at least forced MS to improve their product. If MS falls down the trap of optimizing NT for this bakeoff and pessimizing it for the real world, we introduce another bakeoff test. OTOH, MS can keep us from optimizing Linux for this bakeoff by making new bakeoffs at any time--that way, they help keep us honest.
--The basis of all love is respect
Linux, on the other hand, keeps more of its services in user space. This means it may suffer a little in this type of benchmark but it makes Linux an all around better application server. We are talking apples and oranges here.
To *win* Mindraft 3 we don't need to beat NT totally, just come close (which should be possible on a properly configured Linux box). This will show that the original test was a crock, and that Linux is still a much better price/performance buy.
Then we should go on to do REAL tests that show how Linux smokes NT's butt when you throw in CGI support and a mix of other net services.
Thad
The Bolachek Journals
This is nothing more than Mindcraft going into butt-covering mode...
Mindcraft isn't interested in an honest test, they just want to show
(to the media) that they know what they're doing.. it's a PR game
plain and simple.. they want Linux people to put their stamp of
approval on something that they have no real control over...
Why the time restrictions? the Linux experts aren't allowed to use
any patch that came out after April 20th... One of the main points
about the original test was the unsupported RAID card used... so if
someone were to magically release a patch tomorrow that made that card
run 3x as fast, they wouldn't be able to use it.
By the terms in the paper, as soon as someone from the Linux camp
joins, they're bound to put their name on the PR sheet. (which is
essentially just a confirmation of their second test, which they have
already run and won't show to anybody.) Since Mindcraft has STILL not
levelled the playing field, I strongly urge a boycott of this 'test'.
If Mindcraft REALLY wanted to have an unbiased test, they would invite
Redhat and Miscrosoft to sit down and draw up a mutually agreed-upon
hardware list; that way no side is at a disadvantage.
When you're at war, you don't allow your opponent to choose the
battlefield unless you have no other option. We have another option,
which is not to fight. By allowing someone else to choose the
hardware, the Linux side is at a disadvantage. Don't give them more
ammunition against us.
what a showdown! and i thought kasparov vs. big blue was intense.
;) oh well, hopefully projects like linuxtune will start spreading the word and provide a glimpse in getting the most outta one's setup for the average linux punter.
i really don't think that with the amount of dogma surrounding these benchmarks much good can really come out of the tests anymore. the amount of tweaking that'll be going on will remove these benchmarks from the realms on anything meaningful.
sure, i'd love to have Linus come install our web servers but that ain't gonna happen!
--
Rare Window - free your photos