Slashdot Mirror

← Back to Stories (view on slashdot.org)

ZD Critiques Mindcraft Benchmarks

Posted by CmdrTaco on from the stuff-to-read dept.

SFraser sent us a link to a pretty decent critique of Mindcraft's benchmark which claimed NT was faster. Monday's story about the tests here on Slashdot about the report was one of the most active Slashdot stories ever.

16 of 141 comments (clear)

  1. Mindcraft's #1 error by Anonymous Coward · · Score: 2

    Any real world cmpany who is going to run a Linux based server is going to have a sysadmin with Linux experience. Mindcraft basically admitted that no one conducting the tests knew dick about linux sysadminning just by the moaning about lack of documentation, most of which even someone with even a few months of Linux experience would be able to handle without having to RTFM... for that matter, there can't be too many Linux sysadmins who don't have a continually growing library of O'Reilly-type manuals. If they had really wanted to be fair, they'd have hired a outside consultant to set it up (something that real-world sysadmins do on major projects when they get in over their heads)

  2. Show me a Benchmark w/NT 3 times faster than Linux by Anonymous Coward · · Score: 2

    Does anyone still beleive that this benchmark
    is the result of a poorly tuned Linux system ?

    Personaly, I beleive these mis-configurations were deliberately done, by people that knew exactly
    what they were doing.

    They even choosed the right hardware (4 ethernet cards), because they were aware that Linux would try to route all the packets trough a single network card.

    They don't even show the routing configuration and the subnets used in the test.

    Was Windows NT using TCP/IP or NetBEUI ?

    This way, this benchmark only proofs that NT with
    4 network cards is faster than Linux configured
    to use a single Network card.

    This might be the reason why the Linux system has peak performance at nearly 100 Mbits/s: the limit for a single network card.

    From the list of processes running before the tests were done, we can see that there was no "httpd" and no "smbd"/"nmbd".
    This leads to the conclusion that both services were launched by inetd.
    This is a very very slow option: each time the server receives a new connection the system will launch a new process.
    The defult setting of red hat Linux for apache
    and samba are standalone servers: this is much much faster.
    Using inetd will realy realy HURT Linux performance.

    The settings in the apache flags "KeepAlive" and "MinSpareServers" are consistent with the fact of
    having apache and samba launched by inetd.
    I just cannot call this a tunning. These settings are much worst than the default settings for both services.

    The WideLinks flag was also a very nasty setting: Windows users cannot create Links and this falg wasn't necessary. It you want safety you can chroot the service. Even if you have several shares, you can mount all the shares in the same subdirectory tree and use "chroot" to that subdir.

    Probably without this misconfigurations, NT wasn't
    even fast enough ( 4 network cards versus 1 wasn't enough).

    Finally, I don't think that this poor results have anything to do with kernel SMP performance.

    This is just what I think.

    An. Coward.

  3. Does Mindcraft have a bias? Novell thinks so by Anonymous Coward · · Score: 3

    This isn't the first time Mindcraft has been questioned about how they "test" file server performance. Check out what Novell http://www.novell.com/advantage/nw5/nw5-mindcraftc heck.html> said about a previous study.

  4. Fixed link by bjk4 · · Score: 2

    http://slashdot.org/articles/99/04/14/0042212.shtm l has the full story (with 780 comments.)

    -Ben

  5. Corollary Links by bjk4 · · Score: 2


    Here are some other responses to said report:

    http://lwn.net/1999/features/MindCraft1.0.phtml -- Linux Weekly News
    http://www.linux-hw.com/~eric/mindcraft.html -- Linux Hardware Solutions

    -Ben

  6. .and here's a short list spec for such a web site by Kurt+Gray · · Score: 2

    1. A copy of the Kernel recompile How-To

    2. Some typical Apache httpd.conf settings for
    enterprise web sites.

    3. A detailed list of which Apache modules can
    be disabled on a plain vanilla web site

    4. Ethernet driver recompile and tuning guides.

    5. Which network services you can disable on
    a plain vanilla Linux web server

    6. Web server tuning tips regarding file I/O
    and how NFS, RAID, SCSI, and multiple disks
    affect file I/O performance.

    7. Real world perfromance testimonials.

    8. Slashdt style Q&A forums

    9. Ethernet and network topology advice

    10. Explanation of DNS tricks such as round-robin
    and gateway switching.

  7. tuning howto -- volunteers? by scenic · · Score: 2
    well, I'm seriously thinking about doing it now. I tend to be the Linux proponent on my project team (I run it at home, and just ordered a machine from Penguin :-).

    I have some bandwidth (not a ton, though... like a few hundred MB a day... not slashdot sized :-) to spare. If people would like a resource like this, just email me at sujal@worldnet.att.net and/or sujal@sujal.net with suggestions, comments, etc.

    If someone else is willing to do this, or has already started, please email me so I can help them rather than doubling up effort. Thanks

    Sujal

    --

    politics, food, music, life: FatMixx

  8. Tuning Linux, really? by heller · · Score: 2

    So, as the article mentioned, there's not a lot of info on tuning linux, but it's easy to find for NT. Why? you ask? I did.

    The only answer I came up with is that Linux actually requires little tuning. Probably 95% of the Linux boxes out there need virtually no tuning. It runs so well out of the box that tuning it is hardly necassary.

    Microsoft, on the other hand, has basically acknowledged that out of the box, NT runs like crap. They've provided lots of information on how to make it run better. I could only see them doing this if people are unhappy with its performance.

    I know I'm more than happy with the performance of my untuned Linux boxes and I'm not sure that spending lots of time tuning them would really make a big difference.

    ** Martin

  9. Wow zd-net doing some anti-fud for once. by law · · Score: 2

    It's nice to see ZD doing some anti-fud, I wonder if there will be more. This whole thing smacks of the anti-trust trial video.
    I heard a interesting thing, about technology writers having a more jaundiced eye towards Microsoft after some of the trial shenanigans. It maybe that this is a sign, that technology as a industry, is groing up a bit.

    --
    "Think of it as evolution in action."
  10. Requests for tuning support by Hal+Roberts · · Score: 2

    Some of the items that have been bandied about is stuff that is obscure and difficult to figure out. For example, I wouldn't expect even a fairly skilled Linux user to know that the 2.2.2 kernel had problems with TCP and windows clients.

    However, anybody even vaguely familiar with the linux versioning system should know always to get the lastest version of the stable kernel. No one getting paid to administer a Linux machine should look at a list of kernels 2.2.1 - 2.2.5 and decide to download 2.2.2 !

    Moreover, some of the mistakes they made (compiling samba with -O instead of -O2, setting the number of servers to 10 initially and 1 spare) show a lack of even basic Unix and apache administration skills. While it may not be fair to expect them to nail every possible optimization, there's no excuse for neglecting basic stuff like getting the latest stable kernel and telling apache to start enough servers. Even if this sort of information does require a fair amount of Linux knowledge, who in their right mind would pay over $25,000 for a machine and not even hire an admin with a basic knowledge of its operation ?

    Even if they didn't hire a competent Linux admin, I'd pit Linux's performance under a newbie admin against NT's performance under a newbie admin. The ZDNet article claims that performance tuning information for apache is much more difficult to find than that for IIS. But, if you go to the apache web site and click on the big link named 'Server Documentation', you'll find links to both general and platform specific performance tips. The section on general performance tips includes *gasp* a couple of paragraphs on the importance of setting the StartServers and MinSpareServers directives appropriately. In contrast, the Microsoft site requires you to navigate through a maze of links to try to find information about IIS. I frankly gave up trying to count the number of links until I could get server performance tips for IIS after ten minutes of wading through MS's marketing classifications (am I an enterprise customer or an IT professional ? which support option do I need ? Why don't many of the forms work in Netscape? etc. etc.). The reason that the ZDNet author believes that the IIS information is easy to find is because he's used to using that information, just like the authors of the study found NT easy to tune b/c they were used to tuning it.

    However, if such a newbie-Linux-vs.-newbie-NT-on-an-enterprise-class- machine-for-loads-greater-than-those-han dled-by-Yahoo study were done, it would be quickly dismissed as patently ridiculous. Mindcraft was smarter than that, so they actually pretended to tune the Linux box, but that doesn't change the fact that they were testing the performance of of an enterprise class system and ridiculously high loads under a clueless admin.

    Lastly, I challenge you to call up the Microsoft Tech Support line, tell them you're running a perfomance benchmark on NT and would like some tuning help. I seriously doubt you'd get anything even resembling a helpful response !

  11. The cat has lost it's tail. by sjvn · · Score: 2

    >>Nobody in their right mind would install as a server a Pentium II 266, 64 Meg RAM with a 4 gig IDE drive. This is the hardware that ZD used in their test.
    When Microsoft stops claiming that that's more than enough machine for a NT 4 server, we'll quit reviewing NT on similar platforms.

    Another reason for our choice of platforms is that while we all lust in our heart for quad-pentiums with a gig of memory to call our own, the simple truth is most of us, and most businesses, can't afford them. Yes, CPUs and memory are cheaper than ever, but businesses don't replace their legacy equipment all that often. Our studies show us that a plurality of local LAN business servers out there are less well equipped than our test machines. So it is that one reason we test on low-end systems is that that's what many of our readers still have to deal with it.

    Again, though, if MS just said you need at least X and Y to run NT, we'd use it. It wouldn't be fair to do otherwise. But, they don't, the users don't, so we don't.

    At least with 2000, they're being 'somewhat' more realistic--128MBs and 300MHz minimums. I quote 'somewhat' because I suspect that those minimums will prove to be bare-bone minimums.

    Steven, Senior Technology Editor, Sm@rt Reseller

  12. It doesn't matter who pays... by Weasel+Boy · · Score: 2

    ...what matters is whether it's good science or bad science. Someone's gotta pay for it. While it is quite likely that MS commissioned Mindcraft to perform bad science (they have shown this tendancy in the past), it was certainly within their power instead to commission a fair study.

    I'm all for applying critical thinking and considering the motivation of the speaker. I'm against automatically assuming a claim is false just because someone we dislike made it. I've seen plenty of usenet flamewars in which entirely true study findings were discounted solely on the basis of who paid for them. That's just closed-minded.

    On the other hand, specific point-by-point rebuttals are right-on. Producing evidence that refutes the claim based on better science in golden.

    I guess it's just easier to be a mindless naysayer and pretend to be a sophisticated skeptic than it is to approach disagreeable ideas with an open mind and use logic and rationality to pick them apart, and risk having to admit being wrong.

  13. Finally, a decent article from ZDNet by Mr.+Piccolo · · Score: 2

    The only thing they didn't point out is that Microsoft paid for the whole thing. Everything else they pretty much nailed.

    So, all you gurus out there, it seems there's a definite need for a "Highend-Server-Tuning-HOWTO". Any takers?

    Also, I'd love to see the same test run on NT, a properly tuned Linux, Solaris 7 (which is supposed to be Much improved over 2.6 on x86 hardware), and Netware to see how many OSes beat NT, when Mindcraft claims the other way around. Unfortunately, I don't have the money for a quad-Xeon box or Netware, or I'd make my website useful for once ;-)

    --
    Glückwünsche, haben Sie Slashdot ermordet, indem Sie zum korporativen Druck beugten und Subskriptionen einlei
  14. "Linux support lacking" by dbullock · · Score: 2

    My take: We say usenet support works better. Someone tested it. We failed. We need to address that. Usenet sucks anyway - article availability is non-deterministic, and it's relatively slow.

    That said -- I do think that Mindcraft's attempt to gain support was a token attempt only.

    If we want to keep the reputation as being a better source of support (via usenet) than Microsoft, then our community is at fault for failing to answer or to ask for clarification.

    It does not matter that the invididual was a private individual. It does not matter if it was for a private box, commercial box, or for a publication. From what I heard no answer or request for clarification was ever made via usenet. If we're only going to help article writers then we're a lousy source of support.

    Don't justify why we failed, accept the hit and produce a solution. I've thought of the knowledgebase solution but I never thought to mention it and see if anyone else is interested. My bad. I would like to see a knowledgebase implemented and I'd like to contribute to it.

    --
    http://www.bullnet.com
  15. Amazing pro-Linux from ZD! by Kymermosst · · Score: 2

    I must say that was a very well written article. It pointed out the M$ sponsoring of the "benchmark," which is a key thing to note about the testing: M$ paid for it all.

    The fact that they showed how it is possible to find the information that Mindcraft claimed it was not able to get proves that Mindcraft really didn't try all that hard to do the optimizations.

    I'm glad that ZD is exposing M$ and it's lackey companies like Mindcraft for who they really are!

    --
    "Alcohol, Tobacco, Firearms, and Explosives" should be a convenience store, not a government agency.
  16. The Story has a point by Rocket+Boy · · Score: 2

    I do agree whole heartedly that Linux could, properly tuned, beat the living windows out of NT. But, the whole problem is tuning it. I am an admitted Linux newbie and I am having a hard time finding information to tweak and polish my install. But, I am finding the information slowly but surely. At least I am trying to find it. It looks like Mindcraft was in a room by itself and asked if anyone knew how to tune Linux. I think RedHat dropped the ball on this one.

    RB