madddog on Linux v NT Benchmarking

BogoMips sent us an interesting tidbit running in Performance Computing currently. Jon "maddog" Hall explains some of the benchmarking issues associates with the DH Brown reports, as well as the ubiquitous Mindcraft tests. Very well written article, IMHO.

Comments

The problem with linux

[Erich]

The problem with linux and large enterprise servers is mostly in it's style of development. Most people don't have access to an Ultra Enterprise 10000 ``starfire'' and several disk arrays to just sit around and play with -- they are stuck with (relatively) cheap PC's and Suns and Alphas. Only large companies can afford to shell out millions to buy the equipment, and millions more to pay programmers to develop for it. That's why Sun Enterprise equipment is almost mission-critical-environment ready -- and linux isn't. You can't yank a processor board out of a machine running Linux and still have the thing hum along. Yet.

I am still Shocked

[BadlandZ]

I am still bothered by the fact that all these benchmarking tests and comments don't clearly address one of the basic advantages to open source. You can optimize the binaries for your specific system, and get rid of the 386/486 limitations... Sure, you loose binary compatiblity, but who cares when they can get thier own source and compile it too, and your system sees a 30% boost in speed?

System tuneing is IMPORTANT. Important enought that it can make one OS faster than another. I think we should be pointing out that open source is much more tunable, not only in the ability to modify the code, but also in the ability to optimize it for specific hardware. (quick note on some stuff I tried to see the diffrence, click here)

Why can't someone do some intellegen testing on this, and give credit to the people who REALLY make GNU/GPL and all of open source a success, the folks who write the COMPILERS! Linus did write some nice stuff, as well as many others, but without the right compiler, it's worthless.

Re:Linux Ain't There Yet

[Locutus]

There was a reason to cry 'unfair' at the first benchmarks from MindCraft. The tests were plan wrong and very little could be gleaned from them. The third test ( second test was private and the results weren't published ) provided information which allows corrections to be made.

I think you'll someday find that Microsoft is feeding the press with data to show Linux's weaknesses. That is how they 'compete'. Unfortunate for them, they aren't 'competing' with IBM, its Linux. Weaknesses will be patched quickly and tested by the community too quickly for Microsoft. They won't be able to wait for a liquid cooled CPU to become the norm so NT v5/2000 can beat Linux in future tests.

Fair benchmarking and reporting is not the norm in this industry. Recently IBMs Warp Server for e-business was hammered on InfoWorld (the link is now broken to the article....). It turns out the guy who wrote the column is the Senior Contributing Editor and Columnist of Windows NT Systems magazine. See "Hatchet Job"

Complaining/exposing a injustice is how we open the eyes of the unknowing. Example: At my stock investment club meeting last night, one member insisted that NT was faster then Linux in all cases and that Linux had a weak GUI. I booted OpenLinux v2.2 on my P120 laptop an he started questioning his beliefs. The rest of the goup was surprised at the polish they saw. All but one member are professionals in the technology industry though mostly embedded/realtime systems.

Told you so.

[RISCy Business]

You know, I liked this article. It echoed what I've been saying for ages.

Linux isn't for mission critical large-scale. NT *DEFINITELY* isn't. If you want that, you'd better call your IBM RS/6000 VAR today, because sales are going to jump now.

Linux is not the be all and end all of unix. Period. It never ever will be, as it will more than likely collapse upon itself before we see ext3.

I use Linux. I've used Linux for about 4 or 5 years. I think Linux is great.

But it's not an enterprise OS. Period. Flat out. Never. It's good for small to midrange stuff, sure. Hell, our primary DNS server is Linux, as is our webserver currently. (It *will* be moved to an RS/6000 H70 as traffic increases.) Our secondary DNS server will be Linux. But our network monitoring system will be an RS/6000 43P-140 (aka Model 140) workstation running AIX. Why? Because if hardware starts failing in Linux, I'm screwed. AIX will scream bloody murder before it gets anywhere *near* the point of no return.

Linux has no place in the enterprise as an ERP server. It has no place as your 1,000,000 hit/hour webserver interfacing with SQL and doing dynamic pages. Period. Those of you who find this offensive, kindly contact a proctologist so that you may have your head removed from where it is. That's the way it is. Don't like that? Go work towards changing it. Change is good. But till things change, what I say will hold true.

-RISCy Business | Rabid System Administrator and BOFH

Benchmarks

[oldzoot]

I, for one would like to see a set of benchmark results for Linux that would help a person to make decisions about the configuration of hardware platforms for systems. I would like to see a set of test results for different types of system activity, such as compiling code, raytracing/graphics/visualization, file system access, network bandwidth, combined network/filesystem access etc. This set of measurements could then be run on a variety of hardware types providing the basis of cost/performance decisions in the implementation of systems. One could answer questions like how does a K6/233 compare to a P5/233 ? How does a P5/233 compare to a P6/450 ? How much difference is there between an ULTRA ATA disk system and SCSI? Does the difference change with different processor speeds ? Does 1MB cache make a difference with what I want to do ?
If implementing a cluster, does saving $2K per box make up for the difference ( providing money for more boxes ) if you use ATA and slightly slower CPUs, rather than higher end platforms ?

I would love to see a single ( and evolving ) location for this kind of info. Hell, I would love to work on compiling it. I have seen some sites with benchmark info, but nothing that seems to try to answer specific questions. The sites that I have seen present specific objective numbers but it was hard to derive any context for the differences between systems.