madddog on Linux v NT Benchmarking

BogoMips sent us an interesting tidbit running in Performance Computing currently. Jon "maddog" Hall explains some of the benchmarking issues associates with the DH Brown reports, as well as the ubiquitous Mindcraft tests. Very well written article, IMHO.

Comments

The problem with linux

[Erich]

The problem with linux and large enterprise servers is mostly in it's style of development. Most people don't have access to an Ultra Enterprise 10000 ``starfire'' and several disk arrays to just sit around and play with -- they are stuck with (relatively) cheap PC's and Suns and Alphas. Only large companies can afford to shell out millions to buy the equipment, and millions more to pay programmers to develop for it. That's why Sun Enterprise equipment is almost mission-critical-environment ready -- and linux isn't. You can't yank a processor board out of a machine running Linux and still have the thing hum along. Yet.

Not that hard...

[BadlandZ]

If you use FreeBSD, you simply uncomment one configuration line, add 2-3 flags, just one time, at the very beginning.

After that, to install, all you need to know is three words, "make install distclean", and even that third word is optional.

It does bring up the point that it should and could be easier. What is really needed is an expantion on the basic "uname" call, to include much more system information (specifically, the CPU, amount of memory, etc..) Then, it would be possable for open souce compilers to just make it one generic flag (like SGI's compiler does, -Ofast) and it would grab the system information for you and figure out the best flags on it's own. That is a realistic possability, and if something like POSIX, the LSB, or some other standardization body would implement this type of system call to get hardware information, it could potentially benifit the UNIX community in a way that Microsoft can't keep up with.

I am still Shocked

[BadlandZ]

I am still bothered by the fact that all these benchmarking tests and comments don't clearly address one of the basic advantages to open source. You can optimize the binaries for your specific system, and get rid of the 386/486 limitations... Sure, you loose binary compatiblity, but who cares when they can get thier own source and compile it too, and your system sees a 30% boost in speed?

System tuneing is IMPORTANT. Important enought that it can make one OS faster than another. I think we should be pointing out that open source is much more tunable, not only in the ability to modify the code, but also in the ability to optimize it for specific hardware. (quick note on some stuff I tried to see the diffrence, click here)

Why can't someone do some intellegen testing on this, and give credit to the people who REALLY make GNU/GPL and all of open source a success, the folks who write the COMPILERS! Linus did write some nice stuff, as well as many others, but without the right compiler, it's worthless.

How essential are enterprise servers?

[jnik]

Linux right now doesn't scale horribly well to huge enterprise servers. But it still manages to replace them fairly effectively, by allowing scalability on the level of the machine instead of components.

If I can do the work of a big enterprise server on four P75's, that's still a savings. Linux is proving that for some tasks, a cluster of small machines works just as well as one huge machine. And there's reliability projects in the works as well based on the same principle: have an extra cheap machine which grabs the load if your main server goes down.

Re:Linux Ain't There Yet

[Locutus]

There was a reason to cry 'unfair' at the first benchmarks from MindCraft. The tests were plan wrong and very little could be gleaned from them. The third test ( second test was private and the results weren't published ) provided information which allows corrections to be made.

I think you'll someday find that Microsoft is feeding the press with data to show Linux's weaknesses. That is how they 'compete'. Unfortunate for them, they aren't 'competing' with IBM, its Linux. Weaknesses will be patched quickly and tested by the community too quickly for Microsoft. They won't be able to wait for a liquid cooled CPU to become the norm so NT v5/2000 can beat Linux in future tests.

Fair benchmarking and reporting is not the norm in this industry. Recently IBMs Warp Server for e-business was hammered on InfoWorld (the link is now broken to the article....). It turns out the guy who wrote the column is the Senior Contributing Editor and Columnist of Windows NT Systems magazine. See "Hatchet Job"

Complaining/exposing a injustice is how we open the eyes of the unknowing. Example: At my stock investment club meeting last night, one member insisted that NT was faster then Linux in all cases and that Linux had a weak GUI. I booted OpenLinux v2.2 on my P120 laptop an he started questioning his beliefs. The rest of the goup was surprised at the polish they saw. All but one member are professionals in the technology industry though mostly embedded/realtime systems.

Linux Ain't There Yet

[Simeon2000]

This is kinda ridiculous. Why is it that corporations are expecting the free(Linux) to be as "enterprise server ready" as the expensive(NT)?

Linux, as far as I know, wasn't designed with that in mind. It was originally Linus' hobby, for crying out loud. Also, it is a very young OS. It is right now good at what its intended use is... squeezing good use out of lower-end, and sometimes antiquated PC's.

I mean, try to make a web/mail server for your corporation with Windows NT on a leftover 486/16MB. Can't do it. But I did that with Linux, and it runs fairly fast for its hardware handicap.

But it has not yet been coded to do what NT is aimed at... taking over the high end server market. Linux developers (whom are gentlemen that deserve our utmost respect) are now beginning to seriously address Linux' lack in this area. I imagine that since Linux is already king in low-end servers, now it will turn its attention towards the higher-end market. Thus mindcraft and company have helped shape the development of Linux in a good way... instead of us madly shouting "unfair benchmarks!", we need to simply begin working towards what NT has already claimed... higher speed on the big iron. Hopefully we can do so without the bloat and instability that has plagued NT, though.

----- if ($anyone_cares) {print "Just Another Perl Newbie"}

Told you so.

[RISCy Business]

You know, I liked this article. It echoed what I've been saying for ages.

Linux isn't for mission critical large-scale. NT *DEFINITELY* isn't. If you want that, you'd better call your IBM RS/6000 VAR today, because sales are going to jump now.

Linux is not the be all and end all of unix. Period. It never ever will be, as it will more than likely collapse upon itself before we see ext3.

I use Linux. I've used Linux for about 4 or 5 years. I think Linux is great.

But it's not an enterprise OS. Period. Flat out. Never. It's good for small to midrange stuff, sure. Hell, our primary DNS server is Linux, as is our webserver currently. (It *will* be moved to an RS/6000 H70 as traffic increases.) Our secondary DNS server will be Linux. But our network monitoring system will be an RS/6000 43P-140 (aka Model 140) workstation running AIX. Why? Because if hardware starts failing in Linux, I'm screwed. AIX will scream bloody murder before it gets anywhere *near* the point of no return.

Linux has no place in the enterprise as an ERP server. It has no place as your 1,000,000 hit/hour webserver interfacing with SQL and doing dynamic pages. Period. Those of you who find this offensive, kindly contact a proctologist so that you may have your head removed from where it is. That's the way it is. Don't like that? Go work towards changing it. Change is good. But till things change, what I say will hold true.

-RISCy Business | Rabid System Administrator and BOFH

Benchmarks

[oldzoot]

I, for one would like to see a set of benchmark results for Linux that would help a person to make decisions about the configuration of hardware platforms for systems. I would like to see a set of test results for different types of system activity, such as compiling code, raytracing/graphics/visualization, file system access, network bandwidth, combined network/filesystem access etc. This set of measurements could then be run on a variety of hardware types providing the basis of cost/performance decisions in the implementation of systems. One could answer questions like how does a K6/233 compare to a P5/233 ? How does a P5/233 compare to a P6/450 ? How much difference is there between an ULTRA ATA disk system and SCSI? Does the difference change with different processor speeds ? Does 1MB cache make a difference with what I want to do ?
If implementing a cluster, does saving $2K per box make up for the difference ( providing money for more boxes ) if you use ATA and slightly slower CPUs, rather than higher end platforms ?

I would love to see a single ( and evolving ) location for this kind of info. Hell, I would love to work on compiling it. I have seen some sites with benchmark info, but nothing that seems to try to answer specific questions. The sites that I have seen present specific objective numbers but it was hard to derive any context for the differences between systems.

I thought this was a good piece.

[AssKey]

I liked the fact that instead of reading an
article about how messed up and skewed the
benchmark was, I read an article that suggested
things that can be done to better arm ourselves
the next time something like this pops up.

Good Job.