NOS Crossroads

Mark Wright sent us a link to some benchmarks over at ZD Net that examines assorted NOS Options. NT is benched, as is Solaris, Netware and Linux. Linux holds up quite poorly in this review.

Comments

Static Pages on an Intranet?

[Zachary Kessin]

I develop intranet apps for a living and I wonder why they keep testing "Enterprise" webservers only using static pages. Most of the load on any webserver is going to be on generation of dynamic content. I don't Care if you are using CGI, Java serverlets, mod_perl or whatever.

In most real applications static files will clog the network pipe before it hits the CPU. And as been noted there are some unix webservers that can serve static pages much faster than apache.

But we do need to document all of this better.

Not very 'complete'

[ChrisRijk]

Important things they missed out: Stability/reliabilty, security, availability, interoprability, didn't covert scalability properly...

There were some other things I thought were kinda strange...I'll concentrate on Solaris here.

For Solaris they actually used Solaris on Intel, which is fair enough considering they were looking at doing stuff on the same hardware, but isn't that good for 'real world' situations (A comparison with a Sun E450 would have been interesting) because most people who use Solaris use it on Sun hardware. Some things are a bit unclear - they seem to say they got the Solaris box from Sun, even though Sun don't sell Intel based boxes themselves - they get OEMs to do that. (actually, they correct that later, saying that Sun brought in a Dell PowerEdge box) They don't say when they got the box, but they did mention Sun's Project Cascade (think Samba for Solaris) but didn't mention that products for this are now available (well, availability was annouced a few weeks back, though I don't know about x86 versions).

They gave Solaris (on Intel) a D on RAID due to lack of support for PCI cards (not sure how fair that is) which is kinda funny when Solaris on SPARC has about the best and most reliable RAID setup out there, according to people I've talked to.(NetApps were also highly praised btw) They then criticize Sun for being 'expensive' (the hardware is, sure), when they were not even testing Sun hardware, while Solaris itself is actually very cheap for a commercial OS. (NT is only cheaper than Solaris when your NT box has no clients) They then have contradictory stuff about Solaris - stuck in the datacenter on some pages (the main ones), while on other pages (the Solaris specific ones) they give a different picture...

Btw, in the final page about Solaris they mention a report from the Standish group, but they don't give a URL to it. It's available here - Solaris Vs NT.

The article brought up some interesting points

[Kiwi]

Articles like this, which show some potential weaknesses with Linux, are excellent guides for the developers to continue refining the already excellent OS that Linux is.

It pointed out that:

• Linux + Apache maxes out when it gets 2000 HTTP requests a second
• Linux + Apache maxes out when it is transferring 200 Mbps a second

These kind of benchmarks, although unpleasant to read, have worked to improve Linux in the past. The fact that Apache no longer attempts to perform a slow getaddrbyname (reverse DNS lookup) operation every time someone requests a web page is the result of benchmarks showing NT web servers beating the socks off of Linux web servers that did this inefficient operation.

The web page tunelinux.com is the result of the much-discussed Mindcraft study.

Linus fixed a problem with Linux yielding threads when it was shown by an informal benchmark that NT was much faster when yielding threads in a tight loop. Of course, this being a Usenet test, a long flame war started arguing whether the test was legitimate. Linus had the very mature comment that "Anything that could objectively make Linux look bad should be fixed" (or words to that effect).

My only objection to these ZDNET studies is that they do not always explain in sufficient detail their testing methodology. As long as their story explains their testing methodology, these articles should be studied by the developers with a fine tooth comb.

- Sam

benchmarks

[noom]

Why is it that EVERY time a benchmark comes out which claims that Linux doesn't perform as well as other OSes, people claim that it was because of biased testing? Perhaps people had justification for bashing Mindcraft's tests, but this evaluation seems to have been done very well.

Even Linus says that, thus far, Linux had been developed with stability and maintainance in mind, not necessarily raw performance. Also, for the most part, linux developers haven't had the resources to spend on enterprise class servers for use in performance testing and coding. This is probably why Linux always seems to be the best performer on relatively inexpensive machines -- it has been developed and tuned almost exclusively on them.

I think that most people agree that Linux has a long way to go before it will be the best (performance-wise). The fact that is is GPLed will certainly help, but we need people (companies) with the resources to spend on developing Linux with a goal of performance. It will probably take some time before linux coders stop playing catch-up (i.e. trying to support all the devices and functionality of other operating systems) and start working hard on optimizations.

Frankly, I'm not even sure that a "bazaar" model of development can support this goal. In many cases, when you are writing code (esp. systems code) with a goal of squeezing the best possibly performance out of it, some of the most effective optimizations are nearly incomprehensible to people who haven't spent months examining all of the subtle interactions which make the optimization so effective. Since I doubt that Linus want's a kernel filled with magic that only a few wizards understand, such optimizations may never make it into the kernel (unless the kernel forks). These are the things which turn into debugging nightmares later on. I'll bet that both the reason for the speed of NT compared to Linux as well as its notorious instability are because of this.

Incidentally, no flames please. I've been running Linux exclusively on my machine for a couple years now; that means none of that "well, I still boot Windows occasionally to run games" crap either. I just think that we should takes examine these published benchmarks for valid points and see what we can to do improve our scores. This doesn't necessarily mean benchmark specific tuning (which is what most companies do) either. Its only that just screaming "FUD!!!" doesn't accomplish anything. Hopefully, in a couple years, Linux will be so ripped that it will be difficult for someone to de-tune Linux to make other OSes appear better.

-nooM

Once again 4 intel ethernet cards

[gsaraber]

And again they are using 4 intel ethernet adapters, probably configured to do some sort of striping/loadsharing/whatever .. i think that's what gives NT it's edge in every benchmark..
I think there's going to be a lot more like this .. watch for the 4 ethernets :(

It shouldn't be to hard to implement into linux, bind 1 ip to 4 ethernets and send through whichever one is free ..