Slashdot Mirror
NOS Crossroads
Mark Wright sent us a link to some benchmarks over at ZD Net that
examines assorted
NOS Options.
NT is benched, as is Solaris, Netware and Linux. Linux holds
up quite poorly in this review.
← Back to Stories (view on slashdot.org)
Gee, so a Windows NT fileserver can save a few milliseconds over a Linux fileserver? Oh, and the quad-Pentium NT webserver that I can't afford can save a few milliseconds serving up static web page content?
Well, I DON'T CARE, because I don't have to drive to the office at 3:00 am to reboot the Linux server. That's worth a lot more to me than those milliseconds.
While we're doing these benchmarks, let's quit serving up static web pages and start serving up some CGI-generated content. Watch what happens to NT then, folks.
Yeah, I'm ranting and I'm hiding behind AC, but I'm also speaking from experience.
Linux zealots aren't born. NT MAKES THEM!
^^ Feel free to use the above as a sig. ^^
If you follow the link to the PC Week lab notes, you see that Linux outperformed NT when using NT Workstation clients on the SMB tests. So, the real results aren't so bad. As far the NetBench stuff, that is more a test of Apache vs. other Web servers rather than Linux vs. other NOS's. They need to put squid or some other cache in front of Apache; or maybe use Zeus. And really, if you are going to test Enterprise readiness, reliablility and predictability are the most valued attributes of an Enterprise OS. IMHO, reliability should account for at least 65% of any scorecard.
Loading a new process in NT is slow and memory-intensive. Microsoft's own tech notes admit this (do a search on "CGI" in MS TechNet). Linux is faster at running a new process.
This, BTW, is one of the reasons Microsoft pushes Active Server Pages and ISAPI. The user code runs in the same process space as IIS (unless you use MTS) and doesn't have to be loaded each time it's called.
Predictions --
1. A comparison of CGI-generated content could well show NT IIS getting spanked in terms of pure speed.
2. Microsoft would challenge the results, saying that the benchmarkers should have been using ASP or ISAPI. They would probably throw in snide remarks about CGI being "old technology".
Your point is very important. With more sites becoming interactive (esp. the "enterprise" sites which these benchmarks target), static page delivery should be met with a big yawn.
Posted by Jeremy Allison - Samba Team:
0 ,401974,00.html
:-). I can beat NT comfortably
> Now, as both Solaris and Linux had nearly
> identical graphs for the NetBench part, and
> both were using Samba, I think we know where
> the bottleneck there is...
Err, actually no. The bottleneck isn't Samba.
If you look carefully at the Solaris analysis at page :
http://www.zdnet.com/pcweek/stories/jumps/0,427
You'll find this interesting quote :
"To isolate the disk subsystem as a bottleneck, we created a
temporary RAM disk to hold workload files, effectively
eliminating the need to hit the RAID array for data. In this
configuration, the powerful capabilities of Solaris 7's networking
kernel were unleashed--to the tune of 360M bps on NetBench."
What this means is that when Samba is run on a very tuned SMP
OS such as Solaris (ignoring the disk subsystem for the moment)
then Samba can produce numbers that out perform *all* the other
systems (the peak NT number is around 340 I think). What is killing
Solaris here is their awful disk system. If they had a decent disk
file system they would have had beaten NT when using Samba to
serve Win95 clients as their SMP is so good.
This corresponds well with the results I get in the SGI labs
using IRIX, which is also a highly tuned SMP OS (but with a
better disk file system, XFS
using Samba and IRIX on an SMP box, but IRIX only runs on MIPS
boxes from SGI.
What this means for Linux is that we need to do more work
on the SMP scaling in the Linux kernel, as Samba isn't the
bottleneck here. I'm doing a lot of work on userland caching
at the moment to help out on the Samba side, but Linux just
needs a bit more SMP work. Don't worry, it's coming (I know
*lots* of people working on this)........
Regards,
Jeremy Allison,
Samba Team.
I develop intranet apps for a living and I wonder why they keep testing "Enterprise" webservers only using static pages. Most of the load on any webserver is going to be on generation of dynamic content. I don't Care if you are using CGI, Java serverlets, mod_perl or whatever.
In most real applications static files will clog the network pipe before it hits the CPU. And as been noted there are some unix webservers that can serve static pages much faster than apache.
But we do need to document all of this better.
Erlang Developer and podcaster
There were some other things I thought were kinda strange...I'll concentrate on Solaris here.
For Solaris they actually used Solaris on Intel, which is fair enough considering they were looking at doing stuff on the same hardware, but isn't that good for 'real world' situations (A comparison with a Sun E450 would have been interesting) because most people who use Solaris use it on Sun hardware. Some things are a bit unclear - they seem to say they got the Solaris box from Sun, even though Sun don't sell Intel based boxes themselves - they get OEMs to do that. (actually, they correct that later, saying that Sun brought in a Dell PowerEdge box) They don't say when they got the box, but they did mention Sun's Project Cascade (think Samba for Solaris) but didn't mention that products for this are now available (well, availability was annouced a few weeks back, though I don't know about x86 versions).
They gave Solaris (on Intel) a D on RAID due to lack of support for PCI cards (not sure how fair that is) which is kinda funny when Solaris on SPARC has about the best and most reliable RAID setup out there, according to people I've talked to.(NetApps were also highly praised btw) They then criticize Sun for being 'expensive' (the hardware is, sure), when they were not even testing Sun hardware, while Solaris itself is actually very cheap for a commercial OS. (NT is only cheaper than Solaris when your NT box has no clients) They then have contradictory stuff about Solaris - stuck in the datacenter on some pages (the main ones), while on other pages (the Solaris specific ones) they give a different picture...
Btw, in the final page about Solaris they mention a report from the Standish group, but they don't give a URL to it. It's available here - Solaris Vs NT.
First, Beowulf clusters do not provide high availability to standard data networks. They cannot be used to imporive Linux performance in this application.
Second, they probably chose multi-processor systems to run the benchmark because multi-processor systems are typically used by IT shops in this role.
It is no use complaining that they should not use a particular platform configuration just because Linux does not run well on it. Linux must instead be improved so that it can work well on the platform of choice.
Articles like this, which show some potential weaknesses with Linux, are excellent guides for the developers to continue refining the already excellent OS that Linux is.
It pointed out that:
- Linux + Apache maxes out when it gets 2000 HTTP requests a second
- Linux + Apache maxes out when it is transferring 200 Mbps a second
These kind of benchmarks, although unpleasant to read, have worked to improve Linux in the past. The fact that Apache no longer attempts to perform a slow getaddrbyname (reverse DNS lookup) operation every time someone requests a web page is the result of benchmarks showing NT web servers beating the socks off of Linux web servers that did this inefficient operation.The web page tunelinux.com is the result of the much-discussed Mindcraft study.
Linus fixed a problem with Linux yielding threads when it was shown by an informal benchmark that NT was much faster when yielding threads in a tight loop. Of course, this being a Usenet test, a long flame war started arguing whether the test was legitimate. Linus had the very mature comment that "Anything that could objectively make Linux look bad should be fixed" (or words to that effect).
My only objection to these ZDNET studies is that they do not always explain in sufficient detail their testing methodology. As long as their story explains their testing methodology, these articles should be studied by the developers with a fine tooth comb.
- Sam
The secret to enjoying Slashdot is to realize that it should not be taken too seriously.
Who has the guts to organize and/or sponsor such an event? Magazines would be welcome.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
I wouldn't say that this was a bad review, especially considering that ZD would have dismissed Linux out of hand scarcely a year ago. Linux is harder to configure than NT, tuning information is a lot harder to find, finding a patch to match your kernel revision is an unholy pain in the butt, and you do need more competent staff to administer it than you do with NT's point-and-drool interface. This is not news. To their credit, they did say -- essentially -- that the higher learning curve associated with Linux is repaid by greater power and flexibility. Considering what an big corporate lackey ZD is, that's no small admission.
Proud member of the Weirdo-American community.
For example, the ability to serve lots of hits per second on static web pages from a single box has no relevance to real-life web sites. At 1000 hits per second, a single Linux machine can serve about as many hits per second in these benchmarks as the whole Microsoft web site receives. That seems more than enough, and it's clearly not where real web sites are hitting their limitations (Microsoft uses dozens of their machines for their web site). I think the reason why Microsoft like this kind of benchmark is because it's easy to tune the OS for, even if it has little impact on actual web operations.
Also, the importance of SMP is overrated: the need for SMP on NT and some other systems arises often simply from licensing and system management issues; in many server applications, separate machines are preferable.
The benchmarks also don't take into account cost/performance. ZD claims "NT excels in NetBench". But actually, it only does 50% better for a price of at least $800 more. For that amount of money, you can buy another Linux machine and double Linux performance.
Most importantly, however, I think it is wrong to consider Linux, Solaris, and other UNIX systems to be "competitors". People can (and do) run mixed UNIX environments. For example, I might use Linux for all the web servers and an AIX machine for running a DB2 enterprise database that backs it all. Using Linux means there are lots of directions to grow in and lots of compatible commercial vendors to choose from.
If I develop for NT, I'm stuck when NT runs out of steam on its measly 4 or 8 processor Intel boxes, or when it runs out of its 3G address space. With NT, there is nothing to upgrade to.
Linux clearly isn't for everybody or everything. Only Microsoft seems to have the hubris of thinking that a single OS (theirs) can work well for everybody. Linux is part of a family of operating systems from different vendors that are interoperable and mostly compatible, and that only as a group cover most needs from embedded systems to mainframes. But within its own niche, R/D desktop applications, server farms, and small to medium servers, Linux is actually quite good.
Why is it that EVERY time a benchmark comes out which claims that Linux doesn't perform as well as other OSes, people claim that it was because of biased testing? Perhaps people had justification for bashing Mindcraft's tests, but this evaluation seems to have been done very well.
Even Linus says that, thus far, Linux had been developed with stability and maintainance in mind, not necessarily raw performance. Also, for the most part, linux developers haven't had the resources to spend on enterprise class servers for use in performance testing and coding. This is probably why Linux always seems to be the best performer on relatively inexpensive machines -- it has been developed and tuned almost exclusively on them.
I think that most people agree that Linux has a long way to go before it will be the best (performance-wise). The fact that is is GPLed will certainly help, but we need people (companies) with the resources to spend on developing Linux with a goal of performance. It will probably take some time before linux coders stop playing catch-up (i.e. trying to support all the devices and functionality of other operating systems) and start working hard on optimizations.
Frankly, I'm not even sure that a "bazaar" model of development can support this goal. In many cases, when you are writing code (esp. systems code) with a goal of squeezing the best possibly performance out of it, some of the most effective optimizations are nearly incomprehensible to people who haven't spent months examining all of the subtle interactions which make the optimization so effective. Since I doubt that Linus want's a kernel filled with magic that only a few wizards understand, such optimizations may never make it into the kernel (unless the kernel forks). These are the things which turn into debugging nightmares later on. I'll bet that both the reason for the speed of NT compared to Linux as well as its notorious instability are because of this.
Incidentally, no flames please. I've been running Linux exclusively on my machine for a couple years now; that means none of that "well, I still boot Windows occasionally to run games" crap either. I just think that we should takes examine these published benchmarks for valid points and see what we can to do improve our scores. This doesn't necessarily mean benchmark specific tuning (which is what most companies do) either. Its only that just screaming "FUD!!!" doesn't accomplish anything. Hopefully, in a couple years, Linux will be so ripped that it will be difficult for someone to de-tune Linux to make other OSes appear better.
-nooM
what did you expect? that linux would win?
remember the phrase first they ignore you, then they laugh at you. then they fight you. then you win.
well... this is the fight part. and it is war...
dont expect it to end anytime soon.
Ex Libris Veritas
> If Novell supplied NetWare with a real SMP
;-)
> kernel, NetWare's performance would be
> show-stopping.
Netware 4.x and older really had a problem with SMP, especially if Maximum Service Processes was set too low, but Netware 5.x is a different kettle of fish. It's SMP is very damn good.
> Unfortunately, in its current state, NetWare
> leaves a lot to be desired not only in
> scalability but also in application support.
Netware 5.x doesn't have many applications ported to it, unless you count such small things as Oracle and Notes.
> Couple this with Novell's decision to divorce
> great applications such as ZENworks and Novell
> Directory Services from NetWare, and the value
> proposition for NetWare becomes even murkier
I'm sorry, NDS isn't part of Netware 5.x ? or even 4.x?????? Did these people even install this product? ZEN is bundled with Netware 5.x [admitedly without the Helpdesk or Remote Control functionality] as well. And does a damn fine job. Heck, it's even bundled in the latest Win32 client d/ls.
Only thing I'm dissapointed with in Netware 5.x is the fact that we still don't have a decent Open Source client. Hell, even a closed source client would tide me over.
This is not to abuse the excellent work of the ppl behind such wonders as NCPFS and MARS-NWE, or even Caldera for their client, but we really do need a proper NDS PAM plugin, and KDE/GNOME integration would be good
Netware -> Excellent choice if your too chicken for unix, and haven't seen the light of Open Source. Y2K compliant, has been for over a year.
Unix -> Power. Flexibility. Scalability. UNIX is your friend. Naturally Y2K compliant.
NT -> Lack of stability. Lack of Y2K compliance. Lack of Power. Lack of decent command line driven programs. Pretty though. "Polly wanna Cracker?". Excuse the pun.
-- James "Bragi" Deucker Patrician of Networks
How come I did not read the word 'stability' even once in the article?
Test server:
4 Pentium II/III CPUs
4 Intel NICs
RAID l5
2G RAM
Apache/Samba/no kernel tweaks
Why is it that all companies insist on picking hardware on which Linux performs the poorest? It seems our friends at ZD have been chatting with our friends at Mindcraft methinks (or perhaps M$ themselves).
Despite the fact that we seem to compare more favorably in this study than we did in the Mindcraft study, there is an extrememly important lesson we need to take away from this "losing in the benchmarks" experience as of late: we need to take these deficiencies and turn them into future strengths.
It was put very well by Linus himself in a previous poster's message. To paraphrase, anything that can be interpreted as a weakness in Linux by a media or testing agency must be improved. These are worthy pusuits, and if we keep doing them at the rate they are discovered, (unlike our M$ friends), we will eventually surpass all other OSes in every respect that matters.
We should probably place a particular emphasis on improving our SMP code, because that's the area we probably have the most to gain. All those other driver optimizations will only help us if by some luck the testing agency picks the same ones.
Anyways, I hope everyone won't get discouraged over this recent benchmarking FUD. The acceptance of good things is not always an easy road.
The opinions I post here have nothing to do with my employer.
Erm... may I point out that Apache is, although not
multithreaded in the sense of multiple threads in
one process space, a multiprocess webserver?
That means any SMP is taken full advantage of!
I often find those buzzword-deep remarks about multithreading
rather annoying, as the only reason IMHO it's hyped so much
these days is because NT is so bad at IPC and creating processes.
Also, what buggers me is the silly idea in the article
that Apache would need to fork() for each request. This is nonsense, as
you can configure as much pre-spawned servers as you want! So the reason why they suppose why apache whould perform worse than a multithreaded server escapes me. As to CGI, forget it, that _does_ need a fork(),execve() for each request. Rather, go fastcgi...! This way, application services can be prespawned too and reused between requests. Just my Hfl. 0.05...
All generalizations are false, including this one. (Mark Twain)
And again they are using 4 intel ethernet adapters, probably configured to do some sort of striping/loadsharing/whatever .. i think that's what gives NT it's edge in every benchmark.. .. watch for the 4 ethernets :(
..
I think there's going to be a lot more like this
It shouldn't be to hard to implement into linux, bind 1 ip to 4 ethernets and send through whichever one is free