Slashdot Mirror
NOS Crossroads
Mark Wright sent us a link to some benchmarks over at ZD Net that
examines assorted
NOS Options.
NT is benched, as is Solaris, Netware and Linux. Linux holds
up quite poorly in this review.
← Back to Stories (view on slashdot.org)
Posted by Jeremy Allison - Samba Team:
0 ,401974,00.html
:-). I can beat NT comfortably
> Now, as both Solaris and Linux had nearly
> identical graphs for the NetBench part, and
> both were using Samba, I think we know where
> the bottleneck there is...
Err, actually no. The bottleneck isn't Samba.
If you look carefully at the Solaris analysis at page :
http://www.zdnet.com/pcweek/stories/jumps/0,427
You'll find this interesting quote :
"To isolate the disk subsystem as a bottleneck, we created a
temporary RAM disk to hold workload files, effectively
eliminating the need to hit the RAID array for data. In this
configuration, the powerful capabilities of Solaris 7's networking
kernel were unleashed--to the tune of 360M bps on NetBench."
What this means is that when Samba is run on a very tuned SMP
OS such as Solaris (ignoring the disk subsystem for the moment)
then Samba can produce numbers that out perform *all* the other
systems (the peak NT number is around 340 I think). What is killing
Solaris here is their awful disk system. If they had a decent disk
file system they would have had beaten NT when using Samba to
serve Win95 clients as their SMP is so good.
This corresponds well with the results I get in the SGI labs
using IRIX, which is also a highly tuned SMP OS (but with a
better disk file system, XFS
using Samba and IRIX on an SMP box, but IRIX only runs on MIPS
boxes from SGI.
What this means for Linux is that we need to do more work
on the SMP scaling in the Linux kernel, as Samba isn't the
bottleneck here. I'm doing a lot of work on userland caching
at the moment to help out on the Samba side, but Linux just
needs a bit more SMP work. Don't worry, it's coming (I know
*lots* of people working on this)........
Regards,
Jeremy Allison,
Samba Team.
I develop intranet apps for a living and I wonder why they keep testing "Enterprise" webservers only using static pages. Most of the load on any webserver is going to be on generation of dynamic content. I don't Care if you are using CGI, Java serverlets, mod_perl or whatever.
In most real applications static files will clog the network pipe before it hits the CPU. And as been noted there are some unix webservers that can serve static pages much faster than apache.
But we do need to document all of this better.
Erlang Developer and podcaster
There were some other things I thought were kinda strange...I'll concentrate on Solaris here.
For Solaris they actually used Solaris on Intel, which is fair enough considering they were looking at doing stuff on the same hardware, but isn't that good for 'real world' situations (A comparison with a Sun E450 would have been interesting) because most people who use Solaris use it on Sun hardware. Some things are a bit unclear - they seem to say they got the Solaris box from Sun, even though Sun don't sell Intel based boxes themselves - they get OEMs to do that. (actually, they correct that later, saying that Sun brought in a Dell PowerEdge box) They don't say when they got the box, but they did mention Sun's Project Cascade (think Samba for Solaris) but didn't mention that products for this are now available (well, availability was annouced a few weeks back, though I don't know about x86 versions).
They gave Solaris (on Intel) a D on RAID due to lack of support for PCI cards (not sure how fair that is) which is kinda funny when Solaris on SPARC has about the best and most reliable RAID setup out there, according to people I've talked to.(NetApps were also highly praised btw) They then criticize Sun for being 'expensive' (the hardware is, sure), when they were not even testing Sun hardware, while Solaris itself is actually very cheap for a commercial OS. (NT is only cheaper than Solaris when your NT box has no clients) They then have contradictory stuff about Solaris - stuck in the datacenter on some pages (the main ones), while on other pages (the Solaris specific ones) they give a different picture...
Btw, in the final page about Solaris they mention a report from the Standish group, but they don't give a URL to it. It's available here - Solaris Vs NT.
Articles like this, which show some potential weaknesses with Linux, are excellent guides for the developers to continue refining the already excellent OS that Linux is.
It pointed out that:
- Linux + Apache maxes out when it gets 2000 HTTP requests a second
- Linux + Apache maxes out when it is transferring 200 Mbps a second
These kind of benchmarks, although unpleasant to read, have worked to improve Linux in the past. The fact that Apache no longer attempts to perform a slow getaddrbyname (reverse DNS lookup) operation every time someone requests a web page is the result of benchmarks showing NT web servers beating the socks off of Linux web servers that did this inefficient operation.The web page tunelinux.com is the result of the much-discussed Mindcraft study.
Linus fixed a problem with Linux yielding threads when it was shown by an informal benchmark that NT was much faster when yielding threads in a tight loop. Of course, this being a Usenet test, a long flame war started arguing whether the test was legitimate. Linus had the very mature comment that "Anything that could objectively make Linux look bad should be fixed" (or words to that effect).
My only objection to these ZDNET studies is that they do not always explain in sufficient detail their testing methodology. As long as their story explains their testing methodology, these articles should be studied by the developers with a fine tooth comb.
- Sam
The secret to enjoying Slashdot is to realize that it should not be taken too seriously.
For example, the ability to serve lots of hits per second on static web pages from a single box has no relevance to real-life web sites. At 1000 hits per second, a single Linux machine can serve about as many hits per second in these benchmarks as the whole Microsoft web site receives. That seems more than enough, and it's clearly not where real web sites are hitting their limitations (Microsoft uses dozens of their machines for their web site). I think the reason why Microsoft like this kind of benchmark is because it's easy to tune the OS for, even if it has little impact on actual web operations.
Also, the importance of SMP is overrated: the need for SMP on NT and some other systems arises often simply from licensing and system management issues; in many server applications, separate machines are preferable.
The benchmarks also don't take into account cost/performance. ZD claims "NT excels in NetBench". But actually, it only does 50% better for a price of at least $800 more. For that amount of money, you can buy another Linux machine and double Linux performance.
Most importantly, however, I think it is wrong to consider Linux, Solaris, and other UNIX systems to be "competitors". People can (and do) run mixed UNIX environments. For example, I might use Linux for all the web servers and an AIX machine for running a DB2 enterprise database that backs it all. Using Linux means there are lots of directions to grow in and lots of compatible commercial vendors to choose from.
If I develop for NT, I'm stuck when NT runs out of steam on its measly 4 or 8 processor Intel boxes, or when it runs out of its 3G address space. With NT, there is nothing to upgrade to.
Linux clearly isn't for everybody or everything. Only Microsoft seems to have the hubris of thinking that a single OS (theirs) can work well for everybody. Linux is part of a family of operating systems from different vendors that are interoperable and mostly compatible, and that only as a group cover most needs from embedded systems to mainframes. But within its own niche, R/D desktop applications, server farms, and small to medium servers, Linux is actually quite good.
Why is it that EVERY time a benchmark comes out which claims that Linux doesn't perform as well as other OSes, people claim that it was because of biased testing? Perhaps people had justification for bashing Mindcraft's tests, but this evaluation seems to have been done very well.
Even Linus says that, thus far, Linux had been developed with stability and maintainance in mind, not necessarily raw performance. Also, for the most part, linux developers haven't had the resources to spend on enterprise class servers for use in performance testing and coding. This is probably why Linux always seems to be the best performer on relatively inexpensive machines -- it has been developed and tuned almost exclusively on them.
I think that most people agree that Linux has a long way to go before it will be the best (performance-wise). The fact that is is GPLed will certainly help, but we need people (companies) with the resources to spend on developing Linux with a goal of performance. It will probably take some time before linux coders stop playing catch-up (i.e. trying to support all the devices and functionality of other operating systems) and start working hard on optimizations.
Frankly, I'm not even sure that a "bazaar" model of development can support this goal. In many cases, when you are writing code (esp. systems code) with a goal of squeezing the best possibly performance out of it, some of the most effective optimizations are nearly incomprehensible to people who haven't spent months examining all of the subtle interactions which make the optimization so effective. Since I doubt that Linus want's a kernel filled with magic that only a few wizards understand, such optimizations may never make it into the kernel (unless the kernel forks). These are the things which turn into debugging nightmares later on. I'll bet that both the reason for the speed of NT compared to Linux as well as its notorious instability are because of this.
Incidentally, no flames please. I've been running Linux exclusively on my machine for a couple years now; that means none of that "well, I still boot Windows occasionally to run games" crap either. I just think that we should takes examine these published benchmarks for valid points and see what we can to do improve our scores. This doesn't necessarily mean benchmark specific tuning (which is what most companies do) either. Its only that just screaming "FUD!!!" doesn't accomplish anything. Hopefully, in a couple years, Linux will be so ripped that it will be difficult for someone to de-tune Linux to make other OSes appear better.
-nooM
> If Novell supplied NetWare with a real SMP
;-)
> kernel, NetWare's performance would be
> show-stopping.
Netware 4.x and older really had a problem with SMP, especially if Maximum Service Processes was set too low, but Netware 5.x is a different kettle of fish. It's SMP is very damn good.
> Unfortunately, in its current state, NetWare
> leaves a lot to be desired not only in
> scalability but also in application support.
Netware 5.x doesn't have many applications ported to it, unless you count such small things as Oracle and Notes.
> Couple this with Novell's decision to divorce
> great applications such as ZENworks and Novell
> Directory Services from NetWare, and the value
> proposition for NetWare becomes even murkier
I'm sorry, NDS isn't part of Netware 5.x ? or even 4.x?????? Did these people even install this product? ZEN is bundled with Netware 5.x [admitedly without the Helpdesk or Remote Control functionality] as well. And does a damn fine job. Heck, it's even bundled in the latest Win32 client d/ls.
Only thing I'm dissapointed with in Netware 5.x is the fact that we still don't have a decent Open Source client. Hell, even a closed source client would tide me over.
This is not to abuse the excellent work of the ppl behind such wonders as NCPFS and MARS-NWE, or even Caldera for their client, but we really do need a proper NDS PAM plugin, and KDE/GNOME integration would be good
Netware -> Excellent choice if your too chicken for unix, and haven't seen the light of Open Source. Y2K compliant, has been for over a year.
Unix -> Power. Flexibility. Scalability. UNIX is your friend. Naturally Y2K compliant.
NT -> Lack of stability. Lack of Y2K compliance. Lack of Power. Lack of decent command line driven programs. Pretty though. "Polly wanna Cracker?". Excuse the pun.
-- James "Bragi" Deucker Patrician of Networks
And again they are using 4 intel ethernet adapters, probably configured to do some sort of striping/loadsharing/whatever .. i think that's what gives NT it's edge in every benchmark.. .. watch for the 4 ethernets :(
..
I think there's going to be a lot more like this
It shouldn't be to hard to implement into linux, bind 1 ip to 4 ethernets and send through whichever one is free