Slashdot Mirror
NT vs. Linux - Mindcraft Vindicates Itself
MauricioAP writes "the new benchmarks from mindcraft, or the NT vs. linux, aren't good for Linux, especially for RH guys.
Check this out." Reliability and "bang per $$" aren't addressed by this test or the results might have been quite different. But within the limited parameters, which may or may not accurately reflect real-world conditions, it looks like Mindcraft has been quite fair. (Please read carefully before judging.)
Note the date at the top of the referenced page - June 30, 99. (Which explains why they are using old builds of Linux and old NT service packs.)
-Blake (who didn't realize the Linux crowd hadn't already looked at this updated benchmark
Note, however, that the tested kernel (2.2.6) is one prior to the single-threaded-TCP fix. I would like to see these tests done with a more recent kernel.
Again, we must concede that on unrealistically high loads, in an unrealistic test scenario, a professionally tuned very-high-end PC with 4 CPU will outperform an older Linux kernel.
However (sorry Microsoft), that doesn't matter to me. What is also important is reliability, maintainability, cost, support, standards-compliance, and a host of other things. For me, Linux still beats NT when all these factors are considered. Also, if I wanted a very high-end SMP box for web serving, I'd probably choose Solaris anyway. Microsoft, you're barking up the wrong tree. Let's see this test repeated, but compare NT with a Sun UE450 next time.
--
--
The early bird catches the worm. The worm that sleeps late lives to see another day.
Mindcraft:
The major performance problems are with the TCP stack, which is single threaded in the 2.2.x Linux kernels, and with large-grained kernel locks that degrade multiprocessor performance. The Linux community is addressing these performance problems and others in their 2.3.x kernel series.
Well, I'm glad that they recognise that work is being done on this. It is very much the case that Linux does have SMP scalability problems, and I think we all knew that prior to this report.
Regardless, I still stand by the old motto, there are lies, damned lies and statistics, run Linux, run BSD, run NT, do what you will, but be sure to be happy, with what you run. I would like to see how NT fares against Linux & BSD in the real world, how about this test:
The test will last for one year.
The machines will be under constant varying Web & File serving Load.
The NT box will also run a 16-bit application.
I think we all know what's going to happen over time here...
You can't test NT performance over 15minutes of file/web-serving, NT may have only leaked 15Mbs of the available 1Gb in that time.
OK, I don't know whether they tested for 15minutes or not, but I did look and cannot find anything regarding the duration of the tests. Can someone please comment on this?
Why the hell are we discussing a benchmark ran on a hardware config designed especially for NT:
1. MindCraft once again used a quad ether (but skipped anouncing it) and the infamous "EtherStripping" break your switch stuff.
2. Mindcraft once again used the Dell machine which has a RAID running better under NT than under Linux
The benchmark is faulty by design:
1. If you want these speeds you use a Gig Ether on the server in full duplex mode not a questionable technique that actually breaks lots of real networks.
2. If you want real OS becnhmarking you use an architecture that is equivalently supported by bothe OSes.
Overall:
I have tested Linux with GigE (it can almost pull physical speed on machines much cheaper than the Mindcraft Dell monster) and NT has been officially tested by most GigE manufacturers. The results used to be available at the packetengines site butit looks like they were dropped when moving the site to alcatel. Anybody a link please? I would not quote them so nobody blames me for flamebaiting...
It will be rather interesting if someone finally does this benchmark on a sanely designed network (no etherstripping BS) and with proper hardware.
To conclude I expected better from RH than accepting a doomed bench (on hardware and in a network setup where they cannot win).
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Some people may flame me saying, "You don't care anymore because Linux is losing." Wrong answer.
Here's why: I LIKE LINUX
I genuinely like it. Yeah, so these benchmarks say it is not as fast or as "good." What is good anyway? Good to me is: reliability, configurability, usability, extensibility, scalibility, inexpensiveness, fun, etc. Linux shines in all of these areas and more. Yes, that's right, I said Linux is FUN. Linux is plainly more fun to use. I don't care what any some benchmarks say. We all know that benchmarks are unrealistic. They don't test "real world" conditions and situations. I think we should use their criticisms (only if valid, of course) to help Linux be a better operating system, not to beat some other OS.
----------------
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein
Co-founder and designer at Music Nearby: http://musicnearby.com
I'm sorry, but that post is just so wrong it is laughable. If you had found some site that ran NT and was faster than Slashdot (not hard to do) you would be flamed out of existance.
Where to start?
Slashdot does use multiple webserver - it caches static pages, and
Slashdot does not use Oracle it uses MySQL. Big difference in websites.
"The response time is always good" ????? Not from where I am (Australia) it isn't. Subjectivly, dvdexpress seemed faster to me. Anyway, what does that prove? You are closer to Slashdot than dvdexpress? Slashdot has more bandwidth?
Dvd is graphic intensive, and takes longer to render in Netscape, too.
You can't compare two totally dissimilar sites, on totally different hardware.
I bet I can find apache sites that seem slower than NT/IIS sites. EG: www.Apache.org always seems very slow to me. What does that prove? NOTHING!!!!
Look, I want Linux to be faster than NT as much as anyone, but we can't even be seen trying to spread FUD like MS does. Imagine if MS stuck that up at Comdex as by "a Linux Hacker, posting on the Linux nerd site slashdot.org".
People, please think for a moment before you post, and before you moderate comments like that up. Ask yourself this:
Reader of Slashdot don't need to see arguments for Linux like this, we need to see the opposing view, so we can learn what we need to improve.
Damn.. I just know this will kill my karma, but that is crazy!
--Donate food by clicking: www.thehungersite.com
I am surprised so many people haven't realised there is no such thing as a non-biased benchmark, and that, shock, horror, Linux is perfect (yet).
Benchmarks must reduce the scope of tests and make assumptions, which are not always true, so as to be possible. They also need to be done at a point in time, and not wait 'for the next version, which is so much better'. Doug Ledford of RedHat was there for the tests and has his spin on the tests, where he talks about the difficulty of getting a meaningful benchmark. The Tranaction Processing Council are continually revising their benchmarks to remain meaningful. The big guns, IBM, Sun, HP, Oracle, Sybase, Compaq and Microsoft all use different TPC benchmarks to try and gain ammunition for sales staff. At some point Linux people will need to do the same.
The Mindcraft benchmarks look to be as fair as any I've seen. The reaction to the benchmarks is far more informative than the results themselves.
Linux can still be improved, it isn't as strong as other operating systems in some areas. The fact there is development occuring proves this point.
If you don't like the results, find a benchmark and configuration that gets the results you do like! Where there is a real deficiency lend a hand and be part of the solution.
First of all, as far as I know almost every major company has a habit of cheating in benchmark tests. For example video card drivers detect that a test is being run and enable code that skips most of the drawing primitives. This is easy to do in code that is not open source since it would take a major effort to reverse engineer the device drivers. It might be possible that NT has a feature that detects different kinds of tests and optimizes its performance accordingly (if you are for example testing throughput you would trade throughput for latency times). While this is not cheating in usual sense I think that this would be quite useless in normal mixed load situations.
The second thing is that Microsoft is quite a large company. If it wants to outperform Linux then all it needs to do is install Linux, tune it to its limits and then analyze its performance and find out weak points. Then it makes the same thing with NT. After that it just puts hundred well paint workers to make NT faster than Linux. This is made easier by the fact that if Linux works faster than NT they can just look at sources and figure out what Linux is doing better than NT. Also, it is possible that Microsoft would look at the weak points in Linux and would publish only those benchmarks where Linux performs significantly worser than NT. Anybody who does those same benchmarks would get similar results and the original benchmarks would be considered objective.
Third thing is that those benchmarks might only test peak performance - performance under high load. It is also possible that the structure of the load is untypical. This is true with most benchmarks; they rarely test systems under realistic conditions. Since I have not looked at those benchmark programs I do not know if this is the case. Anyway, peak performance is important if you want to identify bottlenecks and see what are the limits of programs. Peak performance does not tell how programs work under normal every day use.
Last thing is that I think those benchmarks are already outdated. What I would be more interested would be performance of cutting edge Linux system against similar NT system.
As a conclusion I again state that I think those benchmarks look valid. It seems that Linux kernel (and possibly also Apache) still has bottlenecks in its performance. I'm not sure if those have been fixed since this benchmark. However, I think that this benchmark should be thought of as a challenge to improve the performance of Linux. I actually think that Linux did quite well; performance differences are not THAT large when you take into account my comments above.
Though the PCWeek tests favour NT (for reasons well covered elsewhere), they do not do so by the ludicrous margin the original tests gave, and the Linux's community's cries of "foul" were entirely just and accurate: these tests show that Mindcraft did indeed load the die.
Furthermore, Weiner has never managed to justify the claim that he had asked for help in "several Linux discussion groups" when setting up the first test: searches show that he only posted *one* article, and that was met with requests for clarification that was never forthcoming. So as it stands we're quite justified in believing that Weiner is a flat-out liar on top of his other sins. That's not vindication.
--
Xenu loves you!
The real problem with the Mindcraft benchmark has nothing to do with most of what they cited: the graphs are painfully clear that the limited resource is network bandwidth. That's why it's so funny when they say "We'd never test a server that's resource-limited. What's the point?" That's what I'd ask them now.
Note that they test with one and with four processors, but do not test with one or two ethernet cards. In fact, they never mention the complete hardware configuration of the machine, so we just have to assume they used the same f*cked-up four ethernet card configuration.
There were actually benchmarks put out by c't explaining this, with graphs, and real tasks. Linux performance generally did much better until that second ethernet card was added. I'll believe them, that it's a software limitation in the TCP stack, but I'll also believe that they were exploiting a known problem in the Linux kernel--that only happens under these strange conditions--to their ends. Until they show some benchmarks with the ethernet cards mentioned as a factor.
NT vs. Linux Server Benchmarks: informative and interesting, but most of all truthful, with a link to the c't article I mentioned, and many other more realistic benchmarks.
---
pb Reply rather than vaguely moderate me.
pb Reply or e-mail; don't vaguely moderate.
They looked valid to me before phase three. I love Linux, but it has it's limitations. These need to be worked on. It's funny, becouse when users are actually put under the gun, this is how they react:
Linux User - Linux is MUCH better then NT in terms of performance, uptime, and general reliability.
Random NT Statments - Have you ever tried a test of the two?
Linux - "Sure, I do it all the time"
NT - "Ok, let's try, here we go *beatbeatbeat*. Oh look, your conclusions are wrong.
Linux - "The tests where rigged! They cheated!! MOMMY!!"
NT - Ok, let's try it again. You come watch, and tune the Linux box to your hearts content.
Linux - Ok, we'll whip yer but in a fair fight, no problem.
NT - Ok, here we go. *BEATBEATBEAT*. Oh look, the number look exactly the same
Linux - Well, we've fixed many of the problems since the tests. We'd whip you but in a rematch sometime.
NT - Ok, how's about right now? Let's go.
Linux - Err, no thanks. But next time we run the tests on 386/25s!!
NT - That's stupid.
-- I'm the root of all that's evil, but you can call me cookie..
I can't believe anyone would make this assertion. While I agree completely with the other half of your statement, that a good sysadmin must understand the aims of yout IT systems and know how to implement them properly, I would say that thoroughly understanding an OS is a vital prerequsite for that second clause.
Anyone who has worked with an operating system, a programming language, heck, a make and model of car, knows that there are essentially four levels of competence. First, complete incompetence. You have no knowledge, you try things and you screw things up. Second, basic competence. You have some knowledge. You successfully carry out basic tasks. You use the system without damage, but there are vast areas about which you know nothing. Third, competence. At this level you know your way around. You know how things work. You know what all the parts do. Fourth, high competence (guru). You not only know how things work, you know why. You develop a holistic sense of the system/language/automobile. You can imagine how things work. You can be presented with an unfamiliar situation and you can figure out what to do about it.
Most people with whom I have worked in IT (and I've been working professionally as either a system admin or a programmer/analyst for over 12 years now) are at what I would call level 3, and a fair number are at level 4. Thorough knowledge of a system is required to be at level 4.
The notion that one does not require deep knowledge of systems to be a systems administrator is tenable only in a system with nothing ever happens that is outside the training materials. No such system exists.
If you are arguing that deep knowledge of a system is not required to be a sysadmin, then I sure don't want to work at your company. If, OTOH, you are arguing that deep knowledge of a system is not in itself sufficient to be a good sysadmin, well, then I've been wasting your time and I apologize, because I agree with that...
Actually, I've been waiting for the argument that seems to apply most directly: that benchmarks don't really represent the Real World(TM). For example, given the oft-cited media demographics of the "poor little pre-IPOs who just don't have enough money to buy M$'s latest work-in-progress," you'd think there would be some tracking of multi-function boxes.
;-).
For example, if you're using Linux as a major player at work, do you have a _pure_ web server box and a _pure_ file server box? Or do you have one or more boxes that really do lots of different stuff all the time?
This is a totally different measurement and gets directly to the heart of the "Bang for the Buck" argument: if I have $K dollars to spend on _one_ box to do everything, would I rather spend it on the best hardware (with cheap, powerful, easy to use) that I can get? Or am I going to have to spread it around on hardware+expensive OS?
At this point, you're talking real "business case" numbers that you can talk to your boss / capital approver and say "See, the path I recommend makes more sense in terms of total functionality, operator training, total cost of ownership, blah blah blah". (Anybody who's ever had to go through that drill knows what to put there - I just justified a $60K IRIX box for my desk... 8-).
So, the benchmarks that I'd really like to see are things like the following:
1. Given $5K, $10K or $20K to spend on a _single_ general purpose server machine, inclusive of OS and any serving technology, what's the best of class? Linux should win here by definition: cheap OS = big box, expensive OS = littler box.
2. Given the test platforms from #1 above, run through the Mindcrafty benchmarks again. At this point you're comparing dollars to dollars, so the results might be interesting this time. Not useful (steady state measurements don't indicate real life - does the Slashdot effect slowly ramp up or hit all at once...
3. Now show me something interesting. E.g., X web clients connected, file server traffic jumps up, then web clients drop off. You know, scientific method and all that?
At this point, I'd say the Mindcraft data is still far from being useful. Simply put, there's not enough of it.
-- Doctor Bob
NT was designed to do this sort of thing - keyword designed. :P.
/tmp/X11-unix lock files? ERK. :P, the registry is a good thing. Yes when win95 came out there were registry problems but I haven't had any problems since 1996. It's a great idea, it's like having a database to store all your settings.
Linus didn't design Linux for the kind of work which NT is excelling at in these benchmarks.
When Dave Cutler sat down and designed NT, this was the kind of things they were trying to do, fine grain kernel locks, high performance and scalability. The market place has unfortunately seen many of the good things about NT get forgotten (portability for example), but NT still stands there with the ability to scale MUCH MUCH better than Linux can at the present.
Yes you may feel like going out and burning a few MS cds or whatever, but at the end of the day it's true. Improvements ofcourse are being made to linux, and linux may catch up.
However, I'm actually a bit worried about the fundamental design of Linux itself - I'm not saying it's totally 30 year old technology - far from it - but having experience with linux and NT for quite a number of years now, to me, NT seems like it is better designed and had good goals.
I won't bother to argue about whether they were met or not here tho
Some fiddly things about Linux/Unix I don't like are:
-Threading. According to IBM, Linux native threads are mapped processes!??! which makes their JDK rather slow compared to NT.
-Mutexes/Semaphores/CriticalSections etc - why doesn't Linux use them? I mean for god sakes what the hell are linux applications writing *.pid files around for? And what about
-Componentisation - it's happening slowly but only in the past few months (maybe a year). I'm still waiting to see the Unix APIs wrapped up.
-Registry. I've said it before and I'll say it again
Now I don't really care whether the registry is one huge file or several files (user and system) like in NT, but I just want some STANDARD APIs for reading writing settings - fast APIs.
Ofcourse the registry has other uses too, like storing COM/CORBA UUIDs etc etc etc.
Being a database it'll definitely be faster than parsing text files, and even better it's much easier to programatically add/remove/change settings (trying to parse text files to do that sort of thing sucks).
Anyway, it seems everytime something about Linux comes up the response is "someone is working on it". When it comes up again the answer is the same, and then everyone ignores the strengths that NT does have because Linux will have it cause "someone is working on it".
Just give NT, MS and Dave credit, and move on.
Linux is not the solution to everything. It's a great free small-medium server & emerging desktop OS. Let's leave it at that for the next year or so.
In _The Psychology of Computer Programming_(*), Gerald Weinberg wrote a story about a programmer who was flown to Detroit to help debug a program that was in trouble. The programmer worked with the team of programmers who had developed the program, and after several days he concluded that the situation was hopeless.
;)
On the flight home, he mulled over the last few days and realized the true problem. By the end of the flight, he had outlined the necessary code. He tested it for several days and was about to return to Detroit when he got a telegram saying the project had been cancelled because the program was impossible to write. He headed back to Detroit anyway and convinced the executives that the project could be completed.
He then had to convince the project's original programmers. They listened to his presentation, and when he'd finished, the creator of the old system asked,
"And how long does your program take?"
"That varies, but about ten seconds per input."
"Aha! But my program takes only one second per input." The veteran leaned back, satisfied that he'd stumped the upstart programmer. The other programmers seemed to agree, but the new programmer wasn't intimidated.
"Yes, but your program doesn't work. If mine doesn't have to work, I can make it run instantly and take up no memory. "
Moral of the story: correctness first, then speed.
How fast would NT be if they fixed it?
These benchmarks were released on the day of Gates' Comdex Keynote? Coincidence?
Everything we learned from the antitrust findings of fact would suggest that it's not coincidence. I therefore have to hand it to Microsoft, not for winning yet another questionable benchmark contest, but for maximizing the spin benefits thus obtained. That is true art.
This is an example of a Microsoft "spin" attack. There will be more to come - the battle has just begun. Let's admit it, Microsoft won this round in the spin battle - mainly because we weren't fighting, and took a punch below the belt. Referee - what referee? OK, so we learned something about the rules of the game.
Let's do two things:
1) Make Linux better so we win these high-end SMP contests as well. I don't know about you, but I'm treating myself to a 2-processor machine this Christmas, and I want it to kick ass running Linux. With 100,000's of geeks likely doing the same time, we can expect 2000 to be the breakthrough year for geek-SMP. We also need better file I/O. Not that the existing I/O isn't damm good, but it has to be the best, right? (Personally I'm putting my money where my mouth is - as a developer, I can make a difference and thanks, MS for getting me steamed enough to jump in.)
2) Master the PR game. Microsoft can, and will hurt us with PR. Some may say "so what, who cares what Microsoft says, it's how good Linux is that matters" and there's a lot of truth to that. Nonetheless, why don't we cover all the bases? We need an open-source think tank cum swat team whose only purpose is to anticipate, forestall and counter the PR moves that Microsoft makes. We have the collective intelligence to play that game well, and hey, it's a fun game when you win.
Life's a bitch but somebody's gotta do it.
Using one fast adapter instead of four slower ones woudln't help because you would still fail to take advantage of the other processors. The interrupt servicing would be serialized to one processor at a time.
By binding each of four cards to a different processor, you allow up to four interrupt service routines to proceed concurrently. The question is whether the service routines can get into the stack without getting in each other's way.
In Linux 2.0 and before, what happens is that each adapter, when it receives a packet, calls a routine which atomically enqueues it into a global queue. What pushes packets from the global queue is ``bottom half processing'' which is sort of a virtual thread that does its job when returning from system calls or from interrupts.
I don't have a complete picture of the 2.2 architecture (despite successfully porting a network driver to it from 2.0!) but I think that the bottom half stuff is still emulated in a coarse way. Historically, a lot of code depends on bottom half processing to be atomic with respect to itself and you can't change that overnight.
The actual pushing of packets from the global receive queue into the upper network layers is done as part of this bottom half processing. So is the pushing of backlogged transmit data down to the network drivers. If only one processor can do this processing at a time, there is an obvious reduction in concurrency. An obvious extension would be to let all of the processors do bottom half processing concurrently. In the case of network input, for instance, all the processors could enter into a loop in which they are yanking received packets from the global queue and driving the higher level TCP/IP code concurrently.
Here are some of the assumptions that you could make when programming for 2.0 kernels and earlier:
- bottom-half processing is atomic with respect to itself. That is, it can't be interrupted and re-initiated. It can, however, be interrupted to service lower level interrupts.
- bottom-half processing happens only at interrupt level one. Thus by incrementing the interrupt level, you could protect a section of code against being re-entered by bottom-half processing. The start_bh_atomic() and end_bh_atomic() macros would just do this intr_count++ and intr_count--. Thus synchronizing between system call (process context) code and bottom half callback can be done trivially, and without touching the processor interrupt mask at all.
- disabling interrupts protects you against everything.
The real challenge has been not just in going from coarse grained to fine grained locks, but in reworking the assumptions.
For backward compatibility, the old mechanisms still exist in 2.2. For example, you can still use sti() to disable interrupts, and can continue to pretend that this works as before. Under SMP however, it is emulated in a rather gross way! If you want the real interrupt disable instruction, you have to use __sti(), which only disables interrupts on the current processor, not guaranteeing complete atomicity.