Slashdot Mirror
Quantifying "Bandwidth is the Limiter"
John Lazzaro writes "Found this linked from Dave Winers site, an
analysis that puts numbers behind the oft
quoted "Linux + Apache will saturate any
reasonable Internet link given static pages" It basically
assumes the mindcraft tests are accurate, and then tells you
what it means. Most interesting is the comments about
MSs online tech support, the hardware they use, the results
of the benchmarks, and the fact that static content is
practically irrelevant.
I hate to say it, but I think we lost this round, M$ wins. /. for examples.
no, I'm not talking about the benchmarking. I'm talking about the non-acceptance of them. remember that M$ is a marketing company, and little else. no matter whether it's right or wrong, non-technical people simply STOP listening to technical details after a while.
that point has been reached. when the Linux community screamed out about the first test, people listened, even in the mainstream. but now, for all THEY care, a new, fair test has been conducted, and any continued discussion on our part will appear as whining - no matter if it's legitimate.
just look at all the anonymous coward postings on
fact is, M$ won this, because they can a) tell everyone how NT is superior to Linux (ignoring the fact, that this has only been proven for one specific setup under specific conditions) and b) point a finger at us for "whining" (ignoring the fact that we may have justified criticism).
maybe they were aiming at this goal all the time, maybe they're just picking up the chance. but one way or the other, from the MARKETING pov (and we all know that that's all M$ cares about), this has been a huge victory for them.
so I suggest everyone stop whining and go back coding. the time for criticism is past, like it or not. anything else you say be better damn constructive (in the sense of "we should fix this and it'll speed things up") or M$ will surely find a way to use it to THEIR advantage.
Assorted stuff I do sometimes: Lemuria.org
What sort of stats do you have there? peak Hits/sec average page size, uplink size?
The site is distributed among numerous colocations, so there isn't "one" uplink.
OK, how about stats for a representative colo?
And understanding this, you are now allowed to use *the* programming language, Fortran. Swap a "D" for the "E", and you specify double precision while you're at it.
Basically, I think he's right, any reasonable Linux web server can saturate most avaliable bandwidth with _static_ pages.
But, I don't think it's time to settle for that, I think we should go back to look at multithreading and why 4 CPU's gave NT more of a boost, and what that means to what needs to be done in Linux. And, I think that this study also shows that "internet" isn't the problem, so let's look at some faster stuff in the networking world, like SAMBA needs on an "intranet."
Alright. Fine. Linux is not good at striping across multiple NICs (or at least across 4). IMHO that's not a normal thing to do anyhow...
;).
/. or Excite or similar mad dynamic pagegen) that could be forced to work on ASP or PERL under both IIS & Apache? Would be fascinating
;)....
There are really two parts two heavy-duty serving: files and web.
a) Web content
You don't need 4 100Mb NICs to serve real webcontent (no, corporations using "intranets" to be buzzword compliant doesn't count. when i see a serious intranet implementation i may believe it. till then it's just buzzword compliance
No one serves more than cdrom.com, and if they're the only ones straining 100Mb (on 100% static content, at that) 100Mb is good enough fer anyone. Next issue: dynamic content. I want a fair benchmark of IIS and (mod_perl) Apache on PERL and ASP (using mod_asp).
How good is mod_asp ennyhoo? Does IIS do PERL? Does anyone have serious site implementations (on the order of
b) File serving
I'm still not convinced in the real world you need more than a 100Mb card -- if your fileserver can sustain 7 megs/sec (realistic on ethernet, peak is 10 megs/sec) you're doing pretty well with that RAID array, since no HD i've seen can do that alone, especially not when doing simultaneous serves (and therefore not reading contiguous blocks). I'm bothered by the SMB test setup in that sense -- if it was tiny reads out of the cache, big deal. I wanna know who can move real volumes of data, and fast.
Of course, I know the answer. Sun, and after them SGI's Origin2k boxes (the fastest Windows networking fileservers in the world
If you want brainless system installs, look into Ghost by Symantec. It is a dos program that does disk duplication. It is completely configurable from the command line. With a bootable CD containg ghost and the disk image (compressed) an entire machine can be installed in less than 10 minutes. Assuming they have a decently fast CD-ROM of course.
So it's like setting "gunzip | dd" as the init program of a bootable Linux CD, only $40, right?
So, the difference between NT and Linux only shows up given unlimited bandwidth. That doesn't suprise me at all. Think about it. MS has the time and manpower to make their server use more bandwidth than they could ever use. Linux developers on the other hand do not much care how well the server performs beyond the capacity of their lines, because they don't care! There is no itch to be scratched in terms of development, because the system works fine for what they ask it to do.
:-P
In fact, it wouldn't suprise me at all if MS worked *very* hard to make benchmark results like this, even though they don't mean anything. Why? Because at MS, appearances are more important than fact. It's all marketing.
Don't worry about Apache not being able to handle extreme (perhaps impossible) amounts of bandwidth. If the day ever comes when it's important, it'll happen! Somebody will find that Apache just doesn't cut it anymore and will fix it! Or maybe they'll fix the kernel if they need to! That's the beauty of Linux and open source.
So, would you rather spend thousands for an operating system that could use more bandwidth that you could ever pay for, or spend *nothing* for an operating system that will handle just about anything that you can throw at it? I know where *I* want to go today and it ain't with MS!
There. Now I feel better.
Ben
I'm right with you that spending the money on hardware is the right thing to do. However, throwing around "BackOffice" as comparable to the free stuff that comes with Linux is bordering on the FUD Zone.
BackOffice has:
* System Management Server (I know, who wants it, but is there anyting eqivalent for Linux even if you did?)
* MS SQL - It's not Oracle, but does MySQL come even close?
* Exchange server - Again, the MTA in Exchange ain't sendmail, but as far as the mail/groupware/calendaring feature set a fair comparsion would only be Netscape's commercial products.
* SNA Server for talking to older mainframes. (Some people need it.)
* Some other stuff I probably forgot.
Now it could be you don't want this stuff. But if you did, you'd have to buy it, even on Linux. (And, it would probably be more expensive.)
--
Business. Numbers. Money. People. Computer World.
If your big iron actually talks TCP/IP, I don't think SNA Server is needed (although modern SNA Server versions might do other things, like screen scraping to a web page).
Older IBM stuff didn't (or didn't always) use normal LAN protocols, so SNA Server could be used as a gateway between your LAN and the SNA Network. (Someone else could probably explain this better, and use all the correct IBM model numbers!)
As for PHP+MySQL doing what Exchange does - you're right it *could*. However, most places, when given the choice between buying a calendaring package or writing their own would probably buy one. You're argument is like saying you don't need a RDBMS because you've got GCC.
--
Business. Numbers. Money. People. Computer World.
Don't get me wrong - I'm not saying the BO stuff is the best or even necessary in many cases. Just that the usual free stuff isn't really in the same catagory as BackOffice or commercial products announced or shipping on Linux such as Oracle, Sybase, Tivoli, Unicenter, Netscape, or Domino.
--
Business. Numbers. Money. People. Computer World.
Thanks for posting this. It's awesome to get some real life data to throw into this. I wish that more people would do this. Thank you.
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
If someone tries to convince, say, your boss that you should use NT instead of Linux for your website (say you're hosted by a partial T3 and only serve static web pages) and cites these results, will you say:
A) "You're right. These benchmarks prove that NT is superior to Linux and we'd be foolish to go with Linux now, though if we work real hard coding at Linux et al. linux will be able to beat NT at benchmarks so we can switch then."
or
B) "Actually, if you look at the tests rather than the soundbyte about them, you will see that they prove that a Linux/Apache system will be waiting on our internet connection anyhow, and Linux is a much better value protosition than NT is, remote administration to a box with no mouse or monitor is extremely easy, remote displaying comes standard (no need to buy citrix), you get greater stability, and source code that uses open standards so that if life takes one of its many unpredictable turns and Linux is no longer the best solution, we won't be trapped into expensive solutions that we don't want to be in due to vendor lock-in."
So do we give up, or work with what we've got? I suggest that everyone who's whining about "let's stop whining and get back to coding" at least say what you mean, "Let's just give up and try to beat microsoft on their own game instead of doing what we want." The people who do the coding are still doing the coding. This is about advocacy and marketing. I suggest that the people who do Linux Advocacy/marketing don't stop simply because everything didn't go perfectly. These numbers are only a defeat by microsoft if we let them be.
Remember, there is more than one part to the Linux community, just as there is more than one part to the body. Just as I wouldn't suggest that those who do documentation stop doing documentation and start coding, I also wouldn't suggest that those who do marketing stop and start coding.
It takes all sorts to run this world, and we shouldn't start neglecting any part of it, marketing included. Microsoft is going to try to put as much spin on these tests as they can. If we stand by and do nothing, we'll be as guilty as they are of the spinning. Sins of omission are still sins.
I don't suggest that we do it defensively, though. I suggest that we do it confidently and aggressively. act like we're in control. That's the neat thing about self-confidence. If you act like you know what you're talking about, people tend to believe you. So why should we give up acting like we know what we're talking about.
We've got some test results which prove that Linux can handle the needs of 99% of the world. Why exactly should we hush that up in favor of letting it be thought that Linux is slow.
They laughed at Einstein. They laughed at the Wright Brothers. But they also laughed at Bozo the Clown. -- C. Sagan
I have been running a linux webserver in various incarnations and machines since 1993. I started on a 486 with some decent scsi (vl-bus) and 8 megs of ram, with a 28.8 hayes modem, matching the isp's.
I generally ran 3.6 kilobytes per second of bandwidth at the time, which wasn't bad for a 100 dollar a month dedicated line.:)
Now adays we use an alpha (for past 2 years) with 128 megs of ram and a T1. One day (and I really mean one 5 minute period) our line was saturated by what seemed to be a webcrawler from taiwan which literally downloaded the entire site. It filled our pipe. That is the only time that such a thing happened and my apache server didn't even blink. I wouldn't have known but I tend to analyze my peak times.
After we decided to upgrade the software on the alpha (redhat 4.2 being a bit old ) I moved the entire site over to a k6-2 running at 300 with 156 megs of ram or something... That hardware cost about 700 all together. (the alpha cost 5000 2 years ago..) It performs fine. I have never seen the apache webserver crash or not respond based on overusage although sometimes dynamic content will slow down (and that is because we don't really optimize for speed). In any case the performance is great for the price. I have bought one copy of redhat 5.0 and have 3 servers all running great. (Finally retired the 486 cause it was hard to find vlbus hardware)...
The basic fact is that the webserver has NEVER been a crisis point or even a decision point. When we went to shttp we went to stronghold as it was the only easy solution at the time. It has never caused us problem.
Further we serve realaudio, handle telnet sessions. handle email, handle pretty much any protocol a client wants. mysql, php, perl, c compilers...
There is literally nothing that we have needed to do that our linux boxes haven't handled. I am so comfortable with them that I have placed linux boxes as controlers for permanent automated exhibits, and have so far only had one hardware crash even.
The notion that NT would be easier for me to maintain, easier for my clients to interact with, or in any way a more efficient use of money is absurd. I don't even know what they charge for NT, but I believe I would rather purchase an extra 256 megs of ram per machine, or a faster processor, then pay to be a beta tester.
I know I am preaching to the converted here, but I started laughing the first time I saw the comparisons. While I WOULD like linux to catch up with SMP and such, and certainly would like to see it scale better, I happen to like being able to buy a 500 dollar computer when I need more processing, and have it up and running in 2 hours (and that because I generally do a redhat ftp instalation)
I betcha you can't do that with a win NT install.
While I do lookforward to an OC-3, I imagine the price of leasing it will be expensive enough that I can afford any number of widdle iddy biddy linux machines to serve up pages. The days of big iron may not be over, but I don't need a mainframe to do something as brain dead as serving static pages.
D
Hey... why hope? Mindcraft (as they took such pains to point out) is a neutral party. That means that we can hire them to re-run the test. In fact, perhaps we should announce that we intend to re-hire them, at some specified date (any suggestions?) to perform a re-test... same rules as the second test, which means Microsoft gets to tune NT however they like, but we do too. This challenge is just the kind of thing that the media might pick up on, and the results of such a contest are certain to be picked up by the media.
The only hard part is that we'd actually have to beat them the second time round. And that's only going to happen if all the hype about the advantages of Open Source are true. It is true, isn't it?
If anyone likes this idea, let me know.
-- Michael Chermside, mcherm@destinysoftware.com
It isn't directly relevant, but then neither is the original test. In the basic test with a small number of pages being hit, the server can get everything out of a RAM cache and you get a cache hit rate approaching 100%. The random/10E6 test is designed to push the opposite extreme, forcing most hits to miss the cache and show how well the server performs when it mostly has to hit the disk. Neither extreme is in itself likely in the real world, but together they give you an idea of how performance varies so you can evaluate where your site falls along the curve.
But all you have to do is put the link inside an anchor, we could click directly on it, rather than having to cut and paste it. Having said that, Here it is done for you as an example:
See Http://www.ct.heise.de/ct/english/99/13/186-1/ for details, or click the link to go directly to the English translation of CT's Linux Vs NT. Article.
...Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
While the article is rather interesting in and of itself, I think it points to a bigger issue: in computer industry in general, and in benchmarkig in particular, proper scientific tools and methods are often not used.
Statistics is your friend! If some psychologists used similar methodology for their investigation, they would be laughed at -- I won't even talk about hard sciences.
Yet in benchmarking, the perpetrators completely ignore representativeness of their samples -- this is all benchmarks are really supposed to be, controlled investigation of the performance of a representative sample of real-world computing activities. How can you investigate performance if you don't even try to account for various miscellaneous factors by using proper sample selection?..
What can I say?.. The entire thing disgusts me. I would rant more, but I will simply go and sob in the corner about lack of scientific methodology in my field of choice.
--
--
Victor Danilchenko
I remember hearing about how linux follows the TCP spec and uses slow start, but NT doesn't.
When a TCP connection is first established, and one machine wants to start sending data to the other, it isn't supposed to start sending as fast as it can. Rather, it's supposed to send only one packet, then wait for an ack, then send two packets, etc. Otherwise you end up with congestion rendering the network useless as you approach full bandwith. Van Jacobson (remember VJ header compression from the days of SLIP?) has a paper on this.
Since the mindcraft test has unlimited bandwidth, no packetloss, no slow modem connections, etc. TCP stacks that don't do slow start properly can send out more data than ones that do.
To me this highlights the one true evil force in the world. No, not Microsoft. Ignorance. Dr. Science once said "Ignorance is bliss, and tonight, we're a happy country." I could not agree more.
The basic problem is not Microsoft. Not their products, not their technology (or lack thereof), not their marketing people. The problem is the number of people who do not think critically. The NT benchmark does not lie. It simply tells a very narrow slice of truth and "positions" that truth to show NT and IIS in the best possible light.
To me, the one outrageous thing in Microsoft's benchmark page is the chart that shows total cost of ownership. Now, I'm not a CIO, CEO, or CFO, but it seems the me that cost per transaction per unit time is completely irrelevant. What matters (as the author of the article we are commenting on here points out) is cost per transaction and can you handle your transaction volume?
When decision makers look no deeper than the cooked figures from NT's benchmark, when they fail to see if the scenario represents their business and technical reality, then their business gets what they deserve.
What the "Microsoft Advertising for Linux" article does that is lauditory is it cuts through to a core question. Which is cheaper given a certain use case? It wisely does not answer, but merely points out that in most cases, even in most intranets the Mindcraft/Microsoft scenario is extremely unlikely and that Linux/Apache on even limited hardware will handle most loads anyone would reasonable expect.
It also wisely points out that if you are a site in the tiny fraction that will exceed Linux/Apache's capacity, then by all means use NT/IIS.
Then, one more dig of my own at the TCO figures. Even if we grant the validity of the figure cost per transaction unit time (which I do not), what happens if you set up ten servers, or twenty? Linux costs nothing more for ten servers than it does for one. I haven't the time to see how many servers it would take, but there would come a break even point and then a point where Linux/Apache is cheaper even using the dubious measure in the Microsoft study.
Finally, I just want to congratulate the author of "Microsoft Advertising for Linux?" for showing the value of just trying some of your own math and asking, "Hey, is this reasonable?" If we all did this routinely regarding everything from computer bechmarks to medical scare news stories we would live in a much saner and less stressful world. Whichever operating system you buy.
Point #1 actually brings up something I've had on my mind for a while.
What about a code/distribution/architecture fork to attack the desktop market? Would it be desirable to fork the system to be optimized and designed for workstation use, and continue the current design/architecture/code tree for server use?
It seems to me that trying to attack the desktop market with a system that is server centric is a non-technically-optimal idea. Kind of like putting a GUI on NetWare. One size may fit all, but it doesn't end up fitting anyone very well.
The goals of a workstation and server as I see them are radically different. On the server side, how practical/important/useful are things like 3D Video and audio cards? On the desktop, these sorts of things are much more important.
Similiarly on a server I would think task scheduling and prioritization would be handled more evenly than on a workstation. How about a system that can easily "get out of the way" of a game or other app that needs dedicated resources?
Any thoughts?
http://www.bullnet.com
NT beat Linux (on these benchmarks, at least), even when Linux was optimally configured. We whined for a retest with an optimally configured Linux, and got it - hey, we even got a lot of changes in the test setup - UP as well as SMP, different RAID controller, NT as well as Win 95 clients... BUT, we STILL lost.
Rather than whining, I'd say there's a lot to be grateful for - mainly the fact that the testing has shown some prior bottleneck assumptions to be wrong, and has exposed the real problems which can now be addressed. Given the results, the press coverage has also been rather kind to Linux.
serving one static html page 4k size: NT and linux almost on par (linux ahead a few %) both systems answer 900 requests/s when hit with 512 concurent client process.
with 8k size static page: linux is between about 5 and 10% ahead of NT.. at 512 client processes the linux maschine serves about 600 requests/s the NT maschine about 550.
using a 4K page but selecting one random page out of 10E4 pages linux has about 830 req/sec and NT about 720. the linux line seems saturated where the NT line is bended down already: linux 15% ahead of NT
random 4K static page out of 10E6 different pages: linux about 270 req/s while NT has never more then about 30 req/s. that means linux is some 800% ahead..
now some dynamic pages. they used plain old CGI scripts with perl. no PHP or ASP. using all 4 CPUs linux answers 210 till 250 request/s while NT is around 60! that means linux is 316% faster
same as above but using only 1 of the CPUs of the maschine: linux around 100 req/sec, NT around 25 req/sec. linux ahead by 300%
if the script contains a sleep(3) at the beginning (to simulate slow database connection or slow client connections) the results are: linux increases the number of requests linear with the number of requesting processes and reaches about 80 req/s for 250 simulated clients. NT is saturated at around 7 req/sec. (in words: seven). linux wins with over 1000%
the only time that NT is ahead of linux (about a factor of 2) is when using 2 NIC cards instead of 1.
my interpretation of all this: i guess there are very few webservers where one would needs more then 100Mbit/s.. and then one would be propably better of with 2 cheaper systems doing load balancing.. given the extra reliabilty, remote managemnt, etc of linux and the better extraordinary better performance in most tests linux is the clear winner. doing fast CGI scripts is by far more importnat then to efficiently support 2 or more 100Mbit cards.. at least for 99.9% of all webservers or more..
greetings from vienna, austria.
der mond.
The german computer magazine ct published in their last issue (number 13) a test NT/IIS vs. Linux/Apache as a webserver. They uses a 4 processor Siemens box with a raid 5 as disk storage.
Here is what they found out: (All numbers are estimates from the charts. They may be off, but not much)
First, as soon as you have to use more than one network-adapter NT wipes Linux butt on static pages. Linux seem to be pretty bad in this area. Their guess is, it has something to do with the kernel and multthreading. This verifies pretty much the findings of Mindcraft.
The main part however was dedicated to a slightly different scenario. They tested a few different things.
The first test was serving a single static file of 4 kB size. They stopped measuring at 512 serving processes which corresponds on their environment to about 950 hits/s for bot linux and NT, Linux leading. With 8 processes NT leads 560 to 480 hits/s, from 16 to 32 processes they are roughly the same, from 64 processes on Linux leads by 20 to 50 hits/s.
The second test was the same as the first only with 8 kB size. There Linux leads the whole way by 20 to 50 hits/s maxing out at about 550 to 530 hits/s.
The third an the forth thest were again serving 4 kB files. In the third test the number of files (10000) all fitted into the cache, in the forth thest they didn't. In test number 3 NT is only a bit better with 8 processes (380 to 320 hits/s) ans then linux leads by about 100 hits/s maxing out at 820 hits/s to 720 for NT.
As soon as NT needed to use the disk (1000000) its performance wasn't so good. It stayed at a constant 20 hits/s while Linux went from 50 hits/s to 280 for 512 apache processes. It seems NT doesn't like a raid 5 that much. It prefers a raid 0.
The fun really started when the tried dynamic pages with Perl scrips. There the single processor apache was able to serve about twice as much pages than the 4 CPU IIS server. the numbers:
NT 1 CPU: 30 hits/s
NT 4 CPU: 55 hits/s
Linux 1 CPU: 105 hits/s
Linux 4 CPU 200 to 245 hits/s
Then they added, to simulte database queries a delay of 3 s into the cgi script. The the performance for the NT server dropped from 30 to less than 5 hits/s. For Linux, the performance was linear to the number of processes, starting a 5 hits/s for 1 process going to 80 hits/s for 256 processes instead of the 105 for the version without the delay.
Their conclusion [rough translation]:
For a dedicated webserver with static HTML only, additional CPU are not worth the bother. Even on 2 Fast Ethernet Segments, the increas is only about 20%. CPU power seems not to be the limiting factor. [...]
The realitively bad results for Linux with two network adapter indicate, that the Mindcraft result are plausible and NT and IIS are better performing then their free competition, if one want to play by MIncrafts rules.
[Then some text explaining that 1000 hits/s is about then times the peak value they get on their server and those pages have to be static and cached.]
As our test demonstrate, the Mindcraft results can not be applied to situations, where pages have to be generated dynamically, which is the case on nearly all serious web-sites.
In SMP-mode Linux showed some definite weakenesses. Even kernel developer acknowledge , that Linus still has problems with scalability in SMP settings, specially if the load occurs in kernel modules. However, if the load is as with CGI-scrips, in user modules, Linux profits fully from the additional CPU. There is work done to fix these problems.
For real applications as web-servers, Apache and Linux are already ahead. If the pages cannot be served directly from the cache, the situation is even better for Linux: Here the commercial products from Redmont are not even getting close the open souce projects.
[They finish on telling that Mindcraft was right about the fact, that finding tuning info on Linux and Apache can be hard and do not rival a commercial support infrastructure. But once they got to the developer, they got help fast, whereas it Microsft a week to reply. They also mention that you wont get a kernel personalised for your environment from Microsoft.]
[End of rough translation.]
On the whole the article is a bit too pro-linux. They should at least try to compare similar web-application done as CGI and ASP. This would give some relevant information on how the servers compare for similar tasks. I personally don't care whether I have to write a perl-script or ASPs.
Servus,
johi
PS: Not the article was not available on the web the last time I checked.
It is interesting to see someone actually present some useful information regarding "Benchmarks". Note that the article does not refute the benchmark; it just puts it into a different perspective. It also puts into perspective the difference between a benchmark and reality. If you really think about it, how many sites can possibly get the volume of hits required to run into these numbers? I mean, how many sites are being served over a dedicated T3? Most are coming over load balanced T1s or what have you that are shared amoung numerous other physical boxes. (That is not to say that there are not some sites having such a load.) Even if you had such a site, you probably wouldn't want to rely on a single server to handle the whole thing anyway. (Single point of failure? That's asking for trouble.)
Well, that's my 14 cents worth.
If it works in theory, try something else in practice.
I run a medium sized web site on a p90 with
pathetically slow hard drives and too little
memory. Linux and apache of course... I'd like
to see NT even run anything usefull on a p90!
But that damn p90 sure can peg the T1 it's on
with relative ease.
but I digress, a loss is still a loss, and saying
that linux still easily outruns 99.999% of
anyone's bandwdith is just an attempt to sanitize
the loss, ie. "linux is more than fast enough"
Well that's the same as windoze, "windoze is
good enough" for most people. [sic].
nutsaq