Quantifying "Bandwidth is the Limiter"

John Lazzaro writes "Found this linked from Dave Winers site, an analysis that puts numbers behind the oft quoted "Linux + Apache will saturate any reasonable Internet link given static pages" It basically assumes the mindcraft tests are accurate, and then tells you what it means. Most interesting is the comments about MSs online tech support, the hardware they use, the results of the benchmarks, and the fact that static content is practically irrelevant.

Comments

Missed ONE assumption

[BadlandZ]

Hmm.. Internet server is diffrent than Intranet server. And, in such a case, I think it's also important to go back to some SAMBA studies.

Basically, I think he's right, any reasonable Linux web server can saturate most avaliable bandwidth with _static_ pages.

But, I don't think it's time to settle for that, I think we should go back to look at multithreading and why 4 CPU's gave NT more of a boost, and what that means to what needs to be done in Linux. And, I think that this study also shows that "internet" isn't the problem, so let's look at some faster stuff in the networking world, like SAMBA needs on an "intranet."

You're missing the point

[raistlinne]

If someone tries to convince, say, your boss that you should use NT instead of Linux for your website (say you're hosted by a partial T3 and only serve static web pages) and cites these results, will you say:
A) "You're right. These benchmarks prove that NT is superior to Linux and we'd be foolish to go with Linux now, though if we work real hard coding at Linux et al. linux will be able to beat NT at benchmarks so we can switch then."

or
B) "Actually, if you look at the tests rather than the soundbyte about them, you will see that they prove that a Linux/Apache system will be waiting on our internet connection anyhow, and Linux is a much better value protosition than NT is, remote administration to a box with no mouse or monitor is extremely easy, remote displaying comes standard (no need to buy citrix), you get greater stability, and source code that uses open standards so that if life takes one of its many unpredictable turns and Linux is no longer the best solution, we won't be trapped into expensive solutions that we don't want to be in due to vendor lock-in."

So do we give up, or work with what we've got? I suggest that everyone who's whining about "let's stop whining and get back to coding" at least say what you mean, "Let's just give up and try to beat microsoft on their own game instead of doing what we want." The people who do the coding are still doing the coding. This is about advocacy and marketing. I suggest that the people who do Linux Advocacy/marketing don't stop simply because everything didn't go perfectly. These numbers are only a defeat by microsoft if we let them be.
Remember, there is more than one part to the Linux community, just as there is more than one part to the body. Just as I wouldn't suggest that those who do documentation stop doing documentation and start coding, I also wouldn't suggest that those who do marketing stop and start coding.
It takes all sorts to run this world, and we shouldn't start neglecting any part of it, marketing included. Microsoft is going to try to put as much spin on these tests as they can. If we stand by and do nothing, we'll be as guilty as they are of the spinning. Sins of omission are still sins.

I don't suggest that we do it defensively, though. I suggest that we do it confidently and aggressively. act like we're in control. That's the neat thing about self-confidence. If you act like you know what you're talking about, people tend to believe you. So why should we give up acting like we know what we're talking about.

We've got some test results which prove that Linux can handle the needs of 99% of the world. Why exactly should we hush that up in favor of letting it be thought that Linux is slow.

Real experience

[DLG]

I have been running a linux webserver in various incarnations and machines since 1993. I started on a 486 with some decent scsi (vl-bus) and 8 megs of ram, with a 28.8 hayes modem, matching the isp's.

I generally ran 3.6 kilobytes per second of bandwidth at the time, which wasn't bad for a 100 dollar a month dedicated line.:)

Now adays we use an alpha (for past 2 years) with 128 megs of ram and a T1. One day (and I really mean one 5 minute period) our line was saturated by what seemed to be a webcrawler from taiwan which literally downloaded the entire site. It filled our pipe. That is the only time that such a thing happened and my apache server didn't even blink. I wouldn't have known but I tend to analyze my peak times.

After we decided to upgrade the software on the alpha (redhat 4.2 being a bit old ) I moved the entire site over to a k6-2 running at 300 with 156 megs of ram or something... That hardware cost about 700 all together. (the alpha cost 5000 2 years ago..) It performs fine. I have never seen the apache webserver crash or not respond based on overusage although sometimes dynamic content will slow down (and that is because we don't really optimize for speed). In any case the performance is great for the price. I have bought one copy of redhat 5.0 and have 3 servers all running great. (Finally retired the 486 cause it was hard to find vlbus hardware)...


The basic fact is that the webserver has NEVER been a crisis point or even a decision point. When we went to shttp we went to stronghold as it was the only easy solution at the time. It has never caused us problem.

Further we serve realaudio, handle telnet sessions. handle email, handle pretty much any protocol a client wants. mysql, php, perl, c compilers...

There is literally nothing that we have needed to do that our linux boxes haven't handled. I am so comfortable with them that I have placed linux boxes as controlers for permanent automated exhibits, and have so far only had one hardware crash even.

The notion that NT would be easier for me to maintain, easier for my clients to interact with, or in any way a more efficient use of money is absurd. I don't even know what they charge for NT, but I believe I would rather purchase an extra 256 megs of ram per machine, or a faster processor, then pay to be a beta tester.

I know I am preaching to the converted here, but I started laughing the first time I saw the comparisons. While I WOULD like linux to catch up with SMP and such, and certainly would like to see it scale better, I happen to like being able to buy a 500 dollar computer when I need more processing, and have it up and running in 2 hours (and that because I generally do a redhat ftp instalation)

I betcha you can't do that with a win NT install.

While I do lookforward to an OC-3, I imagine the price of leasing it will be expensive enough that I can afford any number of widdle iddy biddy linux machines to serve up pages. The days of big iron may not be over, but I don't need a mainframe to do something as brain dead as serving static pages.


D

Lack of basic education in benchmarking

[Victor Danilchenko]

While the article is rather interesting in and of itself, I think it points to a bigger issue: in computer industry in general, and in benchmarkig in particular, proper scientific tools and methods are often not used.

Statistics is your friend! If some psychologists used similar methodology for their investigation, they would be laughed at -- I won't even talk about hard sciences.

Yet in benchmarking, the perpetrators completely ignore representativeness of their samples -- this is all benchmarks are really supposed to be, controlled investigation of the performance of a representative sample of real-world computing activities. How can you investigate performance if you don't even try to account for various miscellaneous factors by using proper sample selection?..

What can I say?.. The entire thing disgusts me. I would rant more, but I will simply go and sob in the corner about lack of scientific methodology in my field of choice.

--

TCP slow start?

[tap]

I remember hearing about how linux follows the TCP spec and uses slow start, but NT doesn't.

When a TCP connection is first established, and one machine wants to start sending data to the other, it isn't supposed to start sending as fast as it can. Rather, it's supposed to send only one packet, then wait for an ack, then send two packets, etc. Otherwise you end up with congestion rendering the network useless as you approach full bandwith. Van Jacobson (remember VJ header compression from the days of SLIP?) has a paper on this.

Since the mindcraft test has unlimited bandwidth, no packetloss, no slow modem connections, etc. TCP stacks that don't do slow start properly can send out more data than ones that do.

Not Microsoft's fault

[evilpenguin]

To me this highlights the one true evil force in the world. No, not Microsoft. Ignorance. Dr. Science once said "Ignorance is bliss, and tonight, we're a happy country." I could not agree more.

The basic problem is not Microsoft. Not their products, not their technology (or lack thereof), not their marketing people. The problem is the number of people who do not think critically. The NT benchmark does not lie. It simply tells a very narrow slice of truth and "positions" that truth to show NT and IIS in the best possible light.

To me, the one outrageous thing in Microsoft's benchmark page is the chart that shows total cost of ownership. Now, I'm not a CIO, CEO, or CFO, but it seems the me that cost per transaction per unit time is completely irrelevant. What matters (as the author of the article we are commenting on here points out) is cost per transaction and can you handle your transaction volume?

When decision makers look no deeper than the cooked figures from NT's benchmark, when they fail to see if the scenario represents their business and technical reality, then their business gets what they deserve.

What the "Microsoft Advertising for Linux" article does that is lauditory is it cuts through to a core question. Which is cheaper given a certain use case? It wisely does not answer, but merely points out that in most cases, even in most intranets the Mindcraft/Microsoft scenario is extremely unlikely and that Linux/Apache on even limited hardware will handle most loads anyone would reasonable expect.

It also wisely points out that if you are a site in the tiny fraction that will exceed Linux/Apache's capacity, then by all means use NT/IIS.

Then, one more dig of my own at the TCO figures. Even if we grant the validity of the figure cost per transaction unit time (which I do not), what happens if you set up ten servers, or twenty? Linux costs nothing more for ten servers than it does for one. I haven't the time to see how many servers it would take, but there would come a break even point and then a point where Linux/Apache is cheaper even using the dubious measure in the Microsoft study.

Finally, I just want to congratulate the author of "Microsoft Advertising for Linux?" for showing the value of just trying some of your own math and asking, "Hey, is this reasonable?" If we all did this routinely regarding everything from computer bechmarks to medical scare news stories we would live in a much saner and less stressful world. Whichever operating system you buy.

C'T benchmark: linux beats windows 800% and more..

[dermond]

in the last issue of CT magazine, they also had a linux vs. NT web-server shootout. they used a siemens primergy 870 maschine (4 xeon 450 cpu's, 2GB ram, intel etherpro 100 nic, mylex dac 960 raid controler, price of the maschine: about 100 000 DEM ) here the some of the results:

serving one static html page 4k size: NT and linux almost on par (linux ahead a few %) both systems answer 900 requests/s when hit with 512 concurent client process.

with 8k size static page: linux is between about 5 and 10% ahead of NT.. at 512 client processes the linux maschine serves about 600 requests/s the NT maschine about 550.

using a 4K page but selecting one random page out of 10E4 pages linux has about 830 req/sec and NT about 720. the linux line seems saturated where the NT line is bended down already: linux 15% ahead of NT

random 4K static page out of 10E6 different pages: linux about 270 req/s while NT has never more then about 30 req/s. that means linux is some 800% ahead..

now some dynamic pages. they used plain old CGI scripts with perl. no PHP or ASP. using all 4 CPUs linux answers 210 till 250 request/s while NT is around 60! that means linux is 316% faster

same as above but using only 1 of the CPUs of the maschine: linux around 100 req/sec, NT around 25 req/sec. linux ahead by 300%

if the script contains a sleep(3) at the beginning (to simulate slow database connection or slow client connections) the results are: linux increases the number of requests linear with the number of requesting processes and reaches about 80 req/s for 250 simulated clients. NT is saturated at around 7 req/sec. (in words: seven). linux wins with over 1000%

the only time that NT is ahead of linux (about a factor of 2) is when using 2 NIC cards instead of 1.

my interpretation of all this: i guess there are very few webservers where one would needs more then 100Mbit/s.. and then one would be propably better of with 2 cheaper systems doing load balancing.. given the extra reliabilty, remote managemnt, etc of linux and the better extraordinary better performance in most tests linux is the clear winner. doing fast CGI scripts is by far more importnat then to efficiently support 2 or more 100Mbit cards.. at least for 99.9% of all webservers or more..

greetings from vienna, austria.
der mond.