Slashdot Mirror
Scaling Server Performance
An anonymous reader writes "When Ace's Hardware's article Hitchhiker's Guide to the Mainframe was posted on Slashdot, they got 590,000 hits and over 250,000 page requests during one day. This kind of traffic caused only a 21% average CPU load to their Java-based web server, which is powered by a single 550MHz UltraSparc-II CPU. In their newest article, Scaling Server Performance, Ace's Hardware explains how this was possible."
...how many hits they will get this time ?
Are we supposed to be impressed with a computer that can serve 8 hits and 4 pages per second?
Now if they could use the same technology to create a web surfing client which generates that many hits with the same CPU load. Then you could put them together on the same network, and ....
they got 590,000 hits and over 250,000 page requests during one day. This kind of traffic caused only a 21% average CPU load ... they didn't respond to any of them.
Reliable, Great Value Hosting: $7.95/mo 2.4G/120G
if they used this thing to host a pr0n server, it couldn't get farked! Oh, how this would help in the quest of kitten-killing.
Of course, it is incumbent upon all of us to rush out and try to the link to the article. And some of us to actually read it as opposed to just reading the title.
SLASHDOT THEM AGAIN!!!
I'm on it
When I was benchmarking web servers in *1994*, servers could handle 100,000/hr, which is only about 30/sec. You may need a T3 to handle the bandwidth, but any server that can't handle it today is misconfigured.
Is the /. effect a benchmark test for web sites?, since when?
The package said "Windows XP or better. Pentium Class Processor or better"... So I got a Mac with OS X
seeing as it took Slashdot 35 seconds to serve me up this comments.pl?op-Reply page, yes, i think we are supposed to be impressed.
Cretin - a powerful and flexible CD reencoder
Which is very funny: this is an article explaining how a web site survived the /. effect, thus trying to catch the /. readers back for a second round, and getting lots of advertising hits at the same time. If only that server could keep up. /. I saw a report about a 200Mhz (?) PC running Windows 95 and with about 30 hard disks, that also seemed to do very well under the /. effect.
Now, a while back on
Sig for sale or rent. One previous user. Inquire within.
Can you say dry? 290,000 whatever, 21% something, 550 MHz, blah blah blah....
I'm not a computer, I'm a person. Give me graphs or don't bother.
Karma: Good (despite my invention of the Karma: sig)
Are we supposed to be impressed with a computer that can serve 8 [actually 11] hits and 4 pages per second?
If that's what a slashdotting consists of, then probably yes.
There are plenty of Web server machines that rarely crash, if ever. Many of these sites rely on the work on only one machine, just like Aces Hardware. If they have more than one Web machine, they split each up for each category (e.g. Google has a machine each for their "catalogs", "search", and "images" utilities)
Academic: MIT.edu, Stanford.edu, Maryland.edu
Business: Amazon.com, CDNow.com, Slashdot.com, Google.com
Pleasure: TheHun.net, Playboy.com, Napster.com
Reply or e-mail; don't vaguely moderate. Ex-O'Reilly/MIT employee, now a full-time Google employee.
I would be more interested in stats on a webserver that took a puke. It would be interesting to see what started the dominos falling and what ultimatly brought it down. It would be as good a learning experiance as this article is.
...so I could implement it on my PHProjekt server! I have become so dependent upon it that it's almost scary. I'd love to speed it up a bit though. Time to start reading I suppose.
Not to be cynical, but serving (nearly) static pages shouldn't be a huge load by any standard. Even with dynamic (fully dynamic) pages, 250,000 isn't a huge number.
As an example, I run a pretty popular site that pumps out about 250,000 as well, all CGI-created and database fed pages. This is being served by two 1ghz web heads and a 1ghz db server. Granted that those three machine run at 100% load during peak hours, it's still not a huge deal (this is because I haven't finished the local caching mechanism yet). Did I mention that the two webservers also toss 1 million images a day too?
Of course, I don't wan to belittle the article that much -- If anything, it shows the preformance gains one gets when you use efficient hardware (I have no doubt that their 550 mhz Ultrasparc II has nearly the same horsepower as a 1 ghz x86) and efficient caching (caching data in RAM and serving from there, avoiding disk access penalties, is a huge performance increase).
Hilary Rosen's speech was about her love of money and her desire to roll around naked in a pile of money.
I agree with you - Garth Brooks is an amazing writer.
I think the best actor is Martin Lawrence in The Great Escape.
( I think you meant Frederick P. Brooks - but perhaps you were in jest and I'm retarded...)
There are some odd things afoot now, in the Villa Straylight.
/. is down or slow probably 5 or 6 hours out of the day lately. I don't think so.
I don't need no instructions to know how to rock!!!!
Post an article about surving a Slashdot attack only to get slashdotted the second time around.
OTOH, my puny little SDSL connection was seriously maxed out.
Even old hardware can happily serve up hundreds of documents a second, if the pages are static.
>Garth Brooks covers this in his famous book "The Mythical Man Month" where he proves in a controled lab environment that Java under X86 runs on the order of Olog(n) slower than it does on a RISC chip like an UltraSparc.
WTF? Fred Brooks wrote this book, and I don't seem to remember RISC or UltraSparc chips, not to mention Java, in 1974. Garth Brooks is (AFAIK) a country music singer. Try again.
Sig for sale or rent. One previous user. Inquire within.
I never really thought that the problem lied with the server's hardware, but in the bandwidth to the host. Shouldn't an article be written about how to conserve bandwidth during a slashdot effect? Even older servers should be able to handle 100 requests per second. I think most FPS's are alot more taxing than that.
doesn't the faq still have the server info. A bunch of debian boxes and a couple of monkeys...
Ah, but this is a *java* webserver with okay performance. And *that's* quite a big deal :)
(This space left blank for Java evangelists to
rage in).
Whence? Hence. Whither? Thither.
Where Performance is Concerned, Optimization is Key
When our Hitchhiker's Guide to the Mainframe article caught the attention of Slashdot last month, quite a few people were curious to know about how our server handled the traffic. This is a topic we have discussed previously in Building a Better Webserver in the 21st Century and SPECmine - A Case Study on Optimization, but since there was so much interest in some more in-depth information on the topic, I decided to spend some time explaining how the data object caching in our web application can do so much for performance without sacrificing the ability to serve true dynamic content. I'll start with some statistics gathered on December 4th.
Our traffic for the day totaled over 590,000 requests (hits), with over 250,000 of those being requests for pages. Requests peaked at 677 per minute, which of course includes everything (images, pages, files, etc.). The peak number strictly for article pages was 235 per minute. Perhaps the most impressive statistic is that during these peaks, our web application running in Java (including the HTTP server) was only consuming 20% of available CPU time, and all article requests were served in 4 milliseconds or less. Furthermore, our database was only at 7.5% CPU utilization. So, when the system was serving roughly 11 requests per second, the CPU was nearly 75% idle. Let's take a look at the traffic for the day, graphed on a per-minute basis:
In the graph above, we have two different sets of data. The first is requests, which is essentially anything requested from the server -- images, dynamic pages, static pages, etc. The second is article pages, like this one. As you can see, the initial spike in traffic from Slashdot occurs around 1:30 AM. There is other dynamic content, such as the front page or the message board, that is not accounted for in this data, but nevertheless, the graph gives you a good idea of the ratio between article pages (text) and everything else. Traditionally, dynamic content is often one of the most intensive types of content for a webserver, but as you will see in the next graph, that doesn't always have to be the case.
In this graph we see the CPU utilization of both the Java web application and the database, sampled each minute, relative to the article requests per minute. Here you can clearly see the peaks I mentioned earlier happening roughly between 9:30 and 10:00 AM. You can also see that the average CPU utilization of both the web application and the database is rather low: 14% for Java and 6.9% for the database. The Java CPU utilization peaks at 51% for a very short time early on, as it is caching content on demand as traffic begins to spike. You can also see occasional spikes in the database CPU utilization, as cached data is periodically updated. When you consider that there are servers out there that will fall over and die when faced with this kind of traffic, an overall average CPU utilization of 21% for a modest 550 MHz uniprocessor machine is not too shabby.
Advertisement:
Overview
Before we begin, let's take a quick look at what's covered in this review:
* Page 1: Introduction
* Page 2: Scaling with Larger Workloads Effectively
* Page 3: Inside the Web Server Application
* Page 4: Benchmarking with ApacheBench
* Page 5: Benchmarking Web Server Scalability with AutoBench
* Page 6: Static HTTP Performance
* Page 7: Dynamic Cached vs Uncached Article Performance and Conclusion
Let's start by considering some of the bottlenecks present in many web servers and how to manage them...
Maybe the /. team needs to study this article and learn a few things.
How is it that one careless match can start a forest fire, but it takes a whole box to start a campfire?
Sorry, you lost me at Garth Brooks...
You seem to have accomplished your mission.
Lasers Controlled Games!
I think that the main thrust of the article was essentially that size doesn't matter -- it's how you use what you've got.
One thing that sort of made me think though, was the focus on being able to deliver massive numbers of "pages" and "hits". For most sites, this is not an issue -- their bandwidth would be hosed before the server would be. You can only stuff eight great tomatoes into that itty-bitty can. It doesn't take much to saturate a T-1.
If you have nearly unlimited bandwidth, then these server-tuning issues start to become important. I think it is a good idea to focus on how applications are built and used when thinking about performance of servers. Too often, the sole focus is "can I do task X" and not "what is the most-efficient way I can do task X".
A nifty article, all in all.
GF.
Lots of petrified grits
"an overall average CPU utilization of 21% for a modest 550 MHz uniprocessor machine is not too shabby."
Firstly, when said CPU is an UltraSparc II, then 550Mhz is anything but modest. Secondly, I would not expect the CPU to be busy during a slashdotting; it would be hanging around waiting for the disk drives and network card to come up with something useful.
Gateway Timeout
The following error occurred:
Server unreachable
----
Please contact the administrator
I read Slashdot for the
It's just a shame that I can't access either of those articles now!
You should know that your use of the word "farked" and your reference to the inane "kitten killing" meme reveal that you are a moron of the lowest grade. It's especially bad since you're not on fark, and we have our own verb for that here.
Please go back to fark.com, where you can post your mindless drivel with impunity. Here there be bastards.
--
the strongest word is still the word "free"
"The page cannot be displayed".
The lesson here is: put your money where your mouth is and you may end up eating it.
Looks like they are slashdotted this time!!!
Lots of people could use this type performance. I only had a chance to use JSP on one project, a while back. Tomcat was notoriously difficult to install back then. But when it was up, the difference between JSP application server and PHP become apparent. Application servers can make quite the difference.
Just having an application scope for variables saved us a trip the the ldap server per request. PostNUKE, squirellmail, and lots of other large PHP apps could be sped up drastically if some of those features were available in the PHP engine.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
i clicked the URL to read the article, but it took 5 seconds to load the page and the images still haven't rendered. oops.
I prefer reading "The Mythical Man Month" in the original latin. Brooks' translation leaves much to be desired.
I have to agree with the above comments. I'm not impressed at all. 11 requests/sec is *nothing* to brag about.
Currently *one* of our servers is serving up 114 request / sec (and these are dynamic page requests -- all requiring extensive DB interaction). The entire system is serving well over 3,000 requests / sec.
We achieved this level of performance by using the right tools for the job, always. (OT Rant:) **and by hacking**. hacking hardware and software. Where have all the good hackers gone, anyway?
Today I see more and more people willing to shell out $$$ to companies for expensive *solutions*. arrgh.
(/OT Rant)
For the record - our setup:
22 cheap lite weight servers managed by LVS + Heartbeat + Mon. 5 MySQL DB servers (using replication) and C / PHP / JAVA. (all extensively hacked)
this is upreme irony. your server gets /. ed becasue of your thorough and informative look at the scalability of your servers.
250,000 hits/day / 24hours / 3600 secs per hour = 2.89 hits per second. Is this really that impressive. I've heard about other web servers handling thousands of pages per second.
Of course, there are more complex applications where data caching can be implementing, such as discussion forums where multiple users can be adding, editing, and deleting messages simultaneously. But that's a topic for another article.
Most of the applications I write involve updating data almost as often as fetching it from the database. In an environment like Apache where you have individual processes serving content (and database connections are process-centric), implementing caches that are updatable becomes a very complex excercise, without implementing an additional layer.
eToys used a b-tree (Sleepycat?) database layer situated in front of the database layer - they would store objects in the b-tree, and fetch them from there if they had not expired. Once cache amongst all the servers made this worth doing; a Java web server can do something similar, since the objects are stored in memory shared between the various serving threads. The end result is similar to what Ace's Hardware has done.
What have other people done? Since I use Apache, I'm leaning towards a disk-based caching system.
...caching what are essentially static pages stored in the database to eliminate duplicate database queries dramatically increases performance...film at 11. *yawn*...call me when they start having to deal with user interaction.
Do they think this is revolutionary or even noteworthy? Anyone who builds dynamic websites of any size does these things!
We didn't get them the first time, but that's okay. Just buck up and let's give them another shot! If they don't go down the first time, just keep hitting 'em until they do go down. You've got to give it a full 150%, you hear me? Now get out there and click some links!
/. go!
Go team
A deep unwavering belief is a sure sign you're missing something...
I hope they share the information on how and why their server trashed this time one of their stories appeared on slashdot. That would give us both success and failure, excellent edutainment!
--- Sigmentation Fault - Comments Dumped
Many pages get "slashdotted" but obvously slashdot itself *usually* is not affected by this! Maybe they should follow slashdot's lead, yeah?
Reloading their page a couple times (2nd page of the article, not the one slashdot linked to), I'm getting occasional 503 errors, and the rest are taking a very long time to load. Usually the page comes up with some "broken" images that didn't load.
At the bottom of each page, there's a number that seems to indicate the time they believe their server spent serving the page. Usually is says something like "2 ms" or "3 ms"... That may be how long their code spent creating the html, but the real world performance I see (via a 1.5 Mbit/sec DSL line) is many seconds for the html and many more for the images, some of which never show up, and sometimes a 503 error instead of anything useful at all.
So, Brian, if you're reading this comment (which will probably be worthy of "redundant" moderation by the time I hit the Submit button)... it ain't workin' as well as you think. Maybe the next article will be an explaination of what went wrong this time, and you can try again???
PJRC: Electronic Projects, 8051 Microcontroller Tools
And that's why you're already slashdotted???
dont neglect the gaggle of cracked out commando geese
/me hopes it will fall this time ;))
let's /. :)
our web application running in Java (including the HTTP server) was only consuming 20% of available CPU time, and all article requests were served in 4 milliseconds or less. Furthermore, our database was only at 7.5% CPU utilization. So, when the system was serving roughly 11 requests per second, the CPU was nearly 75% idle.
According to my calculations it would be closer to 72.5% idle. Of course for you stat heads you always gotta allow for 3% give or take, so it works out for you, but come on, if you're going to post some stats, why not be as accurate as possible?
Maybe I'm just crabby today I don't know.
You're nothing; like me.
can we say irony?
RC
Tom's is bragging about their speed at serving "dynamic" content. The only dynamic content I see is the advertisements. If you have data that doesn't change for 10 or 15 minutes, of course you can cache it all on the web server side.
Try to apply that to a site where content has to change constantly (DYNAMIC content, one would say), and see how caching speeds things up.
only a 21% average
:)
Aaaah, the one that got away
Live for the present, learn from the past, and dream of the future!
I would paste the contents of the page in this post, because their page is slashdotted at the moment...I think that would be a bit ironic, though. ;)
"Quoting famous computer scientists out of context is the root of all evil (or at least most of it) in programming." - K
Scaling Server Performance my foot. They post an article about how they survived a Slashdotting and now the article is Slashdotted.
It is not off line yet but it was not ready for the slashdotting. That box is slow as helllllll...
Got Code?
until Ace's Hardware gets sued by Ace Hardware for trademark infringement?
We have OLD Cobalt Raq3's (300 MHz AMD K6, 128 MB Ram, single IDE drive) running the latest Cobalt OS, and we JUST had one of these boxes get hammered this week; in a 12 hour period, it handled 625,000 hits (mostly CGI's, but it had a reasonable amount of static content), and at the same time handled 35,000 POP requests, sent 4,500 emails, and did some other random functions (and things like hostname lookups are enabled for weblogs, FTP uploads are happening for weather-wite webcams that were associated with the heavy traffic, etc, so there's obviously not a huge amount of "tune it till it's ONLY gonna do one thing" going on here). Now, the box was taking a whipping compared to it's normal load, but c'mon. I can't say the "Poor little 550 MHz UltraSPARC story" makes me tear up :-)
Pretty much any server can serve hundreds, or even thousands of pages per second (I benchmark a basic PC IIS 5 server serving 17,000+ pages per second), but the problem comes with people turn what should be a static page into a dynamic page. The thought process usually goes like this "I'd like my news to all be dynamically driven so that I don't have to modify my files, but rather just add a new document as a CGI/ISAPI/Script served dynamic page!". This can bring even the largest server to its knees. Another poster mentioned GZIPping, and again this relates to dynamic content: If you have static content any half-decent compression module will cache the gzipped output until the source file changes. The solution, of course, is to find the perfect balance between them: Dynamic content that is statically cached for interval or on-demand periods, allowing your server to effectively server static content to the majority of hits while expensive page regenerations are minimized. Voila- Easy versatility and programmable functionality, but without the massive server requirements.
I say this as a interested party as I design and implemented, and sell, a interval dynamic caching and automatic compression system.
Tuesday, November 27, 2001 8:07 AM EST
------
It was published over a year ago, and undoubtedly was based on their spring/summer 2001 trials. Even then this info wasn't revolutionary, and is even less so now.
creation science book
Oh yeah, now it's burning. Burn baby burn.
... they deserve another taste of slashdot.
Seriously though, when we were testing our server components it happily handled 200 requests for dynamic content per second (not static images or HTML) without flinching.
So I guess this is special because they used the wrong tool (java) to get a mediocre performance, woohoo!
I think they are just publicity whores looking for a bit more limelight.
...they used this brand new technique called caching to improve throughput.
Gawd, there are tons of papers about Web Caching and database caching. There are papers comparing the caches. This is hardly new or exciting.
Heirachical Internet Object Cache. Danzig et al.
World Wide Web Caching: Trends and Techniques. Barish et al
High-Performance Memory-Based Web Servers: Kernel and User-Space Performance. Joubert et al.
A Scalable System for Consistently Caching Dynamic Web Data. Challenger et al.
Holy s-, it's Jesus!
The page cannot be displayed
/. r00lz!
There is a problem with the page you are trying to reach and it cannot be displayed.
</I>
Ummm did tey get their article shlashdotted? That is what you get when you brag!
On the more serious note i find it very interesting that the other article gave an average load of 21% whereas now they are somewhere in oblivion. Could this have anything to do with the timr that the article is posted? (IE: If they post it 9:00 PST or when majority os slahdotters are online and the first article they saw was this one?)
Live for the present, learn from the past, and dream of the future!
They've really simply discovered that dynamically generating essentially static content is a bad idea : the 'dynamic' pages they are talking of are just articles which once written stay the same, and so are serving identical pages to each user.
Using scripting with database look ups to create such pages is obviously not good - much better is to compile your data in to static pages and serve those. I have done this for my own website using XSLT to generate the html pages with consistant links and menu's etc. - but you do have to remember to re-build it after making any changes or adding new content (I use gnu make to handle the dependancies of one page upon another so it doesn't rebuild the entire site everytime.)
They've taken the alternative approach of still using a database for the requests, but then caching future requests for the same page-id's, which has the advantage of being compatible with their original dynamic generation system, but they don't mention how they handle the dependancy / cascading alterations problem if they change the content (though they could always flush the entire cache of course....).
Neither of these approaches can help you though if you have real dynamic pages where every request is unique or there are are too many possible pages for caching to be feasible (for example amazon or google).
Can't get the page boasting about their server's performance during high traffic loads to pull up. Ironic or moronic?
slashdot, news for crazed liberal socialist zealots
I remember reading that original article, and yes, I was impressed at the responsiveness of the server. But before they are congratulated so much, consider this. The original story was posted on slashdot at 1AM.. so the initial spike of activity resulting from the linking being in the top few on Slashdot was directly proportional to the number of people on Slashdot at the time. As you can see from their graphs (if they're showing up for you) that traffic spiked, then continued on during the day.
This time around, the link got posted at 2PM not 1AM, and so far as I can see, they handle this flurry of hits much less gracefully than the previous ones! There are a lot more people online at 2PM than 1AM (all arguments of nocturnal nighthawks and people in other time zones aside).
Just bringing my 2 cents - it's been real slow lately for me as well. I was thinking the first-posters must be gutted, waiting madly for the bloody comment box to load for a whole 2 minutes :-P
Daniel
Carpe Diem
By 1994 servers were capable of handing quite a bit more than 30/sec. Back in the old SunSITE days, my experimental Multithreaded Daemon For Multimedia Access (MDMA) could handle over 50/sec (exponentially distributed) even on an IPX running 2.3, and Netscape's server with pre-forking could easily cope with high sustained loads on appropriate hardware.
Simon
Looks like they're gone completely -- All I'm getting are can't find server or DNS errors now. Looks like the web server is completely offline.
Quote from article:
Our traffic for the day totaled over 590,000 requests (hits), with over 250,000 of those being requests for pages.
Since when does a "hit" not count as a "page request"? When I go to http://slashdot.org/, that is both a hit and a page request (for index.pl).
Just wondering.
I am reading the article but am having lots of trouble going from one page to the next (several minutes of time outs). I guess that just a little ironic. Are people trying to give them problems this time?
for the "slashdot effect"....if nothing else, this article gives me a good idea of number hits / time period. My little domain's web server runs on a 70MHz sparc 5, and I wondered what it could take if only serving *static* pages (which most of mine are)........so if I put an article on there with a 100k or less page and submitted to slashdot...hmmmmmm, maybe
Even for dynamic content, it seems any reasonable web server should easily be able to generate half a dozen pages per second. Of course, it won't be able to if you do something stupid like put all your content into a database.
Seriously ... the numbers aren't that great. I used to admin a DEC Alpha Digital Unix server running at a whopping 300Mhz and it routinely served over 1.5M hits per day along with email, authentication and accounting for over 5,000 people and we rarely if ever saw it over a 0.5 load average. This was 4 years ago.
It's not apples to apples, since we weren't serving the same set of pages (we had around 500 personal homepages, each with a varied combination of static HTML, images and CGI programs) but honestly, if the numbers in this article are supposed to be impressive, we've grown too accustomed to web server feature bloat.
It is more productive to voice thoughtful opinions (reply) than to judge (moderate) others.
Queuing approaches have proven to be much more scalable in other areas - no reason to think it wouldn't work for web servers. Check out SEDA: An Architecture for Highly Concurrent Server Applications for a working implementation in Java that outperformed Apache [insert benchmark caveat here].
More on event-driven servers that minimize data copies and context-switching here.
If you have a really really boring story, it doesn't even matter if you put it on slashdot; your webserver still won't die.
Some static story pages? Who cares?
It all depends if you are actually doing something of interest.
Like the comments in Slashcode, most apps go from static, to dynamic, to static caching of dynamic pages.
At DTN we served up customized portal pages to people with commodity and equity quotes, news, graphs, etc. Since they didn't have any money we had to use a load balanced Pentium Pro and a Pentium II. The app had no problem serving the load, and it was fast.
Now that I work for companies that have money, our apps run really slow. Developers get expensive machines and don't know how to optimize any more.
I've complained about a *major* degradation in performance since the move but, all I get is modded to -1. Seems that it is unacceptable to question the Slashdot admins.
The page cannot be displayed
The page you are looking for is currently unavailable. The Web site might be experiencing technical difficulties, or you may need to adjust your browser settings.
YHBT. YHL. HAND.
Loved your work on the soap opera. Love your album, too! I listen to it every day!
Do me a favor, fax me a copy of your signature, so I can tape it to the album and sell pirates of it and photocopies of the cover on Ebay as "signed collectibles" okay?
Thanks!
ta!
Maeryk
Feminine Protection? What is that? A chartreuse flame thrower?
though this one would be alot harder to beat :-P
Notice that on their site the graph indicates that the page was linked on Slashdot at around 1:30 AM. Their actual peak traffic wasn't until 8 hours later. So instead of being truly hammered by the /. effect, they were able to watch it trail off through the day, probably with a lower number of hits than they would have had if, say, the referring article were posted mid-day. I suspect that if the refering link was posted in /.'s front page during a peak traffic time things would have been far heavier. Basically, as far as Slashdotting is concerned, this was probably a best-case scenario. How many people actually scroll down more than a fold's worth when reading /.? They probably missed out on a huge Slashdotting because of that. Lucky them.
SIG: HUP
Well I can't read the article because it has been slashdoted, but I wanted to bring up a point about generated pages versus static pages.
Slashdot is db driven, but I believe that a static page is generated on an interval to deliver better performance. I'd like to see how
well any site could perform if they regenerated every page each and everytime.
I develop and support and large internal application that is highly database driven, in some cases we can cahce queries, but we always have to generate the page on the fly, just by the nature of the system.
In this case the ways to improve performace is to cut back on the html outputed ( sometimes by limiting views of data ), have a cluster of servers to reduce requests per box ), and then optimize the crap out of every db request.
On a side note it would be interesting to see how mySQL and Postgres compare to Oracle, DB2, and SQL Server under real world high stress sitatuions, my gut as a DB is that they wouldn't come anywhere close.
I like Anadtech, but I just feel this is them touting themselves and really has no impact on most web developers. When you are serving mainly static content without any kind of real logic ( meaning you have to query a database, scroll though results, add numbers, determine who the user is, and generate all kinds of thing dynamically ) you don't really have any room to brag.
Plus, why the hell are they using one ultra expensive Ultra sparc machine. You could get one box doing load balancing, get a bunch of cheap 1 ghz pIII boxes and smoke you ultra sparc.
I can't get the article to load past the second page, but it sounds like they've just got their web server configured to cache queries to the database so that it can respond without hitting the disk.
.3-.8.
;)
Sounds like the same thing MySQL 4.x does. I'm running a site that generates all the content on the fly from a MySQL database, and the upgrade from 3.23.54 to 4.0.9 resulted in a situation where half of the queries that come in are anwered by the query cache, and I saw my server utilization drop from 1.5-2.5 to
Significant and worth doing, but I don't know if their solution is all that special. Just upgrade MySQL.
screenshot
He says you should avoid tying up database connections in processes that aren't using them. With mod_perl we do this by using a reverse proxy. You could do the same with PHP. He also says you should cache. Well, duh. It just seems odd how he puts this in terms of "Java saved us" when in fact these techniques are universal and any experienced developer would be using them by now.
Comment removed based on user account deletion
the only thing that bothered me about it was that they didn't bother to touch on any of the methods for speeding up access to the database when the queries actually need to be run every time, such as when the ResultSet isn't so static. Perhaps a method that was relatively inexpensive for determining if a resultset had changed since it was cached. I'm impressed with the stats, I'd really like to see how linux/sparc would compare to solaris on the same machine.
Of course, it won't be able to if you do something stupid like put all your content into a database.
How is a developer supposed to design a Slash/Scoop/Nuke style board without storing the comments in a database?
Will I retire or break 10K?
lol... /. effect.
I love the
That's great that they handled being /.'d that well... but I can't reach them now...
Back to the drawing board.
"The operation timed out when attempting to contact www.aceshardware.com"
This space intentionally left blank.
Multithreaded Daemon For Multimedia Access (MDMA)
First of all, did your "MDMA" web server support dynamic content at those rates?
In addition, I always thought MDMA stood for 3,4 Methylenedioxy-methamphetamine, or "Ecstasy" for short. Was ecstasy around back in 1994 when you named your server?
Will I retire or break 10K?
but we are back, stronger than ever!
It's nice to see a article like that, just what I was looking for
my sig
They must have lousy bandwidth then. I can't get the page to ever finish loading. Just getting the initial HTML into my browser seems to take forever. I'm not impressed at all with their Java web server. I can serve pages faster with my NeXT Cube (M68040 CPU).
Ace's Hardware needs to research real servers before talking about their "scalable" servers. Their numbers are really saying that their box performs like a dog.
For those of you interested in this topic here is a few pointers and words of wisdom.
Server scalabilty and performance has three basic metrics, thruput (urls/sec), simultaneous connections, and performance while overloaded. Of course, you could add latensy but I'd argue that with the correct design latency is directly proportional to the real work you are doing, bad design insertes arbitrary waits.
I know of a HTTP Proxy by a large ISP that does user authentications & URL authorization (re: database), header manipulation, and on-the-fly text compression at 3000 urls/sec for 2000-4000 simultaneous connections and maintains that performance under load by sheding connections, all this on a dual 1GHz Intel PIII box running a Open Source OS that starts with "L". That is a maximum of 260 Million URL/day, three orders of magnitude greater performance than Ace's Hardware stats.
The simple answer to the question "How do I create a scalable fast network server?" is Event-driven GOOD & Threads BAD. Event driven network communication is two to three orders of magnitude better performing than thread/thread-pool based network communications. See Dan Kegel's C10K web page. That means you must use non-blocking IO to client sockets and databases. Once you accomplish that small feat, dynamic content just consumes CPU; with 2.8 Ghz Xeon processors you have plenty of cycles for parsing HTML markup or whatever. Threads cause cache thrashing, and context switching. While thread programmers don't see the cost in their code, just read the kernel code and you'll see how much work HAS TO BE DONE to switch threads. Event driven programming just takes some state lookups (array manipulation) and a callback (push some pointers onto the stack and jump to a function pointer).
Desgin is FAR MORE IMPORTANT than which runtime you use (execution tree, byte code, or straight assembly). I have done some very high load network programming with Perl using POE.
Python has Twisted Python
Java has the java.nio and the brilliant event/thread hybrid library SEDA by Matt Welsch.
I am also looking into the programming language Erlang which builds concurrancy and event driven programming into the language. Further, Erlang is used by some big telco manufacturers to great effect (high performance and claimed 99.9999999% nine-nines reliability on a big app).
-- I am not a fanatic, I am a true believer.
What I find much more interesting than this article is the way Slashdot handles the massive load. I might be a little off, but I believe Slashdot essentially has the main page updated every minute (five minutes?), so if you just load the main page, you're getting a static document, which is *much* faster.
I've always thought more sites should do this. Why not have the pages you can get away with be static (updated every couple minutes for a 'real-time' feel), and only have the pages that need to be dynamic be generated on the fly? I was playing with ab (the Apache benchmark tool) on one of my computers, and I couldn't believe the difference -- loading a static page, I got something like 100,000 hits (I don't remember the time period); PHP got about 5,000 (unknown, but same as previous, time period). My numbers could be off, but assuming they're not, it would be 20x more effective to have the page generated every few minutes and saved as a static page, at least for high traffic sites. (For low traffic sites, this could probably consume *more* resources...)
________________________________________________
suwain_2
You could read in your comment file everytime and then push out a new comment html file each time a comment is made. That woudl be static, but that would be a pain in the ass to write when databases alread exists, and it won't scale as well as a database either.
They're showing how to make DYNAMIC content scale. Who cares if you can read from disk faster than you can push it down a pipe, that's just a different version of NFS. What they're talking about is how to make somthign worthwile scale, such as say how ebay makes their dynamic auction information scale by caching the listings and keeping the actual auction pages updated realtime. Dynamic is much more powerful and interesing and thus is a good topic for examniation and discussion, i.e. "Stuff that matters."
138974 ms
A little over their 4 ms goal. Specifically, 138,970ms.
________________________________________________
suwain_2
Allmusic.com runs off a single box using M$ and an ISAPI DLL to deal out the pages against a Foxpro database. We have the largest implementation of Foxpro known, lol. 2 mil ain't bad for an M$ setup. In fact, it's probably some sort of record!
Basically, the whole article has 1 message: you should cache stuff. I couldn't agree more. Why doing a database request every time a page is hit? Even if you're going to show the same information say 1000 times? By combining dynamic and static elements, the "server load" part of the slashdot effect can be eliminated, I think slashdot also does this, but differently.
Obviously, if you don't have enough bandwidth, you are screwed anyway, but usually it's the server load that is the problem.
MfG shurdeek
It's friday :-)
They really don't want to post an article like this one on a friday !
Ceci n'est pas une Signature !
As a bunch of people have pointed out, it is unlikely that the /. effect is a matter of "crashing" servers. It is much more likely that most of the "slashdotted" sites on the front page on a given day involve a server which is doing just fine and a bandwidth pipe which is seriously about to burst.
You can saturate most any small-business-affordable pipe with a Pentium classic machine as a Web server. Or to put it another way, there's no point sticking a dual-P4-Xeon Web server with 4GB memory and a RAID-5 on a DSL line.
The computer I'm using right now (a PIII system) could run Apache very nicely in the background and would likely survive quite a hitrate without too much trouble. But if even just a few thousand people were to hit it all at once, there would be a traffic jam, some people wouldn't get served, and the ISP would probably close me down, because I'm only sitting on a 256k pipe.
STOP . AMERICA . NOW
...Too bad the server keeps timing out. (No, really!)
Whatever you can say about them being slashdotted, they are apparently squeezing the max out of their box, the site went down while they did some tuning and it's now back online; according to the article, they were allocating only 1GB out of 2 available for their current needs; probably that they had to use the whole 2GB to survive /. So, kudos to them, they survive at peak time, with a few links from slashdot frontpage. Time to finish reading the NSF (not so f..) article!
have you been defaced today?
You intend to get that number up to 105%?
The best attack is thttpd's bandwidth throttling.. I have seen thttpd take a sound pounding serving pages and it heppily throttled back everyone to a dull roar. BUT serving a nice steady 30 pages a second is nothing... when you get 90,000 requests a second in bursts, espically when the story first hits the front page...
Nothing but a gigabit ethernet connection can even come close to handling that.. and last time I checked a T-1000 line was not an option on internet-1
Do not look at laser with remaining good eye.
It the images are not loading (reliably), the pages come out slow, but the generation time (bottom left) has never been over 3ms. Perhaps its a bandwidth problem they have.
lol. +5, so true though.
1q2w3e4r5t6y7u8i9o0pqawsedrftgthyjukilo;p'azsxdcf
- DotNet Framework Runtime EULA
and then you woke up.
Geeze, I actually responded to a slashdotting troll. What a waste.
Friends don't help friends install M$ junk.
This story is dopey. If you have a web server and it is hitting a CPU bottleneck, you have done something wrong.
Ok, if the server actively plays chess against a hundred people, I'll let you be cpu bound.
Brian sent me the pdf in the mail, though -- so don't be afraid to buy it :-)
They said that the peak load was 11 hits per second, with 4 pages being served. They also said that their CPU was 21% loaded to serve this much traffic.
/. readers to hit them all at once, or perhaps they need a bigger pipe connecting them to the Net.
This says nothing about what they can serve under ideal conditions; this is what they actually served up during an actual slashdotting. If you want to max out their server, you will need to get more
Read the article; on ApacheBench with one particular page they tested, the server tested out at five dozen pages served up per second.
I don't know about you, but I was somewhat impressed by all this. A $1000 Sun does seem to have been a wise choice for them.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
But what would the drives after number 24 (Z) be called?
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
I hate to do this, and get into some kind of "look at my l33t skills" type thing...but seriously, those numbers are just nothing to be impressed with. As several people have pointed out, usually the limitation on a well configured server is the bandwidth available. I have a buddy who runs a few adult sites, and I go ahead and keep his machines updated, optimized, etc, etc. On one web server alone, with simply rebuild Apache with a higher HSL and streamlining only essential services this *one* server is handling an average of 16,000,000 hits per day. (avg approx. 16,000,000 hits, 5,000,000 pageviews, 450,000 unique visitors per day). In fact, only last month did we set up a separate database server in anticipation of him getting even more traffic (I wanted to separate the web server from the db server esp. if we were gonna move to load balancing)...even still the cpu load was consistently low and the site was/is serving dynamically generated content (php) and is all driven by a mysql content management system. I have yet to even max out the usage of the server and do some ulimit type stuff or hard adjustments via kernel changes.... so what is the big deal about this article. I think it would be good to put up an article about how to optimize your web servers both in layout and actual configurations to allow for Slashdot levels of traffic. I doubt this will happen, just as the mirroring content on featured stories to help ease bandwidth or other similar suggestions. The saddest part is that once you spend the time to really optimize a machine or machines...it takes far less time to maintain them.
After what we've done to their server, Looks like the Sun is a mass of incandescent gas, after all :)
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
This is why people should set an expiration time on their static content. If, for example, I set up the images to expire one hour from the access time, multiple visits to the page (and images shared between multiple pages) would only be requested once. An ISP's proxy servers down the chain would only help in this regard.
In addition, for static content, "LastModified" is easy to compute. Clients can request a page, send an "If-Modified-Since" header with the timestamp of the static item, and if the item hasn't changed, return a 304 response and no data.
The same can be done for dynamic content, but it requires a bit more work. Most web servers do these things for static content out of the box.
As was said in the article, the fastest request is the request that never has to be made.
- I don't need to go outside, my CRT tan'll do me just fine.
As for heavily threaded apps, no leading Java servlet container/web server uses a 1:1 model of request to threads: not Tomcat, Resin, Weblogic, WebSphere, nor Jetty.
Too late. It's already been done for months now. As JDK 1.4 comes into greater use, you will see more and more of these offerings using non-blocking I/O (read: event-driven) as the default HTTP handling method.
- I don't need to go outside, my CRT tan'll do me just fine.
If memory serves, Slashdot has a small farm of boxes -- multiple boxes dedicated to Slash and a couple dedicated to MySQL.
Ace's Hardware runs on a single box whose hardware is slower than any one of the Slashdot boxes.
- I don't need to go outside, my CRT tan'll do me just fine.
Most sites aren't CPU bound; They're disk I/O or memory bound. Once you exhaust RAM, you swap to disk, and performance drops precipitously. Once the database must read/write to more sectors than can fit in RAM, page faults ensue, and performance drops precipitously.
Their solution not only saves CPU, but I/O workload as well.
- I don't need to go outside, my CRT tan'll do me just fine.
The article pretty much said two things:
-- Caching objects saves a database hit and makes things fast.
-- Resin scales better than Apache.
That's great and everything, but it really doesn't help anyone else. Ok, so now I want to apply this object caching to my own application. Where does this cache live? If I'm not running Resin, then I guess every apache process has one. How do I handle dirty objects which need to be written back to the DB? What if they have been dirtied in two different processes? If I am using some sort of service external to the web application to do the caching, how fast is that? Faster than the database? Perhaps, but now it has to scale too, and it STILL has to consult the database, only for writes, which are worse than reads.
This happened to work for their application, but in order to be applied more generally, it needs lots more explanation.
Any object persistence mechanism which is smart enough to handle caching in a read-write system with any level of configurability is going to be a large, complicated piece of software itself, and will have its own issues to bring to the table.
Comment removed based on user account deletion
I have read both articles from Ace's Hardware about how they built their "killer" web server. And in all that talk about the server and the applications that run on it they make no mention of what they did to the OS and the machine itself other than putting faster drives in it. They show an Ultra 30 as a server running a GUI, if I wanted "killer" performance CDE would be the first thing to go! They also don't mention any tweaks to improve system and network performance (and there are a few I can think of). Hell I'm willing to bet they didn't change the jumper setting on the Blade to get the memory to run at 100 MHz over the default 84 MHz as shipped! they also don't do anything like multipath or trunk network adapters (which you could easily do with Solaris). It looks like they took a machine out of the box, did a default install of Solaris, loaded applcations and "plugged it into the net"! I wonder how much better their performance would be if they tuned their server like Colin Bitterfield (of Sun Microsystems) did?
I did the site with no database whatsoever. Articles were stored in one XML format on disk, comments another.
So in other words, you stored the XML files in a hierarchical database called a "file system". Assuming one comment per file, how did you handle the half kilobyte of non-gzipable HTTP overhead that each XSLT-driven comment view incurs?
Assuming multiple comments per file, how did you handle locking the comment file so as to ensure that only one server process writes to a given file simultaneously?
Will I retire or break 10K?
> How is a developer supposed to design a Slash/Scoop/Nuke style board without storing the comments in a database?
I said "a database". Keep in mind that there are four main models of database structure: ad-hoc flat files, hierarchical, network, and relational. Here are some methods you suggested:
Store them in the file system
That's a hierarchical database, with keys called "paths" and "filenames". Some Wiki implementations do this, and I can see how it would work in a Slash clone. Here, the file system has to provide an atomic way to generate a new key. (mkstemp() is atomic; mktemp() and tempnam() are not.)
in a DBM file
"Data Base Manager"? Isn't much of MySQL just an SQL frontend to DBM?
in memory
Relying on memory as the main store of data fails the D (durability) in ACID. However, use of memory as a cache in front of a filesystem does provide useful gains.
or in an object database.
I'm not too familiar with object databases. To me, they seem to map the "network model" into a programming language's syntax. Right?
Relational databases are overkill for most web applications
I'll agree that heavyweight SQL databases are overkill for many uses that they're put to. Heck, the name "Oracle" almost sounds like "Overkill".
Will I retire or break 10K?
It's not about hits per second, it's about efficiency, and I think he (Brian I guess) did it very well. Some stupid idiotic slashdotters say, my servers handle this and that, but the point is, HOW MUCH DOES IT COST? HOW MANY MACHINES DOU YOU NEED? WHAT OS YOU WILL USE? WHAT'S THE BANDWITH SPENT? Let's not forget also the attitude to optimize the content to the modem users, well they also need to be taken in consideration. About the cache use, very intelligent, WE USE CACHE EVERYDAY all the time. Let's use it with efficiency. The 2 points where maybe some people are missing are, APACHED WAS ASS KICKED in the ass by this server RESIN, honestly I've never heard of it, I'm off web development since a long time... and PHP and APACHE can't be a good combination when things get ugly, and you need scalability. That's what I think was the most useful information, along with the IPC info. Well I'm impressed, It loads fast for me and I have a Half T1, so I could se if it was slashdotted, like somebody said, if /.ed it's a bandwith problem, never the CPU SETUP.
Kudos to Solaris with it's threads and Sun servers(Blade isn't even a server!! IT's a workstation) with it's architechture.
And I tought JAVA was slow, maybe for other uses, but in web development, it seems to be the best option yet.
Comment removed based on user account deletion
Talking about the /. effect, I think it would be amusing to have the /. clickthrough data displayed as a graph (e.g. clickthrough/min over time) for each link in a story (at least for the links in the initial story).
One usage of these graphs would be a popularity measure of the stories over time.
--f
did your hear about RAID?
I am surprised that java performs so well.
However they seems to be limited by their bandwidth.
PS: My site runs on a 700 MHz 4-proc Pentium III with Win2K, ASP.Net & SQL Server 2000.
Cached Pages Perf is 0.15 ms
Dynamic Page Perf is 20 ms
While i'm happy for them, this wasn't exactly rocket science. I'm surprised they had to roll their own connection pooling. This is standard in ASP/ADO apps and the new .Net apps. Caching the data is routine code you will find in most highly concurrent dynamic sites. Though, "caching" now adays goes a step further, generally XSL is used to build the page, and the result of the first XSL transform is cached to the file system. Some pages then go ahead and allow for an optional 2nd transform which puts in _very_ dynamic content, which can be shed/loss in the event of a traffic spike.
All in all, i agree with 90% of the posts, preventing a slashdotting is more about bandwidth then software/server performance. Yes, obviously hacked togeher asp/php pages that open and establish a connection to the database _per_ page will prolly die before bandwidh, i think even the newbiest of programmers recognize this nowadays.
This article made me yawn, but i bet we just paid his operational cost via add impressions for the new 5 months.
-malakai
-Malakai
A Dragon Lives in my Garage
Sure i did, and last i checked windows 95 didnt have very comprehensive support for raid hardware and no software-raid atall.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!