Building a Scaleable Apache Site?

bobm writes "I'm looking for feedback on any experience building a scaleable site. This would be a database driven site, not just a bunch of static pages. I've been looking for pointers to what other people have learned (either the easy way or hard way). I would like to keep it Apache based and am looking for feedback on the max # of children processes that you've been able to run, etc. Hardware-wise, I'm looking at using quad Xeons or even Sun E10K systems. I would like to stay non-clustered if possible."

Comments

Slashdot

[Anonymous Coward]

Post the URL, not just the question, and let the /. effect take its toll.
Increase the threads etc. until it stays up.

Persistent Connections Are Your Friend

[Aix]

Just in case you haven't thought about this, for a database-backed website, getting rid of the database connection overhead is just about the smartest thing you can do performance-wise. Think mod_perl. Furthermore, consider moving your SQL server to another machine before making any other hardware changes. (If you haven't already...) The demands of an HTTP server are definitely different than those of a SQL server. If you're going to have a lot of dynamic content, plus a decent number of SSL requests, think about putting a proxy in front of your page-generating server. I know these aren't Apache tweaks, but they're worth considering anyway.

Cache, Cache, Cache

[Longstaff]

is *the* most important word around for dynamic sites.

I've built a site that's able to handle 1-2 million dynamic page views per day. There's not a single static page on the whole site except for the 404 page.

/. doesn't generate these pages on the fly, they're generated by a background process that runs every minute or so and stored as a file. There's no reason to requery the database if you don't have to.

One trick that we currently use is a little daemon that runs on our app servers (custom java app). It's essentially a tcp socket interface to a hashtable with an expiration timestamp. Here's how the site works:

1. request comes in
2. front end server takes GET params and queries the local cache daemon to see if those objects are local
3. if the objects are local - great - slap them together and deliver the page, otherwise
4. query the database for the object info
5. populate the cache daemon
6. deliver the page

Another trick we use is dumping the output from one dynamic page to be included by another. So, have a page that generates nothing but an element (eg. slashbox). Have a mechanism on the back end that requests that page and stores the result as a text file. The dynamic page (say, php or jsp) just uses an include directive pointing to the static text file - which can be formatted html.

Of course, the real weak point of the system (without clustering) is the database. Make sure that your data is index properly and that your queries are optimised. We have 2 tables with over a million rows each that get hit all the time. Proper data layout, quick queries and the local caches help our puny dual P3-733 (NON xeon) with a paltry 1GB of RAM dish out well over a million dynamic pages per day.

Re:Cache, Cache, Cache

[Longstaff]

I'm not sure I understand the question, but I'll take a crack.

The local cache is simply time based. Each element in the cache has it's own expiration time and part of the API allows you to specify a TTL for each element. The element's timestamp is checked against its TTL with every request - if it's expired, the daemon deletes the element and simply reports that it couldn't find the object.

Another reactive behavior of the daemon is that it will call a trim() (which walks through the hashtable and purges any expired objects that simply haven't been requested since they turned "sour") whenever the hashtable grows to a specified max size. There's some additional logic that keeps trim() storms from occuring.

On a proactive side, the daemon itself does some housekeeping. After X seconds (we have it set to about 2 hours) it trim()'s itself.

Re:Cache, Cache, Cache

[Longstaff]

Well, we don't actually cache whole pages. The closest we come is our front page, where the entire page is generated, stored as a text file and included except for the header where your login name appears.

If you were to log into our site, your name is displayed on *each* page - how would you cache that whole page effectively?

We cache objects and have the web servers assemble the objects in a page on the fly. So, a news story is an object, a poll box is an object, etc.

One reason for this is different TTLs. Our news stories don't change that often - if ever - once they've been published. A news story TTL may be set to 1 or 2 hours, while our polls are constantly changing and need a much shorter TTL.

Our main goal with this design was to ease our database load. Scaling a database up is *expensive* (Oracle quoted us $250,000 - for the one box) and complex once you start moving to clusters, etc. Scaling the front end is simple - add another server behind the load balancer. We currently have 6 web servers for redundancy and general zippyness, but our load can be handled by 2 or 3 of those.

Re:Cache, Cache, Cache

[Longstaff]

Well, this a problem with any cache system. With ours, you adjust the TTL to an acceptable value of "staleness" down to the second.

When optimising any system, relaxing granularity is something that you should look at. Do I really need the latest version of the news story up to this very second - or can I deal with one that's a minute or more old. In our case, the news stories are edited and reviewed before they're published, so it doesn't matter if the story is 1 minute old or 10 days old.

In an emergency, we can forceably expire an element.

There are cases on our site where we can't cache the data - we *need* the live data. Those cases are scrutinized thoroughly before we actually make a live call to the db to see if there's some way to get around it. However, most of our data is cacheable and we have a hit rate of ~80%

Re:Persistent Connections Are Your Friend - MAYBE

[Longstaff]

Combined with a proper connection pool, they can really save your butt.

However, persistent connections may be too much of a burden for an overworked db server. If you're using PHP/MySQL for example, mysql_pconnect may not be the way to go if you have a few front end servers hitting the database. It seems that the PHP connection pooling limit is per process. If you have 100 Apache processes w/ a 10 connection limit per and 10 web servers, that's a max of 10,000 db connections!!!

One idea might be an intermediate "connection broker" on a per server basis. We use something similar to this.

Apache's fork() model is great for stability, but it really hinders interprocess resource sharing. We're mostly Java based here, which allows us to use beans and such. Does mod_perl allow for resource sharing between processes?

Look in the right place

[linuxwrangler]

You have provided way too little info. First, do you really mean scalable or do you mean high-traffic. They are not the same thing. You can build a high-traffic site using technology that won't cluster/expand well and be screwed when you need a higher-traffic site. Converesly you can build a very low-traffic site that will scale quite well (a technology that allows you to easily add hardware as your traffic dictates for example).

It would be helpful to know, for example, what portion of the traffic (both # of requests and bytes) is static and what is dynamic (include images)? What is the peak (say 98th percentile) expected traffic? What are typical page sizes and how much are they compressible with gzip? etc.

Apache itself doesn't really handle dynamic content - its modules or an underlying app server do that. That is probably where you will have to do the most work.

As another poster mentioned, persistent database connections are essential. You may want to look into a "real" app server. JBoss is open source and just won some awards at Java One. If that is too much complexity at least be sure to use persistent connections in whatever other technology you select.

Persistent connections have a down side. Don't forget that your underlying database must be able to handle both your number of requests and your number of connections. If you just increase Apache processes you may find that the database is unable to manage that many simultaneous connections efficiently. Opening/closing connections for each request kills you. Maintaining hundreds of open connections kills you. This is one of the real strengths of any technology that can handle connection pooling - you will probably find that you only need a handful of connections to handle lots of front-ends and connection pooling allows you to do it efficiently. It can also help you scale by distributing connections to multiple database servers for you when your needs dictate.

The faster you can dispense with a request the better. This includes not only all your processing time but the transmission time to the client. A process/thread can't move on till the client has the data. Therefore...

Design your pages to give yourself a fighting chance. For example: if you have any static images be sure to set your http headers to prevent browsers from reloading them. Even the request overhead to the server to determine that the cached image is up-to-date is more than the size of the image itself so set a LONG expiration.

Trim unnecessary whitespace, using short names (ie. i/x.png instead of buttonimages/left_page_arrow_top.png) and so on.

If the pages are large enough and the clients slow enough then you may want to use gzip (mod_gzip) to compress the data. It will cost you processing time to compress dynamic content but will save you transmission time. If you pay for bandwidth you can see a 50-80% reduction in your bandwidth usage as well.

Note: if your spec of "non-clustered" and scalable still allows multiple machines and if you do have images or other static content you may want to move that content to a separate machine. The Linux kernel http server screams on static content (of course the static-content load on your server may be so small a percentage that it isn't worth the effort).

Try Apache 2.x first. One problem with 1.x on most (all??) platforms is the "thundering herd" problem. You may try to increase performance by running lots of processes but when a request comes in, all sleeping processes are awoken (the thundering herd) and although only one will end up servicing the request, the effect of waking up huge numbers of sleeping processes can be "bad".

Be sure to test with clients of varying speed. We discovered we could crash a site faster with slow clients than fast. Once while testing a Cold Fusion/IIS site it seemed like we could realy get some screaming throughput when testing on the LAN. Unfortunately when the server had to keep threads/connections alive long enough to service slow clients it wasn't so pretty. When we ran the simulation that way we could crash the server in 2 seconds.

Give me more specifics and I may be able to give better advice.

Re:Look in the right place

[babbage]

I was going to reply to the article, but you hit most of the points I was going to ...and a lot of them I wasn't thinking of. So to cut down on the redundancy, I'll just reply here & add a couple more points:

• To control the timeout problem for slow connections/clients, Apache can be tuned to use very short keepalive times. HTTP/1.1's keepalive header can be useful for clustering a burst of multiple requests (such as an HTML file plus a collection of images for it) but the dormant processes it can generate can be more costly than the TCP connection overhead time you were trying to avoid by enabling HTTP/1.1. Oops. Set the timeout low enough to hit the sweet spot between "too many new TCP connections" and "too many idle Apache children".
• Reconsider your resistance to clustering. Yes it can make things more complicated, but it can also make your life a lot easier. Want to ease the Apache or MySQL load? Buy a couple more boxes & have them NFS mount the content or data directories. You can also do clever things like putting all your static content on a server optimized for that purpose (no mod_cgi or mod_include or anything like that) or dedicating hardware to a mod_perl instance, a mod_php instance, etc. Whatever. You gain a lot more flexibility, you compartmentalize things so that (hopefully) you don't have a single point of failure, and it's easier to swap/upgrade/replace components in one area without disrupting things in others.
• As another commenter noted, caching can be a big help. Caching proxies can reduce the load on the main server significantly. Not everything can be cached, but it's possible to strike a balance between readily cachable data (home page, section headers, images, stylesheets) and material that really does have to get generated for each request. On a big site, every little bit helps.
• mod_gzip is your friend. Processing power is always going to be cheaper than bandwidth, so spend your money on compressing data is cheaper than paying for increased bandwidth. Even if not all clients can take advantage of it, if a significant fraction of them can then you'll quickly come out ahead.
• If you have any huge content (audio or video) that places a heavy load on the rest of your systems, you might consider outsourcing it to a company like Akamai that specializes in delivering such content quickly. Services like this are probably expensive, but if you need it then you need it, and going with an Akamai is surely cheaper than setting up & maintaining your own data centers all over the country & world.
• As the above poster notes, consider Apache 2.0. Among the many neat-o features it offers is a choice in execution model: in addition to the fork/exec multiprocess model that 1.3.x used, you can also try threaded modes (which should be a big boost to Win32 servers if you need to go in that direction for anything), and I think maybe some more exotic execution methods. Depending on you setup, you might be able to find a big speedup by switching to threads. (Note though that, as of now, Apache2 and PHP4 don't play nicely together, and the same is probably true for a lot of Apache extensions (mod_perl, others) so make sure that whatever modules you need to use are going to work. Test test test!

Re:Look in the right place

[bobm]

Thanks for the info, I left the specifics out since I'm looking for generic feedback (for the learning).

The site will be mostly serving dynamic content with the average page being about 60-120k of code and around 10k of images. And yes, that's a lot of code but the site is serving up reports and whatnots. There are small pages between reports and the usual login, etc screens.

The real purpose of the question was to see how different tuning is being used in the real world, as the web has matured there has to be some interesting information on keeping the systems up 24/7, etc.

For example we're looking into a replicated database with just the important info (and I know that important is a real fuzzy term) for periods when we need to bring the primary database down.

what would be interesting is the proactive analysis (when do you add more hardware, etc) that is done on a live running system.

thanks

Re:Look in the right place

[sydb]

Trim unnecessary whitespace, using short names (ie. i/x.png instead of buttonimages/left_page_arrow_top.png) and so on.

While the rest of your post contains many good points, I find this comment bizarre. The overhead of a few extra bytes is insignificant compared to the benefit of having maintainable code.

You need to provide way more info

[DevilM]

You say you want it Apache based on dyanmic. Well how are you going to build the dynamic pages? Are you writing your own Apache module? Are you using Perl, PHP, JSP, CFML... what? Are you using Apache 1.3.* or Apache 2.0.x? What database are you using? What kind of application is it? What OS are you using? What about disk subsystem, is it RAID based? If so, what level? Why do you want to use a single big machine instead of many small machines?

Anyway, you need to provide way more information in order to get help. There is no magic way to make a site scalable. It just depends on the answers to all the above questions and more.

Re:You need to provide way more info

[bobm]

Database: Informix on EDS served from an E10K.

Dynamic: currently mod_perl but open to something faster (if there is a proven faster technology).

Apache: current 1.3.x move to 2.0.x when it's ready for prime time.

OS/Hardware: open, currently Solaris/Sun, open to quad Xeon/Linux if it has the performance.

The reason for asking about a single vs multiple machines is that I wanted to get a handle on what one box could do as opposed to the gut reaction to just keep adding servers.

Although I'm not expecting magic I didn't want to get too specific because I'm interested in feedback from across the board, for example how does Orbitz or Yahoo or *New York Times* maintain uptime? I haven't found anywhere that discusses places like that.

Re:You need to provide way more info

[Longstaff]

Places like Yahoo will globally distribute their servers.

Services provided by Digital Island, Mirror Image and Akamai will distribute your content to a node as close to the client as possible. We use those services for our images (only static content we have), but Akamai (at least) is pushing a new distributed processing model. You give them a Java WAR file or a .Net app and they'll push the *app* out to the edge. Expensive, but interesting.

Session Management

[JMandingo]

Assuming you are using multiple web servers, and that your app is complex enough to require a session data management scheme (rather than just passing vars from page to page in the query strings), I recommend using cookies for session data. Naturally this only applies IF you don't mind requiring your clients have cookies enabled, IF you don't need to store anything more complex than strings, and IF the total amount of data you need to store is small.

Another option is to store session data the your top level frame on the client, but this can be messy and hard to debug. Storing session in your database is elegant and easy to debug but can increase the hits on your database to a prohibitive degree. Adding database bandwidth in the future is difficult and expensive. Adding web servers to your system is comparatively cheap and easy.

A good article

[cwinters]

A good reference on this is from one of the eToys architects. It uses mod_perl as the technology but the general strategies -- caching in particular -- will work for any application server technology.

Kegel's site

[jawahar]

Contains very good information. http://www.kegel.com/c10k.html

Re:Kegel's site

[Longstaff]

mod parent up - great link!

Re:Look in the right place, but what is Clustering

[babbage]

It can mean different things. In the sense I'm used to it being applied, you split up services in different ways across multiple machines. Making up some terminology that I don't think anyone would object too very strongly, you can split them vertically (an Apache box, a MySQL box, etc) or horizontally (multiple Apache frontends sharing the same content somehow). A very tight definition might mean getting the computers to act as if they are one unit -- I think this may be what your sales guys are talking about -- but it's simpler to just have them working together loosely without having the extra step of pretending to be one homogeneous entity.

Like I say, there are different ways of doing this, and really you ought to browse through a good bookstore or two to get more details. One strategy that's easy to implement might be to split up your content so that plain html is on www.site.com while your images are on img.site.com, your cgi scripts are on cgi.site.com, and your data is housed on db.site.com (which probably shouldn't be web accessible, by the way -- this protects you!). This is a vertical split. Or you can go horizontal by placing everything behind a load balancer that redirects incoming requests to one of several web servers -- each of which can be getting content from a single shared NFS partition. Or you can do a mix of those: maybe all the front end web servers communicate with dedicated database etc boxes behind them (which, again, would not be otherwise internet accessible).

On a Linux or Unix system, NFS is a pretty easy way to mirror content across all your servers. For a Win2k served site, you could probably get away with CIFS Windows shared drives. Or if you want to be really cutting edge, WebDAV might be able to meet similar needs. Less clever -- but debatably easier -- ways to do it might involve rsync'ing content from a master content server to a set of web-facing server "clients". A variation on that idea ends up being more or less identical to content proxy caching, as one big expensive app server in the back gets it's data cached onto a pool of cheap web facing proxies.

But, like I said at the beginning, the devil is in the details and you really ought to pick up a couple of good books if you want to learn more about this. Strategies for e.g. database "clustering" can vary widely depending on the RDBMS being used: I doubt the method would be the same for MySQL as it would for PostgreSQL, Oracle or SQLServer, for example. Some of those might be able to do this work almost transparently, while others would involve more manual planning & setup.

How about...

[jabbo]

When I worked at XOOM we had a farm of about 30 front-end FreeBSD webservers mounting member directories via NFS and serving an average of 500mbps (peaking up to about 1Gbps at times). The key to that architecture was that member logins were cached via a proprietary daemon that all pages authenticated from. Templates, dynamic content, etc. were all pickled to flat files whenever possible (at first, not much; later, once the merger with Snap! was done, much more, as their content caching system was superior).

The database on the Xoom side was an E450 IIRC. Snap used much burlier hardware because they were basically a silver-spoon project of CNET/NBC.

The lesson for scalability is simple, cache like a motherfucker and make everything you can static. And run DSR. ;-)

If you decouple the database from the webservers you need to make extra sure that you proxy the high-traffic requests, either by running a static-file-dumping daemon process (for content) or a proxy daemon (for authentication). My moderately-low-traffic site at my current job can handle two saturated DS3's worth of traffic with 1024 apache child processes running on each of 2 dual PII boxes w/512MB RAM, plus the database running on a dual PII w/1GB RAM. Doesn't even break a sweat. Postgres (the database) runs 1024 child processes with a lot of buffers, NFS caches are pretty good sized (if your frontend webservers are Sun, you can use cachefs aggressively, I would), and overall it just took some serious tuning to make sure that nothing fazes it.

I'm working on a couple of "community" sites with similar demands (~1million visitors/month) and mod_throttle + caching will solve one's problems, the other is where I stole the throttling idea from :-). Just tune, tune, tune... you'll get it.

For the whiners, Xoom failed in the end because it lost sight of the cheap-ass principles that made it a good stock market scam. Right up until the end, performance on the member servers was sub-4 second per page on average.

Re:ACS/OpenACS

[consumer]

The problem with Apache 1.x/PHP/mod_perl/MySQL/PostgreSQL is that the so-called persistent database connection is per-process based.

And how is this a problem exactly? If your server is handling only dynamic pages (your static stuff should be split onto another server) you will almost certainly need a database handle on every request. Connection pooling is only useful if your application spends a lot of time NOT using the database.

Then there is the problem of running out of db connections for any particular process.

Why would a particular process need more than one database connection? Each process only handles one request at a time.

Apache 2.0 is likely to be better in this respect, but I still think that AOLServer is cleaner.

Apache 2 provides full support for threading, so it can use the same approach as AOLServer. It doesn't sound like you know very much about it, so maybe you should check it out before you tell everyone it's no good.

Great article on Scaling your DB

[RevDigger]

This is a great article on scaling a website really fast. I found their techniques for scaling their database especially interesting.

http://www.webtechniques.com/archives/2001/05/hong / /A>

It's about the guys who built amihotornot.

- H

NFS vs. rsync?

[LedZeplin]

Several of you have discussed using NFS for cluster webservers to access a shared web root. My current setup uses rsync to distribute the files to the cluster nodes and I'm wondering why NFS? It seems that the rsync method would be a lot more failure resistant. If my primary server goes down the cluster nodes can serve the site as it was at time of failure. With an NFS server you would need a high availibility failover other wise all the cluster nodes are SOL right? I'm curious what the plus side to NFS is, maybe I'm missing out on something.