Scaling Server Setup for Sharp Traffic Growth?

Ronin asks: "We are a young startup developing a yet another collaborative platform for academic users. Our platform (a) requires users to log-on to the website for extended period of time, and (b) is content intensive - stuff like courses, quizzes and assignments gets posted regularly. We're using a LAMP setup on a 1 GB P4 server. Our user base is small (about 1,200 users, 5-7% connected at any given time) but we expect it to grow rapidly. We expect sharp traffic growth, and are working to scale our server software & hardware setup linearly. What kind of server setup plan should we go for keeping in mind our content heavy application and that we may have to scale up rapidly. Can anyone share his/her experience with LAMP in dealing with scalability of high-traffic sites? Taking clues from the Wikimedia servers, we understand that the final configuration involves proxy caching for content, database masters/slave servers and NFS servers. We of course don't have such a high traffic, but it will be interesting to note what kind of server config you'd go for."

Comments

Don't forget about C-JDBC.

[aminorex]

I should mention that if you didn't code-in memcached, you probably don't want to retrofit it, just for performance tuning or capacity scaling. In that case, I should suggest C-JDBC. You don't need to use a Java AS node in order to use C-JBDC, either.

I haven't made a production deployment of C-JDBC, so I defer to the experience of others, but from my research, it looks like a hot ticket for scaling DB performance while simultaneously isolating you from the specificities of a given DB product.

Re:all about partitioning

[Kirth Gersen]

I think the original question from Ronin suggests he will not know what "partitioning" means. Indeed, the parent seems to be using it differently from my own understanding of the term.

Let me rephrase the parent like this: make sure that you have easy access to your domain's DNS configuration so that you can easily create new machine names in your domain. *Initially, all of these new names can point to a single machine.* When you write your app, you can create multiple databases -- for instance -- on multiple machine *names*, all of which resolve (at the beginning of your project) to a single machine. (You can do the same thing with image servers, mailservers, etc.)

Later, as you start to need higher performance, you can reinstall each of these databases on a separate machine, update the DNS, and *everything continues to work* with no code changes or funky failure modes at all.

Oh, make sure you can get a largish block of addresses from your ISP/hosting service before you start. You don't want passwords going between machines to have to cross network boundaries.