Slashdot Mirror
Managing Site Growth?
markmcb asks: "I started a web site about two years ago. When it began it was simple. The code was 75% hacked, and administration was easy. However, the times they are a changin'. Now I get hundreds of thousands of hits and have a steady flow of new users. I'm noticing an ever-increasing gap in terms of my site's popularity and its technological progression. Specifically, I have all sorts of 'XYZ for Beginners' books that are no longer of use to me. Even the so-called non-beginner publications seem to only scratch the surface of running a site. As problems get more complex, trying to Google every situation/issue I have with site administration has become less useful as well. I'm finding things like writing optimal code, configuring servers for high-volume, balancing ad income vs. server costs, and maximizing the efficiency from my moderation team have all become issues and that aren't addressed most books. What is the best way for a low-income, non-professional, but enthusiastic web designer/administrator like myself to manage site growth as it leaves the realm of just-for-fun?"
If it is OmniNerd he is talking about he should definitely look at something like Scoop, Drupal, or Slash. Any of them should meet the needs of that site easily and he will probably have a much easier time of finding assistance (paid or volunteer). I've played around with Drupal a fair amount, it's easy to install and configure, it has a lot of modules and themes, I'd definitely recommend he try it out first.
I run a community website which is written in Perl with a MySQL back end.
Despite having just under 5000 users I had 3million hits last month, and shifted 13 Gb of traffic. Not bad for a single (dedicated) host!
There are two things that I'd suggest above all:
I use Danga's memcached which has a perl interface, but there are PHP ones too. This allows me to sensibly cache database queries (don't forget to test things to make sure you expire the cache appropriately!)
A combination of minimising queries and caching has kept me going even under a slashdotting.
If you have written the site code yourself I'd urge you to add a test suite. My site runs a full test suite every day, and I run it manually whenever I make changes - this allows me to be sure that I'm not breaking things when I make changes.
Of course the standard development model of having a "live" site and a "test" site help here too. I develop the code on a laptop and store it under version control (CVS in my case, but it doesn't matter which system you use as long as you pick one) and only when it has passed the test suite do I push it to the live site.
Adding extra hardware can be an option for bigger sites, but I'm not at that point now. I had my biggest strain when the site reached around 1000 users, since then things keep ticking over nicely, and although it is growing it isn't growing terribly quickly which suits me fine. (There are a lot of users who visit the site via google searches and never register/return; I'd like to fix that, but I don't mind too much!)