Managing Site Growth?

markmcb asks: "I started a web site about two years ago. When it began it was simple. The code was 75% hacked, and administration was easy. However, the times they are a changin'. Now I get hundreds of thousands of hits and have a steady flow of new users. I'm noticing an ever-increasing gap in terms of my site's popularity and its technological progression. Specifically, I have all sorts of 'XYZ for Beginners' books that are no longer of use to me. Even the so-called non-beginner publications seem to only scratch the surface of running a site. As problems get more complex, trying to Google every situation/issue I have with site administration has become less useful as well. I'm finding things like writing optimal code, configuring servers for high-volume, balancing ad income vs. server costs, and maximizing the efficiency from my moderation team have all become issues and that aren't addressed most books. What is the best way for a low-income, non-professional, but enthusiastic web designer/administrator like myself to manage site growth as it leaves the realm of just-for-fun?"

Comments

use others experience

[Beuno]

Taking a look at what others have done to solve these issues seems like the best option.
I think it's very unlikely you will find books in that area considering that when you reach a certain level of complexity technology changes too fast to make a book relevant.

Re:use others experience

[Anonymous Coward]

I agree with your point, but not your example. Wikimedia is built on Mediawiki which is poorly written PHP+Memcached+MySQL saved by SquidCache. They need twice as many boxes as a well-designed architecture would need. Another bad example is MySpace which is a coldfusion/asp.net shop that add dozens of servers a day to keep up with demand. Most people estimate that they need 5 times as many boxes.

Adding more hardware is always a way to dig yourself out of a digg, but be careful you don't just look at how the big boys do it and think that's right. Smart people can do more with less. Look to Python, Perl, C# (but not ASP.Net unless you really know what you're doing) who have mature libraries. PHP and Coldfusion suffer too much from the type of app that's built-up and destroyed on every page load. It doesn't encourage separation from what only need to be done once (app initialisation) and page response, which is a large part of optimisation (both in doing things every page load that should be done once, and seperating out the page data and having layers of cache from the DB and web-templating). Learn about http headers and SquidCache.

So far as software recommendations I'd say Python with CherryPy and Kid Templates (but not turbogears). It's fast and simple.

you can always give up

[spacefem]

Something similar happened to me when I started advicenators... a hackjob concept that floated along for few months and then suddenly grew to 10,000 members. I was overwhelmed. I scrambled to write code for a moderator system and abuse reporting, and any security issues we had took time away from that. Finally, about a year ago, I just gave the site away to a loyal member and his wife who knew their stuff and promised to keep the site pointed at the goals I'd started for it (I'm usually pretty satisfied with their methods in doing that).
 
The site had gotten so it wasn't fun for me. I was home every night after work writing PHP, and coming back from a weekend vacation was a nightmare. I also felt like all those members deserved more than I could give them.
 
Business professionals will tell you that it takes a certain type of person to get a business going, another type to get it stable, another type to get it to the top, etc. The web is the same way, and if you're a "starter" who can come up with innovative concepts that take off, then go do that. Don't get tied down with old projects.