Slashdot Mirror

embo writes: "The company I work for has been approached recently by an academic organization looking for advice on providing web space for 30,000 - 40,000 users. They are limited by budget, so I'd like to recommend something with Linux and apache. They are thinking of offering around 50 MB if disk space per user (which at maximum utilization would be ~2 TB of data storage), and no database driven content (though they want to allow CGI through Perl and Python, for example)." This is a huge undertaking. Can anyone think of solutions better than the ones embo outlines below?

"I can only think of a couple of ways of doing this. One is to have an enormous single fileserver and have a cluster of apache web servers that NFS mount the home directories to serve up web pages. Then the users FTP into the main file server to store their web pages. To me, this seems wildly inefficient, and you have no real redundancy if the main fileserver crashes unless you are using a SAN (which is very expensive), or you have a hot backup that is rsync'ed or something. And I'm thinking that rsyncing up to 2TB of data would be an exercise in futility.

My other thought was to have several back end file servers with a fixed number of users on each server, and then send all HTTP requests through an LDAP server first, which would then do a redirect to the machine that user's web page resided on. The big problem then is how to make sure users are FTP'ing into the machine that their account is on? They may also use FrontPage extensions with Apache, and this could complicate things even worse.

I know there has got to be a better architecture for this. How do enormous sites like Yahoo and Excite tackle this problem? They have hundreds of thousands of users! Better yet, how could they tackle it with Open Source tools? Would, for example, a Turbo Linux cluster help this problem any, or would I still have to replicate the data across every node in the cluster (meaning I'd need up to 2TB of storage for each cluster node!) Then what happens if they decide they want to add another 10,000 users? I can't find pointers to information, or ideas on how to do this *anywhere*. Can you fellow Slashdotters give me any advice?"

I've been so busy on the code frenzy that I've been behind on the quickies! Tragic! First lets get the serious quickies out of the way: chris sent us the Atlanta Linux Showcase Tutorial and Conference program for the 3rd Annual ALS, comming up October 12-16, 1999, in Atlanta Georgia. Registration is open. Bl0w0ff noted that The dockapp warehouse has been upgraded and redesigned. k-rist sent us SimShatner. Here is a site selling a video history of Atari with interviews with the guys that did Pac-Man and all that early stuff. Someone sent us a link to another place you don't want to see a BSOD. Want some Blair Witch Parodies? irishmikev sent is a Southpark Parody and stairs sent The Blair Family Circus Project. How about a pair of strange places to put a server? Gareth Walwyn sent us one in a potted plant and GFD noted thatLinux Today has a story about a box that runs in a real Pizza Hut Box. If strange Linux boxes ain't your bag, someone submitted Apple Fritter which contains strange cases for Apples (Legos, Radios, and more) Jade wrote in with how to apply for the position of Sith Apprentice. and rjh pointed us to the iMaul (seems like a lot of stuff is coming in pairs today) Evan Vetere noticed that despair.com has new de-motivators. Matthew McCabe sent us tuxtiles which is taking votes on designs for "Linux Blankets". Since we're mentioning merchandise, I gotta plug Think Geek which is the first place I've seen with good stuff. They mailed us a box of freebies, but I actually woulda bought most of the stuff they sent me (mugs with #include <beer.h> and some sweet perl shirts and other cool stuff). Most of the "Geek" sites just sell crap but most of this was actually clever. We probably should also note that Copyleft finally has the new Slashdot shirts from our contest winners, they look great. ralphb was the first to say that Time Digital has an article on Slashdot.