Slashdot Mirror

You could go with an expensive commercial solution like BigIP from F5, but those will run you at least $30k or so. You could also use Polyserve Understudy, which does pretty much the same thing only under Linux, and it's only about $400 or so. If you have all this expensive Cisco equipment and a Cat6000, you can run Local Director on that without buying additional hardware.

However, I suggest:
http://www.linuxvirtualserver.org or
http://linux-ha.org or
http://www.eddieware.org

It all depends on your application that you're running. If it's just http, any of these will work, but if it's something else, you're stuck with linux-HA or Linux Virtual Server. Eddie will only do http as far as I know. Plus Eddie uses Erlang, which may affect performance.

Anyway, they also forgot about Eddie which is mainly designed for redundancy and load balancing for web servers.

The best way of dealing with complexity is to simplify and limit features to the essentials. JavaScript, graphics, fancy layout, etc. all can present an bottomless hole for time and bandwidth.

Another thing to realize is that the relational database you are using is probably one of your biggest bottlenecks. Relational databases are slow. MySQL is actually one of the faster ones, but that's because it doesn't make a lot of the guarantees or provide a lot of the features that a "real" relational database provides. You can get a lot of performance from your relational database by tuning it, but ultimately, the architecture and functionality itself present a limit. If speed is of the essence, consider using dbm, plain files, or memory mapped files (Apache has several "databases" internally, and that's its approach.)

For a single person project, Java is probably not the best implementation language. It really shines for multiprogrammer projects. You may find that Python, PHP, or Ruby are better choices. Perl and Tcl are also widely used, but they are also the oldest of the scripting languages and have a lot of rough edges and clunkiness. The performance of all of them is excellent when used as server plug-ins. Perl and PHP have by far the largest libraries and toolsets.

Some of the packages I haven't used but that look interesting are the following. Enhydra takes a much better approach to dynamic HTML than most other packages, I think. Erlang / Eddie address scalability, reliability, and clustering in a really clean way (but you have to learn a new language). Zope I think has a lot of good ideas, but I'm not sure I'd use it for a large server.

Most load balancing solutions use some kind of network routing hacks. That's efficient, but can be a pain to configure. However, there is now a load balancing module for Apache that works as a proxy; that's probably also worth looking into.

The Coda file system is a free, next-generation AFS; while I'm not sure Coda is mature enough yet to be used in heavy-duty applications, systems like it help with scalability by replicating files automatically and have been used on some really large web sites.

I would stay away from commercial "application server platforms". They often are mostly repackaged open source software, and they are expensive and complex, and while they claim to be general, they are also usually created with fairly specific commercial applications in mind ("shopping cart", etc.). I think those kinds of packages are worthwhile for corporate developers that work with large teams of programmers and need consistency, documentation, training, and support.

So, to recap, this kind of stuff is still a lot of work and it seems like you are ahead actually. But there are a lot of proven open source tools available that have been used on really big projects. There are also a lot of open source developments coming around that may make this kind of project a easier.

My guess is at the time of the test their quad Xeon machine ran about $50,000, now down to $30,000 or so.

For $1000 to $2000 per box, I can build a Linux eddieware system that has redundancy, reliability, performance, and REAL scalability. Let's all just sit back and imagine a web server farm with 20 times the horsepower of slashdot taking on that poor little quad Xeon machine.

There, doesn't that feel better?

Eddieware is somewhat nifty.

At least worth a look, anyway.

Why not use multiple small servers than load
balance them with eddie. check out the
eddieware project. Cool thing is, it runs on
FreeBSD AND Linux and it's open source :)
http://www.eddieware.org

> But besides Netscape, which `corporation' has
> actually gave us any interesting source code
> (not hidden behind 1,0000 lines of legal crap)?

Ericsson has Open Sourced the programming
language/compiler/etc Erlang
and also the web-farm thingy (written in Erlang)
called Eddie.

They're both pretty significant hunks of code.

Eddieware does DNS load balancing (and hence isn't bottlenecked like the Linux Virtual Server Project), LAN load balancing, IP migration and admission control. In addition to linux 2.0.x and 2.2.x it works under FreeBSD and Solaris. Checkout http://www.eddieware.org .

Checkout the Eddieware press release at www.eddieware.org/txt/press990503.html . Funny how this didn't get a mention in the main slashdot articles but a closed source solution does!

Eddieware does DNS load balancing (and hence isn't bottlenecked like the Linux Virtual Server Project), LAN load balancing, IP migration and admission control. In addition to linux 2.0.x and 2.2.x it works under FreeBSD and Solaris. Checkout http://www.eddieware.org .

Checkout the Eddieware press release at www.eddieware.org/txt/press990503.html . Funny how this didn't get a mention in the main slashdot articles but a closed source solution does!

Ericsson, creators of the Erlang functional programming language, ATM switches and so forth, released Eddie as Open Source some time ago. It seems to do much more than some of the programs suggested here.