Scalable, Fault-Tolerant TCP Connections?

pauljlucas asks: "My company is developing custom server software for an instant messaging type server (under Solaris). Every client maintains a TCP connection to the server when it is 'logged in'; the server maintains state of who's logged in where. For large-scale deployment, there are two problems: scalability and fault-tolerance. A single server can handle at most around 64000 open sockets. To go beyond this, you need many servers. Another way would be to 'fake' a TCP stack in user-space (by reading/writing raw TCP packets) thus not having one real socket per connection. For fault-tolerance, ideally one would like N servers to maintain the exact state, at least for the server process, so that if one goes off line, the other(s) can pick up seamlessly. I'm thinking that both of these issues must have already been solved without having to write lots of custom software. Is anybody aware of off-the-shelf software and/or hardware solutions (either commercial or freeware)?"

Comments

LocalDirector

[Huusker]

You can get a TCP load balancer like Cisco LocalDirector or one its competing clones. They are expensive tho ($20,000)

Your actual problems are somewhat different

[Twylite]

You indicate that your scalability problem kicks in at around 64000 simultaneous clients. Having developed high-performance scalable servers I would recommend taking a look at The C10k Problem, which is rather sobering: handling 10000 simultaneous connections is already a bitch.

Basically if you are using select() or poll() and have 10000 connections, you can expect 30% of your timeslice (on a fast machine) to be taken scanning the connection list for available data. Your performance also goes down the drain after about 250 connections (empirical observation; our server handles async requests which are offloaded to a hardware device for processing, so most server overhead is packet handling).

To get more connections than that you need to look at OS-specific methods: IOCompletion ports on NT, /dev/poll on Linux & Solaris, and kqueue on FreeBSD. These scale must more linearly out to 10000 connections (depending of course on if your server can handle the total load), but still don't give you the ability to scale endlessly.

Finally, I know of no intrinsic reason (although OS implementations may have arbitrary limits) why a server should be restricted to 64k connections. A TCP/IP connection is defined by two endpoints, each being an IP address/port cobination. Your server's ip/port is always one of those endpoints, and the client are unique. Client connections on the other hand are limited by the number of available port on the computer, i.e. 64k.

For fault tolerance, your best bet is to look at the architecture of Enterprise Java servers. Effectively you have a load balancer up front which redirects packets to one of several application servers; any persistent information must be saved off to a database server, and if necessary two or more database servers must be configured to synchronise with each other on a continual basis.

I am not aware of any software that can magically do this for an arbitrary application, although you may find network shared memory libraries which can more or less accomplish this. But if you are concerned about performance, you are unlikely to find COTS software to do the job (since the current business model is to throw more computing power at it).

You want linuxvirtualserver

[pthisis]

First off, the Kegel's c10k page referenced earlier is definitely worth a read. And if you're under the impression that having 2^16 TCP port numbers limits you to 2^16 connections, that's not accurate. You can have hundreds of thousands of connections to one machine, presuming you manage them properly (as the c10k page points out).

More importantly, you should check out http://www.linuxvirtualserver.org/ ; it's aim is exactly what you want:

"The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the cluster is transparent to end users. End users only see a single virtual server."

Sounds like a perfect match.

Sumner