Writing High-Availability Services?

bigattichouse asks: "I have a project coming up that will require some serious load capabilities accepting socket connections. while I have a design that can be distributed over multiple servers (using queued reads/writes to the db) and is as low-overhead as I can make it - I am concerned about falling into common problems that may have been overcome in many other projects. What strategies (threading, forks, etc) give the best capability? What common pitfalls should I avoid?"

Comments

One common pitfall ...

[Anonymous Coward]

... is attempting to parallelize a program that would otherwise have been more efficient had it just been kept serial.

All too often I've read the argument: "Oh, performance isn't good, so I'll parallelize it". That doesn't hold much weight, as not all things are efficiently parallelizable.

So, before anyone suggests that you start pthread_create()ing threads everywhere, give some serious thought as to maxing out the serial performance first.

Re:One common pitfall ...

[gnuadam]

I'd not say this is perfectly good advice.

When you carefully optimize your code to acheive maximum serial performance, you get just that, maximum serial performance.

The algorithm that acheives maximal parallel performance, in my experience, is often quite different. What you really need to do is to carefully plan your code for maximum benefit in the resources you have available.

If you want to design a parallel code, start with that assumption, not from the standpoint of parallelizing a serial code.

Beware slow connections

[linuxwrangler]

In a former job we totally hammered an app on our internal lan and got many times the requests rate we would need in the real world.

Fat, dumb and happy we figured that the real world couldn't hammer us as hard as we could internally. Wrong! Slow connections require maintaining connection resources much longer than on an internal network where the response can be created and dispensed with almost instantly.

Maintaining all those simultaneous connections depleted our resources and the app went into full meltdown mere seconds after being released on the public servers.

We beat a hasty retreat to the old code, licked our wounds, and learned a valuable lesson.

The C10K problem

[Panoramix]

You probably know about this paper already, but just in case you don't:

The C10K problem

The paper deals with web servers handling ten thousand simultaneous TCP connections. But most of it is not particularly related to HTTP or web problems, but with more general socket I/O stuff --particulary with the ways of dealing with readiness/error notifications (e.g. select(), poll(), asynchronous signals, etc.). It also discusses other kind of limits (threads, processes, descriptors).

It is quite enlightening. It may be a bit outdated --I remember reading it about the time Netcraft was doing all that noise about Windows being faster than Linux as a web server-- but I'm sure most of it is very relevant.

What kind of system?

[Pyromage]

In general, there are many things you can do. Pooling, caching, etc. can help in many situations. But what situation are you in?

Are you writing a web app where you have to hold session data across TCP connections?

Are you writing an app that will have sustained connections (more than one request per connection?)?

These different situations require different strategies.

DB reads more common or writes? How big's the difference?

What kind of system is your target? Can you trade memory for speed (caching)?

Take a look at SEDA http://seda.sourceforge.net. While you probably won't be rewriting your app to use this framework, many of the strategies may be useful and applicable to your app.

Also, just note the difference between efficient and scalable: some designs will take longer than others on short loads, but many of those make tradeoffs that are only noticable under high stress. Consider what tradeoffs you've made so far: some may be good or bad, and more may need to be made.

All this was said without knowledge of what you app is other than a DB app. I am not an expert, but I doubt an expert could say all that much with that little information.

Multiple strategies for HA systems

[isj]

The goal of HA is usually that the end-user or the client applications will never detect that part of the system has been down. One strategy is:

• Separate the system into component
• For each component:
  Devise a mechanishm for dealing with the situation where the component is unavailable for several hours. If that is not possible you must implement redundancy.

Another (or additional) strategy is to implement self-monitoring. The component should monitor themselves for faults, and optionally monitor other components and restart them if necessary. The gotcha here is not to mask any errors for any high-level monitoring system.

You also need error detection&recovery in all components.

One thing that sometimes really bites you with TP is the long time it takes to detect that a connection is broken. You need application-layer keep-alives to detect this rapidly. Changing the kernel parameters for TCP timeouts can be necessary too.

Finally, you may want to have a look at Self-healing servers

In one case....

[OwnerOfWhinyCat]

Though Pyromage's criteria requests are vital to making good suggestions, I had a high-burst rate problem for a server application that I solved slightly "out of the box." Since I wrote the client as well, I switched from the "connectioned" TCP interface to the "connectionless" UDP one. Since my application had to track the state of every pending request in any case, going to the connectionless protocol only meant adding 4 more states. This cut the kernel overhead significantly, and the total packet counts went down by half.

If you can supply the rest of the data it's likely that other good tradeoffs can be suggested.

use erlang

[sesquiped]

Erlang makes writing applications like this much much easier than in any other language or framework I've seen.

Check out this tutoral on making a fault-tolerant server in Erlang.

Re:Less fluff, more detail

[Twylite]

Ah, telecoms :) Is this the industry/application in question, or just hypothetical? There was mention of throughput (indirectly) and availability, but not of latency in the original question. Also there was mention of queuing queries to a back-end database ... this doesn't sound like a minimal-latency scenario.

Anyway, the technologies you mention are not likely to be acceptable in such a scenario -- but MOM is quite likely to be appropriate. In fact many cellular services are based on MOM (conceptually, although they may not use commercial MOM products), as you receive requests and send responses based on discrete messages from/to the handset.

Slow connections, and lots of 'em!

[Pierre Phaneuf]

I like a single thread/process per CPU design, where each thread/process use event-driven I/O to operate. A few things to keep in mind:

Never forget how a lot of idle connections can kill you, for example a thousand of people connecting to your fast server over 56k modems, sucking only a packet now and then. If you have a thread/process-per-connection design, like Apache, you'll get screwed real hard when you have a bazillion thread/process doing *almost* (but not quite) nothing, swamping the I/O scheduler and context switching like mad. If you use a select/poll-based approach, scanning all these inactive file descriptors, looking for those that are readable/writable, wastes a lot of time. Check out the new epoll stuff or Ben LaHaise's callback-based AIO interface.

You should use something like libevent or liboop to abstract your event loop, so that you can use select/poll on old or unpatched kernel, but so that you use epoll and other fancy event dispatching mechanisms on your production servers.

Here are a few URLs for you:

http://kegel.com/c10k.html
http://pl.atyp.us/co ntent/tech/servers.html