Slashdot Mirror

The C4.1 contribution protocol I eventually wrote for ZeroMQ solved this problem. You have to develop rules that catch bad actors (yet not learners) and then educate project managers on how to fire people when needed.

Our rules for instance ask that you solve one problem with one patch, that you never break existing stable APIs, that you respect style guidelines, and so on. When people break these rules we give them several chances to improve their behavior. If they persist in doing it wrong, we remove them.

Turns out, when the rules are very explicit and teach people how to make good patches, then it's very rare we have to fire people.

The rules are at http://rfc.zeromq.org/spec:22

You might benefit from looking at zeromq, which can simplify this type of coordinated processing, both in single- and multi-node systems. There's a Python binding, so you should be able to give it a go quite quickly. Not guaranteed that this is the right approach for your particular requirements, but it does sound similar to stuff I've worked on in the past, and in my opinion zmq does simplify away a lot of the complexity in a reliable way. Performance is pretty amazing too! See the zeromq guide for details

Check out Twitter's own Storm system built on top of ZeroMQ.

http://storm-project.net/

http://www.zeromq.org/

You may find something you like.

I'm not sure, apparently ZMQ has some problems over 'public internet' (not because of the link, but it does not work properly when receiving 'unknown messages')

If you want to know more about ZMQ see the videos http://www.zeromq.org/intro:read-the-manual

I never studied UDT, but if you only need point-to-point communication, then UDT may be better for you.

However, if you need other communication modes, like Publisher/Subscriber, multicasting, etc or even interprocess communication, try ZMQ

Linux is extremely good at networking. Stick the LGPL zeromq library on top and you get low latency messaging middleware. Afaik this is also really popular in financial IT systems. People are doing 5M-8M messages per second with it.

The drop-off in per-core efficiency is due to multithreading models based on shared data. Shared data requires locks, semaphores, and waits, and as the number of threads trying to share the same data increases, they get more and more conflicts, and waits, so that even the very best multithreaded application sees no speedup after four or so cores. But it's unlikely to have such heavy work on a desktop, and for dedicated back-end servers it's frustrating because above four cores, there's no obvious gain. Perhaps that is why Intel and AMD are not yet fighting a core war.

However the shared data design is just one option. If you do message-passing concurrency you can create applications in any language that can scale linearly to any number of cores. It's possible in any language, just requires the right inter-thread communication model.

I assume the core war will come once message-passing concurrency becomes a standard way of writing concurrent applications.

You don't even need Erland, you can use a lightweight message-passing library like ZeroMQ that lets you build fast concurrent applications in 20 or so languages. It looks like sockets but implements Actors that connect in various patterns (pubsub, request-reply, butterfly), and works with Ruby, Python, C, C++, Java, Ada, C++, CLisp, Go, Haskell, Perl, and even Erlang. You can even mix components in any language.

You get concurrent apps with no shared state, no shared clock, and components that can come and go at any time, and communicate only by sending each other messages.

In hardware terms it lets you run one thread per core, at full efficiency, with no wait states. In software terms it lets you build at any scale, even to the scale of the human brain, which is basically a message-passing concurrent architecture.

Inter-thread synchronisation is slow. If the code is local to a thread (and doesn't use slow devices like network or persistent storage), execution time of most functions is tens of nanoseconds. However, when inter-thread synchronisation - even a non-blocking synchronisation - kicks in, execution time grows by hundreds of nanoseconds, or even surpasses one microsecond. All kind of time-expensive hardware-level stuff has to be done... synchronisation of CPU caches, memory barriers etc.

The best of the breed solution would run in a single thread and omit any inter-thread synchronisation altogether. It seems simple enough to implement except that single-threaded solution wouldn't be able to use more than one CPU core, i.e. it won't scale on multicore boxes.

A good multi-core solution would be to run as many instances of ØMQ as there are cores on the host and treat them as separate network nodes in the same way as two instances running on two separate boxes would be treated and use local sockets to pass messages between the instances.

This design is basically correct, however, the sockets are not the best way to pass message within a single box. Firstly, they are slow when compared to simple inter-thread communication mechanisms and secondly, data passed via a socket to a different process has to be physically copied, rather than passed by reference.

Therefore, ØMQ allows you to create a fixed number of threads at the startup to handle the work. The "fixed" part is deliberate and integral part of the design. There are a fixed number of cores on any box and there's no point in having more threads than there are cores on the box. In fact, more threads than cores can be harmful to performance as they can introduce excessive OS context switching.

Inter-thread synchronisation is slow. If the code is local to a thread (and doesn't use slow devices like network or persistent storage), execution time of most functions is tens of nanoseconds. However, when inter-thread synchronisation - even a non-blocking synchronisation - kicks in, execution time grows by hundreds of nanoseconds, or even surpasses one microsecond. All kind of time-expensive hardware-level stuff has to be done... synchronisation of CPU caches, memory barriers etc.

The best of the breed solution would run in a single thread and omit any inter-thread synchronisation altogether. It seems simple enough to implement except that single-threaded solution wouldn't be able to use more than one CPU core, i.e. it won't scale on multicore boxes.

A good multi-core solution would be to run as many instances of ØMQ as there are cores on the host and treat them as separate network nodes in the same way as two instances running on two separate boxes would be treated and use local sockets to pass messages between the instances.

This design is basically correct, however, the sockets are not the best way to pass message within a single box. Firstly, they are slow when compared to simple inter-thread communication mechanisms and secondly, data passed via a socket to a different process has to be physically copied, rather than passed by reference.

Therefore, ØMQ allows you to create a fixed number of threads at the startup to handle the work. The "fixed" part is deliberate and integral part of the design. There are a fixed number of cores on any box and there's no point in having more threads than there are cores on the box. In fact, more threads than cores can be harmful to performance as they can introduce excessive OS context switching.