Slashdot Mirror
Open Source Message Queuing System
psicode writes "John Davies has announced AMQ, an effort at JPMorgan Chase & Co. to create an open-source message queuing system that can compete with proprietary message systems like IBM MQSeries and Tibco/RV. The announcement was made at the
annual conference Web
Services on Wall Street during Davies' presentation on February 1. eWeek has an article today with more details and some funny statements about Red Hat, SuSE and Sun possibly integrating AMQ into their "kernel". If JPMorgan Chase & Co. follows through with their announcement and they come up with a suitable open-source license, AMQ could become the Apache of messaging systems."
The trick is in Web Services, I was lead developer on a LARGE project during the DotCom Era (2000 ish) that used Vitria, a great system and its interop capablities blew my mind at first but if we could have utilized web services we would have halved developemnt time, not to mentio forgoing a 1 Million $ Vitria Liscence.
It was cool though, we took about 6 different Apps on TOTALLY Different Platforms, 1 only HP UX, 1 , Two on Solaris ( on on 7 one on 2.51) One on DOS (Yes DOS it only ran there) and a couple on Windows, tied them all together through Vitria then hooked it up to a Web Front end, It was sooo slick the first VC that came to see our Proof of Concept (which was totally functional )gave us 15 Mil in VC
But alas another DotCom fatality, we went from 12 employees to 165 in a year......can you say BAM
How about Joram which is supposed to be pretty good.
--Stupidity is Self Curing!
They should check out tipc for message queueing. TIPC is a reliable messaging protocol designed for clusters. TIPC is fairly transparent whether a task is local or somewhere else on the network.
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
Horus and Ensemble allow for process groups. Doesn't that do a superset of this stuff? It is open source.
Horus is/was used to run the NYSE, and a prototype radar for the Aegis battlecruisers.
Why is it not good enough?
http://www.thebricktestament.com/the_law/when_to_
AMQ is not to be confused with ActiveMQ. ActiveMQ is an open-source Java JMS implementation. Sun's JMS API defines a unified interface to a messaging system. The problem with JMS is that it is Java centric and Sun did not specify a wire protocol or the binary message format. AMQ is an implementation of a messaging system that is cross language/platform, something that is, from what I know, currently only povided by commercial messaging implementations which are quite pricey.
It is the best way to get data from distributive systems to/from the mainframe. So... are they going to write a COBOL interface from 3270? Without IBM's help, I don't see that happening.
For distributive->distributive, web services is all you need.
And how can any of those things going wrong be prevented with another protocol and not SMTP?
1. server screws up the queue - possible on anything with badly coded software
2. messages delivered twice - each message has a unique message-id, allowing the detection of duplicates.
3. message stuck in a loop - possible with any routable store-and-forward system
4. same message repeated multiple times - see #2
5. spool being filled - possible with any store-and-forward system
Sounds like the same issues rehashed, SMTP or message queuing.
-molo
Using your sig line to advertise for friends is lame.
My company built and sold a message-oriented messaging product that merged the pub/sub and transactional models several years back, and it included guaranteed delivery. And yes, there was a lot of discussion about open sourcing the code base. The system is still in daily production in a bunch of Wall St. firms.
What these firms all wanted as a key requirement was clean integration with C/C++, and extreme performance. We had to handle thousands of messages (ranging into megabytes each) per second, with full guaranteed delivery, over IP networks and through firewalls. Oh, and none of the flakiness and admin problems of MQSeries.
None of the available JMS products could handle all these requirements.
I wrote my own AMQ implementation about 4 years ago for a project that I was working on. The project was done in Java on Weblogic 5.1 (flashback - I am working on W5.1 on my current contract today,) and message beans were just introduced but we did not have them in our implementation.
The project was for life insurance industry (for a little known company named Worldinsure that spent over 2.5*10^7 dollars on Aeron chairs and plasma monitors and - surprise, surprise - went belly up.) The app. would ask hundreds of questions that needed to be populated page by page. On some page the app would have to generate a PDF file, and since the Adobe pdf generator would crash the JVM after a couple of uses, we desynchronized it.
I wrote a message queue that was generic enough to start whatever action that was appropriate for the message type. It was not too difficult, a relational db (Oracle) was used as the message storage mechanism and I had 2 different options for message retrieval - message polling from the java code and Oracle triggers. The tricky part was the multithreaded message queue processor that could start any action - a java class in a seperate safe thread. Every message had a status - New, InProgress, Failed. Messages could be setup to be retried with a retry-decrimenting counter, so a message would become 'Failed', once the retry counter reached 0. Message types had priorities assigned to them, so some messages would take priority over others. To make sure that all messages eventually were drained, there was also a priority timeout. Messages could be 'peeked at' and retreived. Once a message queue processor decides to retrieve a message, this message is not deleted, but marked with status 'InProgress', so that other processors would not pick it up as a new message.
The interesting side-effect of this design was that it was possible to run as many message queue processors as one wished, thus an application could spread the load onto multiple computers (this adds redundancy and scalability.)
All of this can be implemented quite easily to follow JMS standard, but JMS standard is even simpler (in terms of functionality) than what this designed allowed.
You can't handle the truth.
For desktop users and departmental servers, integration into the kernel is just bloat. For financial institutions that use MQSeries to move real-time market and trade data, these things have to be tightly integrated down to the system level to gain the real-time guarentees and throughput required in those environments. MQSeries can be directly wired into S/390 or AIX at the system to level to provide the quality of service required (think a commodities broker for a major hedge fund initiating a trade with the wrong price and you get the idea). Without this capability, Linux and its brethern will have a hard time pentrating the core of the financial industry. It's also why Sonic and IBM can demand the amazing prices they get get for SonicMQ and MQSeries. For Redhat and Suse, it gives them a cost-effective answer to high dollar question. They can charge these banks 20% of the total cost of MQSeries for 24/7/365 support and make a killing.
Have you ever typed 5 quick messages in AIM...
You're throttled so that you don't kill the system.
Ok, so now put an API ontop where you are sending thousands of messages per second.
Now get really clever and have the API sign up for 1,000,000 AIM handles, pool them and round robin between them.
Sorry, thats not transactional. Transactional means that either the message will be delievered and accepted, or it won't. There is no in between state. The sender is notified of the results. There cannot be a gurantee of delivery. The receiving side may fail, but any steps that it performed will be wound back to its original state.
-molo
Using your sig line to advertise for friends is lame.
This whole thing is suspect anyway until they release their source code. It isn't "open-source" if the source is kept secret!
Have you read my blog lately?
FTP, email and databases can be used to implement low-volume, low-speed message queueing. There's nothing wrong with that - many web-based systems use that.
Where MQ plays a role is in the high-speed, low-latency version of this. I work with an MQ system that does 1500+ messages per second, transactionally, for prolonged periods of time, with a latency 0.1 second. And this runs on a 4-CPU Linux box.
MQ solves a different problem than a database. A database allows data updates and multiple readers. In MQ, each message has one reader and there are no updates. These different semantics allow vastly different performance.
Doubtful on the licensing. 90% of an MQ is in the backing store you use and the transactional logic, and much of that has been in the practice and the public domain since the mid-80s. The communications channel and routing logic is also mostly well known and understood. There's no magic voodoo here. You could build a reliable MQ tool on top of PostgreSQL, Oracle, or MS SQL Server with little effort, certainly not $1billion.
You have to remember, the average Slashdotter is a kid. They wouldn't have experience with real computer systems in the real world (like MQSeries). All they think of is their l33t AIM shit.
I'm sure if there was an age poll the average age is 17-20. Neophytes, babies that need their diaper changed, etc.
At my work, we use SonicMQ, which works pretty well. Its Java based, but has a C and COM API as well as a HTTP Server so you can post messages to it via HTTP. Anybody can write a messaging system, but getting one that fails over, clusters, routes, etc. is a bit more effort.
I've found messaging based systems to be really enjoyable to work with. They take a lot of thought, but you can massively scale (e.g., the Aggregator pattern). Also, they encourage loosely coupled systems.
Also, when you use XML as your message payload you can get very good interop (provided you pick a messaging system that provides APIs for the clients you need).
We tried to make "Web Services" work, but you really need messaging infrastructure (e.g., security, routing, guaranteed delivery). You can get a fairly cheap / robust integration architecture with a decent messaging system and an XML canonical message format.
It looks like the messaging layer will get commoditized. Looks like trouble for MQSeries, SonicMQ, Tibco, etc. Not sure where AMQ will go, but ActiveMQ is bound to go places since it is a part of Geronimo.
There is also (well at least on the Java side) an effort to drive standardization up the stack with JSR 208 - Java Business Integration. Its basically a spec for Enterprise Service Bus (ESB) / Service Oriented Architecture.
Amen! I have been crying. Sometimes I even run into people who are ignornat of the history and think EJBs invented the whole idea of distributed objects. It drives me crazy. We had a project at work built with C++/CORBA. It was pretty mature and worked really well. There was another project built on EJBs and we were suppose to get the two products talking. At the first meeting, they had a whole slideshow showing why we should rewrite our stuff with EJBs, showing all the advantages and how you could have remote objects and other things "not possible in C++". We were laughing until we realized there are MANY people who believe that. After we explained to them that their EJBs are more or less CORBA, they looked at us funny and I don't think they really believed us. They figured since they had just discovered distributed objects, it must be new and we were some crazy people who did not understand the power of EJBs...
It's a shame. CORBA is really useful and I think very easy to work with. Granted, there are a few things to learn if you're using the C++ mapping, but if you take the time, it's actually pretty straightforward and consistent.
It seems whenever something is at the point where it is mature and works well, it is thrown out and some new thing is invented to replace it - and they end up re-inventing the wheel many times - usually with less then spectacular results.