Slashdot Mirror
Open Source Message Queuing System
psicode writes "John Davies has announced AMQ, an effort at JPMorgan Chase & Co. to create an open-source message queuing system that can compete with proprietary message systems like IBM MQSeries and Tibco/RV. The announcement was made at the
annual conference Web
Services on Wall Street during Davies' presentation on February 1. eWeek has an article today with more details and some funny statements about Red Hat, SuSE and Sun possibly integrating AMQ into their "kernel". If JPMorgan Chase & Co. follows through with their announcement and they come up with a suitable open-source license, AMQ could become the Apache of messaging systems."
I think web services are going to be the next big thing.
yay
I thought that's what CodeHaus's ActiveMQ was for.
CodeHaus rocks, even if they are overly java biased for my tastes.
The trick is in Web Services, I was lead developer on a LARGE project during the DotCom Era (2000 ish) that used Vitria, a great system and its interop capablities blew my mind at first but if we could have utilized web services we would have halved developemnt time, not to mentio forgoing a 1 Million $ Vitria Liscence.
It was cool though, we took about 6 different Apps on TOTALLY Different Platforms, 1 only HP UX, 1 , Two on Solaris ( on on 7 one on 2.51) One on DOS (Yes DOS it only ran there) and a couple on Windows, tied them all together through Vitria then hooked it up to a Web Front end, It was sooo slick the first VC that came to see our Proof of Concept (which was totally functional )gave us 15 Mil in VC
But alas another DotCom fatality, we went from 12 employees to 165 in a year......can you say BAM
Buzzword buzzword buzzword, incorrectly used terminology, buzzword buzzword.
wdd
How about Joram which is supposed to be pretty good.
--Stupidity is Self Curing!
"Message queuing is a communication tool that allows applications to reliably interconnect in a distributed environment where one of the applications may or may not be available at any given time. The queue acts as a holding container for messages as they are sent between applications. The applications send messages to and read messages from queues to communicate back and forth. An application writes a message to a queue, which will then be received and processed by another application at some point determined by the receiving application. This type of communication is designed for asynchronous use where the applications involved are not waiting for an immediate response from the other end."
9 11
From http://www.developer.com/net/asp/article.php/3297
Perhaps it's worth mentioning since they are integrating an app into the kernel. At least that's what it sounds like. Well if you wonder why kernels seem to get more and more bloated...
You mean they can't just fork POPFile::MQ and use it :-)
John.
Wall Street is also looking for a new Open Source Ethics and Honesty implementation as current proprietary implementations seem to be faulty, buggy, and expensive to maintain.
We have had MQSeries at each IBM implementation that I have worked at (SAP software).
My perception of MQ is that we would be better off throwing it away and FTPing files back and forth while using SAP's job control to kick things off. I cannot imagine how many weeks we have lost in a production environment because of some fiddly piece of junk involved in the MQ process decided to hiccup.
"they come up with a suitable open-source license, AMQ could become the Apache of messaging systems."
If they want to be the Apache, then they'll have to use the Apache license too. Effects have causes.
-russ
Don't piss off The Angry Economist
Have you ever noticed anyone who doesnt know the answer to what the person asks they tell them in a (rude) way to look it up on a search engine or to read RTFM?
Crap. In my rush to be a smart ass, I forgot to select Plain Old Text. Message was supposed to read:
Might I suggest exercising your Google skillz?
Try these search terms:
message queuing
MQSeries
MSMQ
Java MQ
what is a message queue?
Thanks for your patience.
The Spoon
Updated 6/28/2011
Message queueing is application-to-application messaging, which once-only guaranteed delivery. In addition, both the reading and writing of messages can be done transactionally, with a transaction spanning multiple queue operations.
Think of it like a database where an application can write a record that can be read and committed by exactly one reader; however, if that reader crashes before committing, another reader can pick up that record and try to commit it, etc.
Financial services firms use this tie their many apps together.
Messaging is transactional, with guaranteed once-only delivery. The messaging server (queue manager) uses database technology such as write-ahead logging to achive this.
Implementing this across TCP/IP generally requires a lon-lived conenction, as you have seperate read/write/commit/backout operations that belong together. Doing so over HTTP is theoretically feasible, but a long-lived TCP connection makes it a lot easier. Compare to databases... insert/commit isn't over HTTP either.
As a voting libertarian, I whole-heartedly encourage companies to actively develop whatever technical solution that best fits their needs. This is one example of how open source software really fits in with the libertarian world view. It's not the "free software versus only commercial software," but a mix and match that lets both coexist. If someone wants to develop their own software, then let them and don't even begrudge them the right to give it away.
This is what really pisses me off about the copyright expansionists. They don't want people to be able to easily develop software for free. Copyright expansionism, as pushed by groups like the "Progress and Freedom Foundation" is not about freedom, it's about protecting business against hobbyists and other "asymmetric competition." I bet it makes such groups' heads spin to think that one of the biggest corporations on Wall Street is now about to dive head first into open source development and that the result of this development could send shockwaves through a segment of the commercial software industry.
The freedom to write open source software is the same freedom to write closed source software. If you erode the foundation for the former, then you have no right to demand respect for the latter. That is why when the big copyright cartels lobby Congress, I see nothing wrong in doing things like buying academic licenses for software and taking them out into the real world. The copyright holder makes no profit on the academic license, only the seller does.
Click here or a puppy gets stomped!
an open-source message queuing system that can compete with proprietary message systems like IBM MQSeries
Somewhere in IBM's headquarters there is a camel. A straw has just been placed on its back.
Direct away from face when opening.
They should check out tipc for message queueing. TIPC is a reliable messaging protocol designed for clusters. TIPC is fairly transparent whether a task is local or somewhere else on the network.
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I have noticed the phenomenon you mention, though I would hardly call it "anyone." I suppose I could have elaborated on the details of how a system-based message queuing system is designed to be a lossless, connectionless method for intrasystem messaging, much like a small town post office, where one piece of software delivers a message, often packaged as XML or CSV, and marks it for entry into a particular queue or stack. When other pieces of software are subscribed to the queue in question, they pick up the messages that get registered there. The messages are either destroyed on pick up or stay in the queue, depending on what type the queue is. But all this is just a surface look at how MQs are used and how they work. A layman's introduction, it might be said. It also doesn't cover the fact that they can be used as the back-end for IM systems, if you write the appropriate client software. I was thinking that this person might like to have their choice of in-depth articles to read that would educate them far more than I was willing to in the short time alloted to me for reading Slashdot each day.
But thanks for your concern. I'm done trolling for now, how about you?
The Spoon
Updated 6/28/2011
Some years ago a company I worked for was quoted one hundred thousand dollars as the cost to license IBM MQ Series messaging. There could be some serious savings in using an OSS product like this.
By the technical ignorance of some of the posters here... Before you post, please make sure you have at least an inkling about what you're talking about! A Message-Oriented Middleware system has absolutely nothing to do with IM, or Web Services (except in a very indirect way), or even REST systems. /. seems to be in dire need of real geeks, it seems...
I work for a major airline (and the only one that has always been profitable =)) Because of the nature of our business, we have to use both Tibco and MQSeries for our communications. We must use both because of the transactional nature of running an airline (communication to / from the aircraft) and the non-transactional (Sabre / Ticketing). If there was a product that gave us the ability to do guaranteed / clustered messaging in both a synchronous and synchronous environments we would be all over it. In the end we end up with MQSeries butting messages on a Tibco bus that everyone uses as a pub / sub system. This system is used for everything from liqueur kits to engine maintenance to baggage and flight plans. If there was a combination of point to point (MQSeries) and Tibco (pub/sub) I would be in heaven!
Stranded.org
Horus and Ensemble allow for process groups. Doesn't that do a superset of this stuff? It is open source.
Horus is/was used to run the NYSE, and a prototype radar for the Aegis battlecruisers.
Why is it not good enough?
http://www.thebricktestament.com/the_law/when_to_
>This is what really pisses me off about the copyright expansionists.
I really appreciate source code licenses that actually let you use the source code permanently such as BSD.
Copyright expansionist vehicles, such as GPL, end up costing too much in the long run.
Did you read the article? The whole point of this project, is they need something that can support more than just Java! As much as the people on the TSS.com like to feel, Java is not always the best solution and sometimes you do need to write some high perf. sensitive code in unmanaged code like C/C++. You can't do that with a JMS based solution, at least not easily.
What is a message queue, why should I care, what does this have to do with anything, what are they talking about? In short: what kinds of messages are we talking about, what are sending them to what, and why do we need to queue them?
"A message queue is a queue onto which messages can be placed."
thanks a fucking lot google
-- 'The' Lord and Master Bitman On High, Master Of All
It is the best way to get data from distributive systems to/from the mainframe. So... are they going to write a COBOL interface from 3270? Without IBM's help, I don't see that happening.
For distributive->distributive, web services is all you need.
GPL = something for something.
Changa hates change.
Keep in mind that Wall Street was probably *the* major engine driving Web applications into the mainstream in the 1990s. One reason Wall Street went crazy touting the Web as an unlimited business panacea was that it was actually like that for its own industry. By the time they promoted it in 1995, Wall Street shops had been churning out httpd patches, CGI apps, Perl revisions and code, DB connection software and techniques, TCL patterns, and all kinds of R&D that underpinned much of the development momentum. Message Queueing has a similar arc, because Wall Street both inhabits the bloodiest edge on which they can survive, and has the least tolerance for failure. The flakiness of network messaging is the root of some of the worst inhibition on network growth among users and deployers - Wall Street has been there, and has answers. Let's use them.
--
make install -not war
My company built and sold a message-oriented messaging product that merged the pub/sub and transactional models several years back, and it included guaranteed delivery. And yes, there was a lot of discussion about open sourcing the code base. The system is still in daily production in a bunch of Wall St. firms.
What these firms all wanted as a key requirement was clean integration with C/C++, and extreme performance. We had to handle thousands of messages (ranging into megabytes each) per second, with full guaranteed delivery, over IP networks and through firewalls. Oh, and none of the flakiness and admin problems of MQSeries.
None of the available JMS products could handle all these requirements.
No, it's akin to JMS (Java Messaging System) or, say, the TIBCO/Rendezvous system used to broadcast trading information to NASDAQ clients.
Here at my workplace, we use "guaranteed messaging" to integrate disparate systems such as SAP R/3, as iSeries/AS400 running BPCS, a bespoke warehousing system. Messages (i.e., details of a transaction, for example an XML doc. containing details of a sales order) are transmitted between systems to keep them in sync. If the target system is down the messages are queued for delivery in the messaging system itself, and delivered later.
Not a great explanation, but I hope it helps.
XML blaster is an open source message oriented middleware that has bindings for multiple languages (C,C++, Java, PHP, Python, Perl).
I have never used it, but it's been available since a long time.
My heart is pure, but make no mistake, it's pure evil
I wrote my own AMQ implementation about 4 years ago for a project that I was working on. The project was done in Java on Weblogic 5.1 (flashback - I am working on W5.1 on my current contract today,) and message beans were just introduced but we did not have them in our implementation.
The project was for life insurance industry (for a little known company named Worldinsure that spent over 2.5*10^7 dollars on Aeron chairs and plasma monitors and - surprise, surprise - went belly up.) The app. would ask hundreds of questions that needed to be populated page by page. On some page the app would have to generate a PDF file, and since the Adobe pdf generator would crash the JVM after a couple of uses, we desynchronized it.
I wrote a message queue that was generic enough to start whatever action that was appropriate for the message type. It was not too difficult, a relational db (Oracle) was used as the message storage mechanism and I had 2 different options for message retrieval - message polling from the java code and Oracle triggers. The tricky part was the multithreaded message queue processor that could start any action - a java class in a seperate safe thread. Every message had a status - New, InProgress, Failed. Messages could be setup to be retried with a retry-decrimenting counter, so a message would become 'Failed', once the retry counter reached 0. Message types had priorities assigned to them, so some messages would take priority over others. To make sure that all messages eventually were drained, there was also a priority timeout. Messages could be 'peeked at' and retreived. Once a message queue processor decides to retrieve a message, this message is not deleted, but marked with status 'InProgress', so that other processors would not pick it up as a new message.
The interesting side-effect of this design was that it was possible to run as many message queue processors as one wished, thus an application could spread the load onto multiple computers (this adds redundancy and scalability.)
All of this can be implemented quite easily to follow JMS standard, but JMS standard is even simpler (in terms of functionality) than what this designed allowed.
You can't handle the truth.
What JP Morgan and other Wall Street firms use message queuing software for is trading in the financial markets. So if the software bugs out, they stand to lose millions of dollars per minute or more depending on the size of the trades they are executing. So I see this story as a positive for open source software because if Wall Street people think they can get a usable mission critical piece of software out of the open source community then no other potential target for OSS is invulnerable.
"Lack of technical competence coupled with the arrogance of power, as usual, leads to no good end."
For desktop users and departmental servers, integration into the kernel is just bloat. For financial institutions that use MQSeries to move real-time market and trade data, these things have to be tightly integrated down to the system level to gain the real-time guarentees and throughput required in those environments. MQSeries can be directly wired into S/390 or AIX at the system to level to provide the quality of service required (think a commodities broker for a major hedge fund initiating a trade with the wrong price and you get the idea). Without this capability, Linux and its brethern will have a hard time pentrating the core of the financial industry. It's also why Sonic and IBM can demand the amazing prices they get get for SonicMQ and MQSeries. For Redhat and Suse, it gives them a cost-effective answer to high dollar question. They can charge these banks 20% of the total cost of MQSeries for 24/7/365 support and make a killing.
Have you ever typed 5 quick messages in AIM...
You're throttled so that you don't kill the system.
Ok, so now put an API ontop where you are sending thousands of messages per second.
Now get really clever and have the API sign up for 1,000,000 AIM handles, pool them and round robin between them.
What does it do? Virtually nothing. Just takes a few buytes of data and sends it from here to there. Thats the easy part - the hard part is that it does so with absolutely guaranteed results - if you send it, you know they will get it. Imagine chains of more or less reliable machines, software stacks and networks all linking together sender and receiver, and the horrendous business impact of failure of even a single message, and the non-negotiable need for auditing, and you start to see why there might just be a little tiny bit of complexity in there.
Some systems throw a few extra dyanmics in on top, such as a publish/subscribe model which takes you beyond just sender and receiver, but the essence of these things is not functionality but utter, total reliability.
Which seems to me why an open source model is not, in itself, an obvious way to produce such a piece of software. More than anything else - except perhaps database systems - people look to the vendors of these things to guarantee 100% reliability. Such a project, if open source, would require a big name behind it before any corporate would use it - and large corporates by their nature are the normal customers for these things.
[x] auto-moderate all posts by this user as insightful
Everyone's asking "Why?" and debating the technical bits. I'll tell you why. So they can get a deep discount on MQSeries from IBM.
It's been out for five years on freshmeat, and supports CORBA, RMI, XML-RPC, nativ socket, or persistent HTTP.
See the link at http://freshmeat.net/projects/xmlblaster/.
- jon
Ganymede, a GPL'ed metadirectory for UNIX
There's been a opensource, cross-platform solution for a number of years (shameless plug), http://www.osmq.org/index.html
~~~ They call me Little John, but don't let the name fool you...in real life I'm very big.
FTP, email and databases can be used to implement low-volume, low-speed message queueing. There's nothing wrong with that - many web-based systems use that.
Where MQ plays a role is in the high-speed, low-latency version of this. I work with an MQ system that does 1500+ messages per second, transactionally, for prolonged periods of time, with a latency 0.1 second. And this runs on a 4-CPU Linux box.
MQ solves a different problem than a database. A database allows data updates and multiple readers. In MQ, each message has one reader and there are no updates. These different semantics allow vastly different performance.
In the sense of MQSeries and Rendevous, it's exactly what you describe except with guaranteed delivery. I know they sound a bit useless today but it's like transaction support for network communications among other things.
i nd ex.cfm?FeatureID=731
Let me give you a PERFECT example of how I've seen MQ Series used.
DB2 has/had (gone with Stinger, I think) something called userexit. This was a compiled program that was called whenever DB2 was in archival log mode.
In our case, we use a TSM (Tivoli Storage Manager) userexit. When DB2 is ready to archive a log file, it ships the log to TSM and it's on tape. One company I've heard does MQ Series userexits.
DB2 userexit's to archive the log and ships it to MQ Series. MQ Series breaks it up into smaller chunks (64k maybe? I can't remember) and sends it to a remote MQ Series server. From that server, the logs are stored locally and backed up or used to roll-forward a read-only copy of database.
The upside is that this remote MQ server is across the country at another datacenter and you know the log files got there and they got there the way they were when the MQ server got them.
Check this link. I think it's provided the best simple description of Message Queuing I've seen yet:
http://www.techworld.com/applications/features/
"Fighting the underpants gnomes since 1998!" "Bruce Schneier knows the state of schroedinger's cat"
At my work, we use SonicMQ, which works pretty well. Its Java based, but has a C and COM API as well as a HTTP Server so you can post messages to it via HTTP. Anybody can write a messaging system, but getting one that fails over, clusters, routes, etc. is a bit more effort.
I've found messaging based systems to be really enjoyable to work with. They take a lot of thought, but you can massively scale (e.g., the Aggregator pattern). Also, they encourage loosely coupled systems.
Also, when you use XML as your message payload you can get very good interop (provided you pick a messaging system that provides APIs for the clients you need).
We tried to make "Web Services" work, but you really need messaging infrastructure (e.g., security, routing, guaranteed delivery). You can get a fairly cheap / robust integration architecture with a decent messaging system and an XML canonical message format.
It looks like the messaging layer will get commoditized. Looks like trouble for MQSeries, SonicMQ, Tibco, etc. Not sure where AMQ will go, but ActiveMQ is bound to go places since it is a part of Geronimo.
There is also (well at least on the Java side) an effort to drive standardization up the stack with JSR 208 - Java Business Integration. Its basically a spec for Enterprise Service Bus (ESB) / Service Oriented Architecture.
I bet it'll take at least 5 years before this new thing makes any inroads in financial customers, but this must be really scary for IBM's MQSeries folks.
Things like MQ Series are bread-and-butter of many sales reps as there's no serious competition to their solution - I haven't talked to many banks, but those to which I did all use MQSeries.
OSS is moving from the edge to the data center, it's already half-way there. Five more years and banks will be able to get 80% of their apps under GPL license.