IBM MQ does messages of up to 100MB. Each queue can hold up to 2 TeraByte of messages, and a queue manager server can hold tens of thousands of queues. You do of course need the disk space to hold all this:-)
So MQ is often used to send file-like data between institutions. Especially since it has built-in features like Confirm-On-Arrival and Confirm-On-Delivery. Note that MQ may not be as fast as FTP in this... but it is transactional, once-only, guaranteed delivery. When the message data is about large amounts of money, the overhead is generally worth it.
The biggest issue with two-phase commits occurs when multiple servers are involved. A crash of one of the servers involved in an XA transaction can lead to that transaction being in limbo until that server recovers; a sufficiently bad crash may lead to the transaction being in limbo until an admin manually kills it. In many cases, the other components in the transaction will not be able to contiunue in such a scenario; thius leads to extended outages. Wall Street folks hate that...
There's three parties involved: writer, server (queue manager) and reader. The writer puts a message to a queue; the server writes it to a transaction log as an uncommitted message. The writer commits; thhe server writes a commit message to the transaction log, then informs the writer that the commit suuceeded.
Now the reader can see the message. The server marks the provisional read in the transaction log; no other readre can see the message. Once the reader commits, the commit is written to the transaction log, and the message is slated for deletion.
A crash at the writer/reader leads the server to back out uncommitted work (the write or read is undone unless committed). A crash at the server leads to a reply of the transaction log after the server restarts; again, if the transaction log doesn't contain the commit, the operation in progress is undone.
There are race conditions: a writer or reader can issue a commit and then a crash can occur (reader/writer, server, network). In such a case, the reader/writer may not know whether the commit succeeded. They key si that the server has a definite opinion on this that will survive it crashing...
MQ does do store-and-forward. It puts a message on a dead-letter queue (DLQ) if it is certain that a message cannot be delivered - i.e. if the destination queue on the remote system doesn't accept a message of the relevant size, or doesn't allow the user write permission. If the target queue is full, the system will normally pause and try to re-deliver (this can be configured).
The DLQ is essential becuase the store-and-forward system delivers messages in order - so a single stuck message might block a communications flow.
Finally, messages on the DLQ include the complete origianl delivery info. So you can fix the undelrying problem and redeliver. I have a perl script of 100 lines to do so... not rocket science.
FTP, email and databases can be used to implement low-volume, low-speed message queueing. There's nothing wrong with that - many web-based systems use that.
Where MQ plays a role is in the high-speed, low-latency version of this. I work with an MQ system that does 1500+ messages per second, transactionally, for prolonged periods of time, with a latency 0.1 second. And this runs on a 4-CPU Linux box.
MQ solves a different problem than a database. A database allows data updates and multiple readers. In MQ, each message has one reader and there are no updates. These different semantics allow vastly different performance.
Most MQ systems use database technologies at their core. A transaction manager and write-ahead logging, plus an execution manager (watchdog).
Using some of the more advanced MySQL components for this (InnoDB comes to mind) should be feasible. It might not meet stringent performance requirements - MQSeries can do 1500+ transactional messages per second, sustained.
POSIX message queues are a non-transactional form of Inter-Process Communication. MQ systems are transactional between systems. They solve a very different problem.
The nice thing of IBM MQ is that there is a perl API on CPAN. I own it, I ought to know:-)
If the AMQ system is C-based, writing a perl API should be easy. And my boss might even pay me to do so, just like I'm paid to maintain the perl MQSeries API:-)
Messaging is transactional, with guaranteed once-only delivery. The messaging server (queue manager) uses database technology such as write-ahead logging to achive this.
Implementing this across TCP/IP generally requires a lon-lived conenction, as you have seperate read/write/commit/backout operations that belong together. Doing so over HTTP is theoretically feasible, but a long-lived TCP connection makes it a lot easier. Compare to databases... insert/commit isn't over HTTP either.
It is very encouraging that Microsoft feels required to come out with Open Source competition to add tools to Windows. If this is competition for the GIMP, what can we expect once OpenOffice / KOffice / the GNOME office tools become as popular as the GIMP?
The techniques described (Phase Trees) were tried in databases in the 1970s (see the survey article by Verhofstadt in ACM Computing Survey, volume 10, issue 2, 1978). Databases have moved away from them because they either do not offer undo (roll-back) support, or do so at the cost of efficiently.
One of the big innovations in databases of the late 1970s is that a transaction log can actually reduce I/O costs because you can (a) cluster unrelated writes and (b) avoid seeks - becuase all updates are written to one log, and the main data is updated lazily.
If you don't want transactions and roll-back, the "careful replacement" strategies are definitely the way to go.
Well, read Jack Vance's "The Demon Princes", which has exactly that - a hostage interchange which auctions off hostages if the ransom isn't being paid in time...
Slashdot Mirror
IBM MQ does messages of up to 100MB. Each queue can hold up to 2 TeraByte of messages, and a queue manager server can hold tens of thousands of queues. You do of course need the disk space to hold all this :-)
So MQ is often used to send file-like data between institutions. Especially since it has built-in features like Confirm-On-Arrival and Confirm-On-Delivery. Note that MQ may not be as fast as FTP in this... but it is transactional, once-only, guaranteed delivery. When the message data is about large amounts of money, the overhead is generally worth it.
The biggest issue with two-phase commits occurs when multiple servers are involved. A crash of one of the servers involved in an XA transaction can lead to that transaction being in limbo until that server recovers; a sufficiently bad crash may lead to the transaction being in limbo until an admin manually kills it. In many cases, the other components in the transaction will not be able to contiunue in such a scenario; thius leads to extended outages. Wall Street folks hate that...
There's three parties involved: writer, server (queue manager) and reader. The writer puts a message to a queue; the server writes it to a transaction log as an uncommitted message. The writer commits; thhe server writes a commit message to the transaction log, then informs the writer that the commit suuceeded.
Now the reader can see the message. The server marks the provisional read in the transaction log; no other readre can see the message. Once the reader commits, the commit is written to the transaction log, and the message is slated for deletion.
A crash at the writer/reader leads the server to back out uncommitted work (the write or read is undone unless committed). A crash at the server leads to a reply of the transaction log after the server restarts; again, if the transaction log doesn't contain the commit, the operation in progress is undone.
There are race conditions: a writer or reader can issue a commit and then a crash can occur (reader/writer, server, network). In such a case, the reader/writer may not know whether the commit succeeded. They key si that the server has a definite opinion on this that will survive it crashing...
MQ does do store-and-forward. It puts a message on a dead-letter queue (DLQ) if it is certain that a message cannot be delivered - i.e. if the destination queue on the remote system doesn't accept a message of the relevant size, or doesn't allow the user write permission. If the target queue is full, the system will normally pause and try to re-deliver (this can be configured).
The DLQ is essential becuase the store-and-forward system delivers messages in order - so a single stuck message might block a communications flow.
Finally, messages on the DLQ include the complete origianl delivery info. So you can fix the undelrying problem and redeliver. I have a perl script of 100 lines to do so... not rocket science.
FTP, email and databases can be used to implement low-volume, low-speed message queueing. There's nothing wrong with that - many web-based systems use that.
Where MQ plays a role is in the high-speed, low-latency version of this. I work with an MQ system that does 1500+ messages per second, transactionally, for prolonged periods of time, with a latency 0.1 second. And this runs on a 4-CPU Linux box.
MQ solves a different problem than a database. A database allows data updates and multiple readers. In MQ, each message has one reader and there are no updates. These different semantics allow vastly different performance.
Most MQ systems use database technologies at their core. A transaction manager and write-ahead logging, plus an execution manager (watchdog).
Using some of the more advanced MySQL components for this (InnoDB comes to mind) should be feasible. It might not meet stringent performance requirements - MQSeries can do 1500+ transactional messages per second, sustained.
POSIX message queues are a non-transactional form of Inter-Process Communication. MQ systems are transactional between systems. They solve a very different problem.
The nice thing of IBM MQ is that there is a perl API on CPAN. I own it, I ought to know :-)
:-)
If the AMQ system is C-based, writing a perl API should be easy. And my boss might even pay me to do so, just like I'm paid to maintain the perl MQSeries API
Messaging is transactional, with guaranteed once-only delivery. The messaging server (queue manager) uses database technology such as write-ahead logging to achive this.
Implementing this across TCP/IP generally requires a lon-lived conenction, as you have seperate read/write/commit/backout operations that belong together. Doing so over HTTP is theoretically feasible, but a long-lived TCP connection makes it a lot easier. Compare to databases... insert/commit isn't over HTTP either.
It is very encouraging that Microsoft feels required to come out with Open Source competition to add tools to Windows. If this is competition for the GIMP, what can we expect once OpenOffice / KOffice / the GNOME office tools become as popular as the GIMP?
The techniques described (Phase Trees) were tried in databases in the 1970s (see the survey article by Verhofstadt in ACM Computing Survey, volume 10, issue 2, 1978). Databases have moved away from them because they either do not offer undo (roll-back) support, or do so at the cost of efficiently.
One of the big innovations in databases of the late 1970s is that a transaction log can actually reduce I/O costs because you can (a) cluster unrelated writes and (b) avoid seeks - becuase all updates are written to one log, and the main data is updated lazily.
If you don't want transactions and roll-back, the "careful replacement" strategies are definitely the way to go.
Well, read Jack Vance's "The Demon Princes",
which has exactly that - a hostage interchange which auctions off hostages if the ransom isn't being paid in time...