IBM Websphere MQ is a messaging and queuing middleware provider that can be leveraged via the Java Message Service (JMS) specification. In this manner, Pentaho Data Integration exposes IBM MQ Consumer and Producer transformation steps allowing access to produce and consume messages through IBM Websphere MQ. IBM Websphere MQ provides this functionality through IBM's implementation of JMS, called the Websphere MQ classes for JMS.
As the MQ steps are layered atop the JMS Consumer and Producer transformation steps, there is overlap and great similarities between the documentation for JMS and IBM Websphere MQ. For the sake of usable documentation, some of the JMS documentation is repeated here.
With JMS (and as such, also with IBM Websphere MQ) it is possible to process messages in two basic methods:
Queues
A message can be sent to a specified MQ Queue and can be received by exactly one consumer. If there are no consumers available at the time the message is sent it will be kept until a consumer is available that can process the message. This is also know as a point-to-point model.
Topics
In Websphere MQ a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message. Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message. This is also known as a publish/subscribe model.
You need to locate the Websphere MQ classes for JMS Java libraries from your IBM Websphere MQ installation. You can also find these libraries in your IBM Websphere MQ Client SupportPac. The Websphere MQ Java libraries version that the PDI plugin steps were built against is 7.0.1.9. The libraries that you must have available for distribution into the PDI JMS plugin are:
IBM licensing prevents Pentaho from distributing these libraries directly, hence the need to add them post install of Pentaho Data Integration.
To install the plugin to Pentaho Data Integration Spoon client:
To install the plugin to the Pentaho Data Integration Server:
Follow the steps above. The location of the pdi-jms-plugin directory is different; look for the plugin directory under <pdi-server-install>/data-integration-server/pentaho-solutions/system/kettle/plugins directory.
To install the plugin to the Pentaho Business Analytics Server:
Follow the steps above. The location of the pdi-jms-plugin directory is different; look for the plugin directory under <ba-server-install>/biserver-ee/pentaho-solutions/system/kettle/plugins directory.
The following set of sample transformations should help get you started using your new messaging steps.
In the following sample solution we use the Websphere MQ Consumer and Websphere MQ Producer steps and use Websphere MQ Version 7.0 http://activemq.apache.org as a Websphere MQ server.
This was tested with PDI EE Version 4.3 GA and Websphere MQ Version 7.0.
Setup for the sample solutions:
Execute the transformation MQ Producer - QUEUE.ktr
This transformation creates 10 test messages and puts these into the message server queue JMSQueue.
You can see that the queue is created and the messages have arrived at the server. We used IBM MQExplorer to monitor our sample server.
Now retrieve the messages by the transformation MQ Consumer - QUEUE.ktr
After the execution, the messages are removed from the queue. See further commentary on this in the Notes on Transactional Behavior section.
The ongoing for the topics is pretty much the same, but you need to start the consumer first! It is also possible to start multiple consumers as subscribers.
The transactional behavior is set to AUTO_ACKNOWLEDGE by default in the IBM Websphere MQ steps: With this acknowledgment mode, the session automatically acknowledges a client's receipt of a message either when the session has successfully returned from a call to receive or when the message listener the session has called to process the message successfully returns.
This means, producers and consumers are decoupled from each other and these messages are loosely coupled and asynchronous.
By default, the messages are always removed from the queue after a successful receipt of the message by the JMS client - Pentaho Data Integration in our case. We are working toward a means of delaying the removal of the message until an acknowledgment has been made by the client. This feature is NOT in this release at the time of this writing, please see PDI-7403 for the feature request.