IBM MQ Series Adapter Configuration
Overview
IBM MQ Transport Adapter (hereafter MQ TA) is an add-in for FIXServer that allows clients to communicate with FIXServer over IBM MQSeries middleware. The document contains full instructions on how to install MQ TA for testing.
Starting FixEdge version 6.10.0 it is recommended to use IBM MQ Client library version 8.x.
The older and newer versions may lead to MQ adapters failures
MQ TA is responsible for:
- Establishing and maintaining sessions with MQ Series Server to communicate with Clients.
- Transmitting messages to its clients.
- Delivering messages from clients.
- Facilitating monitoring and administration.
Monitoring and Administration
Information about the configured TA's session parameters list and dynamic state is sent to the subscribed monitoring application when FIXEdge starts.
When the TA session dynamic state is changed FIXEdge sends the update to the subscribed monitoring application.
MQ TA provides the following monitoring information:
- Status
- Number of messages sent
- Number of messages received
- Time of the last successful message sending
- Time of the last successful message receiving
- List of clients
Integration Guide
Installation
IBM MQ Transport Adapter is present on the FIXEdge C++ package in the bin directory.
MQ Adapter requires IBM MQ client libraries installed in the host with FIXEdge, the libraries can be found on the official IBM MQ site.
Make sure that path to IBM MQ client libraries is added to the path environment variable.
Configuration
IBM MQ Transport Adapter can be configured using FIXICC or directly by editing the FIXEdge.properties file of the correspondent FIX Edge instance. All properties to be set up are described in the table below. You can also use the FIXEdge.properties file from the package as an example.
Make sure that configuration of the adapter correlates with the MQ Series Server parameters.
Property Name | Default Value | Description |
TransportLayer.MQAdaptor.Description | MQ Transport Adapter DLL | Description of the transport adapter. |
TransportLayer.MQAdaptor.DllName | bin/MQTAAddin.dll | Path to the transport adapter module. In case this parameter is not specified, the TransportLayer.MQAdaptor.AdapterId is applied to define the adapter's library by ID. |
TransportLayer.MQAdaptor.AdapterId | This parameter is applicable since FIXEdge C++ 6.14.0 version. The parameter to define the adapter's library by ID. In case this parameter is not specified or TransportLayer.MQAdaptor.DllName parameter is specified too, the TransportLayer.MQAdaptor.DllName is applied. Required value: MQ | |
TransportLayer.MQAdaptor.TimeIntervalBeforeReconnect | 1000 | Time interval in milliseconds between reconnect attempts. |
TransportLayer.MQAdaptor.StorageDirectory | FixEdge1/log | Required. Logging directory for storage of outgoing messages. |
TransportLayer.MQAdaptor.BufferSizeForMessage | 65536 | Message buffer size. The max value is 4194304. 0 means a default value 64 * 1024. |
TransportLayer.MQAdaptor.MQ_HOSTNAME | localhost | MS Series Server hostname. |
TransportLayer.MQAdaptor.MQ_PORT | 1414 | MS Series Server port. |
TransportLayer.MQAdaptor.MQ_MANAGER_NAME | TestMQ | MS Series manager name. |
TransportLayer.MQAdaptor.MQ_CHANNEL_NAME | TestConnection | MS Series Channel name. |
TransportLayer.MQAdaptor.MQ_CCSID | 437 | MS Series Server CCSID. The coded character set identifier to use with the WebSphere MQ queue. |
TransportLayer.MQAdaptor.SslKeyRepository | /var/mqm/ssl/key | Path to SSL certificate storage file without extension (SCKR, MQSSLKEYR) e.g. /var/mqm/ssl/key or C:\\Program Files\\IBM\\WebSphere MQ\\ssl\\key. |
TransportLayer.MQAdaptor.SslCipherSpecification | TRIPLE_DES_SHA_US | SSL CipherSpec name (SSLCIPH), required to use SSL, e.g. TRIPLE_DES_SHA_US. |
TransportLayer.MQAdaptor.SslClientAuthentication | false | Remote peer authentication (SSLCAUTH). |
TransportLayer.MQAdaptor.SslPeerName | CN=QMGR.*, OU=IBM, OU=WEBSPHERE | Remote peer DN filter (SSLPEER). |
TransportLayer.MQAdaptor.WaitIntervalForGet | 1000 | Time interval in milliseconds to wait between polling messages on MQ Series Server. |
TransportLayer.MQAdaptor.NumAttemptReconnect | 10 | The number of reconnect attempts to the MQ Series Server. |
TransportLayer.MQAdaptor.SessionNumber | 1 | The number of sessions with MQ Series. Each session is a set of MQ Series queues. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.ClientID | MQClient | MQ session ID. |
TransportLayer.MQAdapter.Session.<SessionNumber>.FIXVersion | FIX44 | In every session, FIX messages use a particular version of the FIX protocol. Use this parameter to set the version of the FIX protocol for a given session. Valid values: |
TransportLayer.MQAdaptor.Session.<SessionNumber>.SmartXMLProcessor | true | Enables mode of processing XML/FIXML messages:
|
TransportLayer.MQAdaptor.Session.<SessionNumber>.ToClientQueue | MQClient_to | The name for the Queue dedicated for messages coming from clients to MQ via FIXEdge C++. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.FromClientQueue | MQClient_from | The name for the Queue dedicated for messages coming from MQ to clients via FIXEdge C++. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.ErrorQueue | MQClient_error | The name for the Queue dedicated for messages, which cannot be handled because of errors. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.SenderID | - | Optional. SenderCompID (Tag = 49) in the received message is set to the property value. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.TargetID | - | Optional. TargetCompID (Tag = 56) in the received message is set to the property value. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.MessageDataFormat | MQFMT_NONE | This parameter is applicable since FIXEdge C++ 6.15.1 version. Optional. Defines the format of the message data to be sent to IBM MQ via the IBM MQ TA session. Ignored for incoming messages. Valid values: MQFMT_NONE | MQFMT_STRING |
TransportLayer.MQAdaptor.Session.<SessionNumber>.LogIncomingMessages | false | This parameter is applicable since FIXEdge C++ 6.15.1 version. Optional. Defines whether incoming messages from IBM MQ that were correctly parsed and routed to BL will be stored in the MQ-<ClientID>-Incoming_<Timestamp>.out file. The MQ-<ClientID>-Incoming_<Timestamp>.out file will be created in the TransportLayer.MQAdaptor.StorageDirectory directory regardless of the value of the LogIncomingMessages property. Valid values: true | false |
TransportLayer.MQAdaptor.Session.<SessionNumber>.FromClientDropMessageFilter | - | Optional. Defines a regular expression (perl syntax) to drop matched messages coming from MQ. |
TransportLayer.MQAdaptor.Session.<SessionNumber>.SmartXMLProcessorFromClientMessageFilter | - | Optional. Defines a regular expression (perl syntax) to apply SmartXMLProcessor to matched messages coming from MQ. Overrides default filter: case insensitive "<\?xml version.*|<FIXML.*" |
TransportLayer.MQAdaptor.Session.<SessionNumber>.SendForceCommitTimeMs | 0 | This parameter is applicable since FIXEdge C++ 6.16.1 version. Optional. This property determines the maximum duration that can elapse before a transaction is forcibly committed to IBM MQ. It acts as a timer that starts after the start of the transaction. Once this timer exceeds the period defined by SendForceCommitTimeMs, if there are uncommitted messages, a commit operation is triggered. Valid values:
|
TransportLayer.MQAdaptor.Session.<SessionNumber>.ReceiveForceCommitTimeMs | 0 | This parameter is applicable since FIXEdge C++ 6.16.1 version. Optional. This property determines the maximum duration that can elapse before a transaction is forcibly committed from IBM MQ. It acts as a timer that starts after the start of the transaction. Once this timer exceeds the period defined by ReceiveForceCommitTimeMs, if there are uncommitted messages, a commit operation is triggered. Valid values:
|
Configuration Sample
The following configuration describes the basic FIX to MQ conversion message flow.
FIX message from the SENDER session is converted to FIXML and pushed to the middleware via MQ.
All messages from MQ are converted from FIXML to FIX and send to the RECEIVER FIX session.
Configure sessions in FIXEdge.properties
FIXEdge.properties#------------------------------------------------------------ # FIX Layer Section #------------------------------------------------------------ FixLayer.FixEngine.Sessions = SENDER, RECEIVER FixLayer.FixEngine.Session.SENDER.Version = FIX44 FixLayer.FixEngine.Session.SENDER.StorageType = persistentMM FixLayer.FixEngine.Session.SENDER.Role = Acceptor FixLayer.FixEngine.Session.SENDER.SenderCompID = FIXEDGE FixLayer.FixEngine.Session.SENDER.TargetCompID = SENDER FixLayer.FixEngine.Session.SENDER.EncryptMethod = 0 FixLayer.FixEngine.Session.SENDER.InSeqNum = 0 FixLayer.FixEngine.Session.SENDER.OutSeqNum = 0 FixLayer.FixEngine.Session.SENDER.Description = FIX session for test mq sender FixLayer.FixEngine.Session.SENDER.StartTime = 08:00 FixLayer.FixEngine.Session.SENDER.TerminateTime = 23:59 FixLayer.FixEngine.Session.SENDER.RecreateOnLogout = true FixLayer.FixEngine.Session.SENDER.IntradayLogoutTolerance = true FixLayer.FixEngine.Session.SENDER.ForceSeqNumReset = false FixLayer.FixEngine.Session.SENDER.SocketPriority = EVEN FixLayer.FixEngine.Session.SENDER.HandleSeqNumAtLogon = false FixLayer.FixEngine.Session.RECEIVER.Version = FIX44 FixLayer.FixEngine.Session.RECEIVER.StorageType = persistentMM FixLayer.FixEngine.Session.RECEIVER.Role = Acceptor FixLayer.FixEngine.Session.RECEIVER.SenderCompID = FIXEDGE FixLayer.FixEngine.Session.RECEIVER.TargetCompID = RECEIVER FixLayer.FixEngine.Session.RECEIVER.EncryptMethod = 0 FixLayer.FixEngine.Session.RECEIVER.InSeqNum = 0 FixLayer.FixEngine.Session.RECEIVER.OutSeqNum = 0 FixLayer.FixEngine.Session.RECEIVER.Description = FIX session for test mq receiver FixLayer.FixEngine.Session.RECEIVER.StartTime = 08:00 FixLayer.FixEngine.Session.RECEIVER.TerminateTime = 23:59 FixLayer.FixEngine.Session.RECEIVER.RecreateOnLogout = true FixLayer.FixEngine.Session.RECEIVER.IntradayLogoutTolerance = true FixLayer.FixEngine.Session.RECEIVER.ForceSeqNumReset = false FixLayer.FixEngine.Session.RECEIVER.SocketPriority = EVEN FixLayer.FixEngine.Session.RECEIVER.HandleSeqNumAtLogon = false
Configure IBM MQ Adapter
FIXEdge.properties#------------------------------------------------------------ # Transport Layer Section #------------------------------------------------------------ #Comma separated list of identifiers of Transport Adapters should be loaded. TransportLayer.TransportAdapters = TransportLayer.MQAdaptor #------------------------------------------------------------------------- # MQ Adaptor configuration #------------------------------------------------------------------------- TransportLayer.MQAdaptor.Description = MQ Transport Adaptor DLL TransportLayer.MQAdaptor.Type = DLL TransportLayer.MQAdaptor.DllName = bin/MQTAAddin-MD-x64.dll TransportLayer.MQAdaptor.TimeIntervalBeforeReconnect = 1000 TransportLayer.MQAdaptor.BufferSizeForMessage = 65536 TransportLayer.MQAdaptor.MQ_HOSTNAME = ecsa00400539.epam.com TransportLayer.MQAdaptor.MQ_MANAGER_NAME = FIXEdge TransportLayer.MQAdaptor.MQ_PORT = 1414 TransportLayer.MQAdaptor.MQ_CCSID = 437 TransportLayer.MQAdaptor.MQ_CHANNEL_NAME = TestConnection TransportLayer.MQAdaptor.WaitIntervalForGet = 1000 TransportLayer.MQAdaptor.NumAttemptReconnect = 10 TransportLayer.MQAdaptor.StorageDirectory = FIXEdge1/log TransportLayer.MQAdaptor.UseTransactions = true TransportLayer.MQAdaptor.SessionNumber = 2 # MQ QA session TransportLayer.MQAdaptor.SessionNumber = 1 TransportLayer.MQAdaptor.Session.1.ClientID = MQHub1 TransportLayer.MQAdaptor.Session.1.ToClientQueue = MQClient_test TransportLayer.MQAdaptor.Session.1.FromClientQueue = TransportLayer.MQAdaptor.Session.1.ErrorQueue = MQClient_error TransportLayer.MQAdaptor.Session.1.SmartXMLProcessor = true TransportLayer.MQAdaptor.Session.1.SendTransactionSize = 300 TransportLayer.MQAdaptor.Session.1.ReceiveTransactionSize = 1 TransportLayer.MQAdaptor.SessionNumber = 2 TransportLayer.MQAdaptor.Session.2.ClientID = MQHub2 TransportLayer.MQAdaptor.Session.2.ToClientQueue = TransportLayer.MQAdaptor.Session.2.FromClientQueue = MQClient_test TransportLayer.MQAdaptor.Session.2.ErrorQueue = MQClient_error TransportLayer.MQAdaptor.Session.2.SmartXMLProcessor = true TransportLayer.MQAdaptor.Session.2.SendTransactionSize = 300 TransportLayer.MQAdaptor.Session.2.ReceiveTransactionSize = 1
Configure BL
<?xml version="1.0" encoding="UTF-8"?> <FIXEdge> <BusinessLayer> <Rule Description="Convert ExecutionReport from FIX.4.4 to FIXML.4.4"> <Source Name="SENDER" /> <Action> <Convert SourceProtocol="FIX.4.4" TargetProtocol="FIXML.4.4"/> <Send Name="MQHub1"/> </Action> </Rule> <Rule Description="Route MQ messages back to receiver and convert from FIXML.4.4 to FIX.4.4"> <Source Name="MQHub2" /> <Action> <Convert SourceProtocol="FIXML.4.4" TargetProtocol="FIX.4.4"/> <Send Name="RECEIVER"/> </Action> </Rule> <DefaultRule> <Action> <DoNothing/> </Action> </DefaultRule> </BusinessLayer> </FIXEdge>
Please refer to the Convert page for the full list of possible versions to convert.
- Configure Queues Managers on the IBM WebSphere server.
Error queue
An error queue is a local queue to which error messages can be sent by the WebSphere MQ connector when a message operation fails. For target queues, the error queue acts as a backup option for the target queue when messages cannot be written to the target queue
As per the solution diagram above, there are the following reasons for messages sent to the error queue:
- FIXEdge can't send data to MQ (e.g. when there is no connection to the MQ server).
- FIXEdge can't receive data from the MQ server.
- Other various MQ errors (e.g. configuration issues, or message content issues).
FIXEdge can't produce data to MQ
The reason for getting messages to ErrorQueue is diverse, it is necessary to analyze the logs.
To process messages from the queue, it's recommended:
Use different Error Queues per session. It would be easier to troubleshoot issues if the session would have its own error queue.
It can be configured with the FIXEdge property:TransportLayer.MQAdaptor.Session.1.ErrorQueue = MQClient_error_1 TransportLayer.MQAdaptor.Session.2.ErrorQueue = MQClient_error_2
Use the dmpmqmsg tool for moving messages from the error queue to the producer. All messages from the error queue should be replayed.
dmpmqmsg -m MQClient -I MQClient_error -o MQClient_error_2 -s FIXEdge
-m QueueManagerName.
-I Input queue name.
-o Output queue name.
-s Message content, as all messages have senderCompId and TargetCompId can be used for filtering.
FIXEdge can't consume data from the MQ server and other various MQ errors
These cases are rare ones if there were no configuration changes.
Issues not causing messages in the error queue
There are cases that do not lead to the messages in the Error queue.
E.g.: FIXEdge can't push the data to the FIX connection because the destination is down.
For troubleshooting, in this case, it's recommended to upgrade FIXEdge to version 6.9.0 with a Managed Queue feature.
It was designed for cases when messages should be pushed to the counterparty when they come back online and do not support recovery with resend requests.
Transactions in MQ
FIXEdge MQ Transport Adapter has a transaction mechanism. Before sending it to MQ the messages are persisted in the files.
A path to the files can be configured with a property:
TransportLayer.MQAdaptor.StorageDirectory = FIXEdge1/log
The file storages are needed for keeping undelivered messages that require resending after FIXEdge restart (e.g. in case of failure).
Each file represents an MQ Session and the name of the file contains the ID of the session, e.g.:
MQ Adapter property | Storage file |
---|---|
TransportLayer.MQAdaptor1.Session.1.ClientID = MQHub1 TransportLayer.MQAdaptor1.Session.1.ErrorQueue = errQ | MQ-MQHub1_2005211210150121.out MQ-MQHub1-Errors_2005211210150131.out |
TransportLayer.MQAdaptor1.Session.2.ClientID = MQHub2 TransportLayer.MQAdaptor1.Session.2.ErrorQueue = errQ | MQ-MQHub2_2005211210150801.out MQ-MQHub2-Errors_2005211210150801.out |
MQ Session logs have data and corresponding transaction flag:
- Status of the undelivered transaction: Record valid='Y'.
- Status of the finished transaction. The message has been sent to MQ: Record valid='N'.
Examples of the log from MQ_<ClientID>_xxx.out:
<Record valid='Y' size='Ó ' data='8=FIX.4.4 9=3259 35=AE 49=FIXClient 56=FIXEdgeMQHub1 115=STRING 128=STRING 90=1 91=D 34=169 50=STRING 142=STRING 57=STRING...10=139'/> ... <Record valid='N' size='Ó ' data='8=FIX.4.4 9=3259 35=AE 49=FIXClient 56=FIXEdgeMQHub1 115=STRING 128=STRING 90=1 91=D 34=208 50=STRING 142=STRING 57=STRING...10=137'/>
Removing the storage files causes undelivered data will be deleted.