Deployment notes and troubleshooting

This topic contains additional notes for the deployment and diagnosis of applications using the IBM® Reliable Transport Extension for WebSphere® sMash.

Broker naming and preferences

When a messaging broker is created, it stores information regarding its configuration using the Java™ platform preferences mechanism. These are stored using a platform specific location outside the scope of the IBM® WebSphere® sMash application.

Once a messaging broker has been created, it cannot be moved to another machine.

The broker preferences are stored using the value of /config/messaging/broker/name as a key. Broker names should be unique on the local machine to ensure each maintains a separate identity.

Managing the message store

When it is first started, a messaging broker creates a message store file in which messages are kept until they are delivered. It is important to ensure that the message store is located on a suitable filing system to preserve the integrity of the information it contains. For example:

The filing system must not use write caching, encryption or compression.
The filing system should use redundancy or mirroring to protect data in the event of a hardware failure.
The filing system should be large enough to contain the message store and transaction log.

The initial size of the message store file will be approximately the same as the associated transaction log but the message store will grow as necessary to store the messages that are yet to be delivered. A broker which continues to receive messages that are not being consumed can conceivably run out of disk space and will then be unable to provide a messaging service to client applications. A messaging deployment should be designed to discourage large message buildups, and report them promptly when they occur. Other precautionary measures that could be taken include deploying message stores onto a separate partition of the filing system from the main operating system or to a filing system to which additional storage can be added.

See Configuring a messaging broker for more information about configuring the message store location.

Messaging trace and first failure data capture location

If a problem is detected, the messaging broker places any first failure data capture (FFDC) and trace files in a diagnostics directory inside the message store directory. The location of the message store directory can be specified using /config/messaging/broker/messageStore but defaults to messageStore inside the application root directory.

For example, the diagnostics for broker1 might be found in messageStore\broker1\diagnostics.

When reporting a problem involving the messaging broker, the contents of the diagnostics directory should be collected, in addition to the contents of the application logs directory.

Client IDs and connection pooling

The javax.jms.Connection class exposes a method setClientID(String clientID). When creating Connections, the IBM Reliable Transport Extension automatically generates unique client IDs comprising three numbers separated by hyphens. The first of these numbers is a timestamp and thus indicative of the Connection's age. There are three different types of Connection recorded in the log files.

Those generated by the Connection API's messaging protocol are assigned no prefix.
Those generated by the external client are prefixed 'E'.
Those generated by a bridge between broker instances are prefixed 'B'.

Broker logs will be found to contain messages of the form,

    INFO [ FMBT1756 testBroker1 Client '1211276930793-33-434379236' - Connected. 
        Listener: MqttTlsListener9091:/127.0.0.1:9091-/127.0.0.1:2854, client address: /127.0.0.1:2854 ]
2008-05-20 10:49:25 zero.messaging.broker.control.ZeroLogDestination::logMessage Thread-11
    INFO [ FMBT1756 testBroker1 Client 'E1211277011629-1-18916478' - Connected. 
        Listener: MqttTlsListener9091:/127.0.0.1:9091-/127.0.0.1:2868, client address: /127.0.0.1:2868 ]
2008-05-20 11:21:01 zero.messaging.broker.control.ZeroLogDestination::logMessage Thread-11

External client application code can change a Connection's clientID once by calling Connection.setClientID(String clientID). Further calls to this method will result in an Exception of the form,

javax.jms.IllegalStateException: The client ID for this connection is read-only.

Both the external client and messaging protocol pool Connections by default. This must be borne in mind when attempting to correlate connectionIDs with message producers.

General messaging and broker-related problems

This section describes some of the problems that can result from incorrectly configuring a broker. The key settings are laid out in the Advanced configuration section of the "Configuring a message broker" topic. It's important to ensure that each of the maxMessageSize, maxQueueDepth and transactionLogSize parameters are set correctly for your environment. A good rule of thumb is that transactionLogSize should be between five and ten times maxMessageSize. NegativeAckExceptions are seen at the client in many error cases. Their reason codes are significant. The most common reason codes seen at the client are 4, message too large, 5, a timeout, and 400, queue full. When using the external client, the primary JMSException will be thrown directly to the application. As noted in the messaging protocol reference, the top-level JMSException must in that case be obtained via the wrapped ConnectionException's getCause() method. In both cases, further information can be obtained by calling getLinkedException() on the top-level JMSException.

Exceeding the maximum message size

NegativeAckExceptions with reason code 4 will be generated in the client when attempting to send a message larger than the broker's configured maxMessageSize.

javax.jms.JMSException: An exception thrown while sending the message by MqttProducer for clientId 
        E1211212817546-1-18916478 connected to MicroBroker tls://localhost:9091. 
    at com.ibm.msg.client.mqtt.internal.ProviderMessageProducerMqttImpl.send(...
    at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(...
    ...
    
// With LinkedException
com.ibm.mqttclient.v4.base.MqttNegativeAckException:  (Reason code=4)
    ....

Timeouts

If you are seeing reason code 5 timeouts, consider increasing the value of deliveryTimeout as described in the broker connection configuration section. Timeouts can occur under situations of heavy load, or when sending very large messages.

Exceeding the maximum queue depth

Two different error messages may seen if maxQueueDepth is exceeded. When sending messages under non-transacted Sessions, a client exceeding maxQueueDepth will receive am exception of the form,

javax.jms.JMSException: An exception thrown while sending the message by MqttProducer for clientId 
        E1211209604787-1-18916478 connected to MicroBroker tls://localhost:9091. 
    at com.ibm.msg.client.mqtt.internal.ProviderMessageProducerMqttImpl.send(...
    at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(...
	...
	
// with LinkedException
com.ibm.mqttclient.v4.base.MqttNegativeAckException:  (Reason code=400)
    ...

When producing messages under a transacted Session, the exception received will be slightly different:

javax.jms.JMSException: Commit failed on session.
    at com.ibm.msg.client.mqtt.internal.ProviderSessionMqttImpl.commit(...
    at com.ibm.msg.client.jms.internal.JmsSessionImpl.commitTransaction(...
    ...
	
// Again with LinkedException	
com.ibm.mqttclient.v4.base.MqttNegativeAckException:  (Reason code=400)
    ....

Exceeding the transaction log size

If a broker receives a message which is larger than transactionLogSize, it will output a SEVERE error on standard error, referencing a FFDC file in messageStore/<brokerName>/diagnostics/ which will report an error along the lines of,

CWSOM1027E: Log file was too full to accommodate 2002728(long) bytes, 
    checkedBytesInLogBuffer=190(long) bytes, flushSetReserved=8205(long) bytes, 
    flushSetAllocated=447245(long) bytes.
com.ibm.ws.objectManager.LogFileFullException: CWSOM1027E: Log file was too full 
    to accommodate 2002728(long) bytes, checkedBytesInLogBuffer=190(long) bytes, 
    flushSetReserved=8205(long) bytes, flushSetAllocated=447245(long) bytes.

FFDCs of this nature may well cause the broker to shutdown and restart; if this happens then the client process will experience a timeout. This fact can be gleaned from the linked exception in the client, e.g.,

com.ibm.mqttclient.v4.base.MqttNegativeAckException:  (Reason code=5)

Problems with the `messaging` protocol

A common problem seen with the messaging protocol is to call Connection.doPOST(String queueName) for a value of queueName that is not defined in the /config/connection/destinations section of the zero.config file. Doing this will yield an error message of the form,

	SEVERE [ CWPZC1124E: Unable to identify protocol for request. Request target: queue1 ]
zero.core.connection.ConnectionException: Unable to identify protocol for request. Request target:queue1
	at zero.core.connection.engine.TransportFactory.getTransport(...

Problems with the external client

When establishing an external client connection to a secured broker instance, a common mistake is to forget to call WrappedQueueConnectionFactory.setAlgorithm(String algorithm) first. Correct algorithm strings include "SunX509" for Sun JREs (this is the default value) and "IbmX509" for IBM JREs. Failure to set the algorithm correctly will result in an Exception of the form,

javax.jms.JMSException: Failed to create a connection for client E1211279066755-1-905065970 
        to MicroBroker tls://localhost:9091.
    at com.ibm.msg.client.mqtt.internal.ProviderConnectionMqttImpl.connectToBroker(...
    at com.ibm.msg.client.mqtt.internal.ProviderConnectionMqttImpl.start(...

// With LinkedException
com.ibm.mqttclient.v4.base.MqttNotConnectedException: FMBD3016 JMSTT Module TLSInitiator: 
        SSL-Initialisation failed. Configured Algorithm is not known.
    at com.ibm.mqttclient.v4.internal.base.MqttBaseClientImpl.createConnection(...
    ...