000030658 - AFX test connector settings button times out and the test connector capabilities work or the test connector capabilities fail but the test connector settings button succeeds in RSA Identity Governance & Lifecycle

Document created by RSA Customer Support Employee on Jul 5, 2016Last modified by RSA Customer Support on Oct 27, 2019
Version 8Show Document
  • View in full screen mode

Article Content

Article Number000030658
Applies ToRSA Product Set:  Identity Governance & Lifecycle 
 RSA Version/Condition: 6.9.x, 7.0.x, 7.1.x
IssueAfter running out of disk space, the RSA Identity Governance & Lifecycle AFX test connector settings button times out and the test connector capabilities work. After resolving the space issue and restarting AFX, the test connector capabilities fail but the test connector settings button succeeds.
 

Additional Symptoms:



  • This is true for all connectors regardless of connector type and target host.
  • A key symptom is the test connector settings button timing out. For example, when targeting an invalid hostname instead of getting the expected connection failure messages, the test times out.
  • There are no useful logs in $AFX_HOME/esb/logs.
  • The $AFX_HOME/activemq/data/activemq.log has messages similar to the following:



2015-06-25 20:41:15,780 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor |
Queue:AFX-SETTINGS-MySQL-Database
java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
    at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
    at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)
    at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
    at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
    at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
    at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand cannot be cast to
org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
    at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
    at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
    at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
    at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
    at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
    at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
    ... 7 more
2015-06-25 20:41:15,781 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue |
Queue:AFX-SETTINGS-MySQL-Database
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException:
org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
    at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
    at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)
    at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
    at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
    at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
    at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.lang.ClassCastException:
org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)

CauseThere are one or more items in the activemq batch that are malformed or corrupt and are preventing new instructions from being batched. Basically the KahaDB file-based persistence database used by ActiveMQ to queue instructions is corrupt.

This issue can occur after an AFX server crashes with insufficient disk space.
 
ResolutionTo resolve this issue, recreate the KahaDB database (queue).

NOTE: Perform the following task when there are no AFX transactions actively being processed as removing the queue will purge any pending items. To ensure there are no active AFX transactions being processed, the first step is to shut down AFX.
  1. Login as the afx user.
  2. Stop AFX.
  3. Navigate to $AFX_HOME/activemq/data.
  4. Remove the existing KahaDB database.
  5. Restart AFX. This will create a new KahaDB database.


$ afx stop
$ cd $AFX_HOME/activemq/data
$ mv kahadb kahadb.old
$ afx start


Please see related RSA Knowledge Base Article 000030081 -- RSA Identity Governance & Lifecycle AFX connectors remain in a Deployed state; Could not locate data file kahadb/db-XXX.log for additional information.

Attachments

    Outcomes