| Article Number | 000030658 |
| Applies To | RSA Product Set: Identity Governance & Lifecycle
RSA Version/Condition: 6.9.x, 7.0.x, 7.1.x |
| Issue | After running out of disk space, the RSA Identity Governance & Lifecycle AFX test connector settings button times out and the test connector capabilities work. After resolving the space issue and restarting AFX, the test connector capabilities fail but the test connector settings button succeeds.
Additional Symptoms:
- This is true for all connectors regardless of connector type and target host.
- A key symptom is the test connector settings button timing out. For example, when targeting an invalid hostname instead of getting the expected connection failure messages, the test times out.
- There are no useful logs in $AFX_HOME/esb/logs.
- The $AFX_HOME/activemq/data/activemq.log has messages similar to the following:
2015-06-25 20:41:15,780 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor |
Queue:AFX-SETTINGS-MySQL-Database
java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand cannot be cast to
org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 7 more
2015-06-25 20:41:15,781 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue |
Queue:AFX-SETTINGS-MySQL-Database
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException:
org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.lang.ClassCastException:
org.apache.activemq.store.kahadb.data.KahaProducerAuditCommand
cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
|
| Cause | There are one or more items in the activemq batch that are malformed or corrupt and are preventing new instructions from being batched. Basically the KahaDB file-based persistence database used by ActiveMQ to queue instructions is corrupt.
This issue can occur after an AFX server crashes with insufficient disk space.
|
| Resolution | To resolve this issue, recreate the KahaDB database (queue).
NOTE: Perform the following task when there are no AFX transactions actively being processed as removing the queue will purge any pending items. To ensure there are no active AFX transactions being processed, the first step is to shut down AFX.
- Login as the afx user.
- Stop AFX.
- Navigate to $AFX_HOME/activemq/data.
- Remove the existing KahaDB database.
- Restart AFX. This will create a new KahaDB database.
$ afx stop
$ cd $AFX_HOME/activemq/data
$ mv kahadb kahadb.old
$ afx start
Please see related RSA Knowledge Base Article 000030081 -- RSA Identity Governance & Lifecycle AFX connectors remain in a Deployed state; Could not locate data file kahadb/db-XXX.log for additional information. |
on Jul 5, 2016