AFX Connectors remain in a Deployed state and 'Detected missing/corrupt journal files' error in RSA Identity Governance & Lifecycle

[#1]

Article Number

000038512

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle
RSA Version/Condition: 7.0.x, 7.1.x, 7.2.x
 

Issue

The AFX Server is in a Running state but all previously working RSA Identity Governance & Lifecycle AFX Connectors are stuck in a Deployed state and do not progress to a Running state. For example,

Image description

A restart of AFX, RSA Identity Governance & Lifecycle, and even a system reboot does not fix the problem.

The $AFX_HOME/activemq/data/activemq.log has an error similar to this:
 

2020-01-09 09:58:05,051 | ERROR | Failed to start ActiveMQ JMS Message Broker. 
Reason: java.io.IOException: Detected missing/corrupt journal files. 40 messages affected. 
| org.apache.activemq.broker.BrokerService | main
java.io.IOException: Detected missing/corrupt journal files. 40 messages affected.
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:626)
at org.apache.activemq.store.kahadb.MessageDatabase$6.execute(MessageDatabase.java:460)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:458)
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:315)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:357)
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:222)
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:180)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:186)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:485)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

 

Cause

KahaDB is a file-based persistence database used by ActiveMQ and is corrupt.

We have seen the KahaDB database become corrupted on a system that ran out of disk space while AFX was actively being used.
 

Resolution

To resolve this issue recreate the persistence database and restart AFX:

1. Recreate the persistence database as described in RSA Knowledge Base Article 000038511 -- How to create a new ActiveMQ KahaDB for use with AFX in RSA Identity Governance & Lifecycle.
2. If recreating the persistence database does not resolve your issue, the next step would be to download and install the AFX Server Archive. For instructions on how to do this, please see RSA Knowledge Base Article 000037993 - How to download and install the AFX Server Archive in RSA Identity Governance & Lifecycle.
3. If downloading and installing the AFX Server Archive does not resolve your issue, please see RSA Knowledge Base Articles 000038029 -- Troubleshooting AFX Connector issues in RSA Identity Governance & Lifecycle and 000038030 -- Troubleshooting AFX Server issues in RSA Identity Governance & Lifecycle for additional articles on resolving AFX issues.

If none of the above resolves your issue, please contact RSA Identity Governance & Lifecycle Support for further assistance.