RecoverAtomicAction errors in server.log; Uunable to login after restarting RSA Identity Governance & Lifecycle

[#1]

Article Number

000027952

Applies To

RSA Product Set: Identity Governance & Lifecycle 
RSA Version/Condition: 6.x

Issue

Users are unable to login to RSA Identity Governance & Lifecycle after the system was rebooted. 

The symptom for this problem is the inability to log in. There may be a migration required error in the User Interface (UI) and the following error may be seen in the home/oracle/jboss/server/default/log/server.log:
 

WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.recovery.RecoverAtomicAction_4] -
RecoverAtomicAction: transaction not activated, unable to replay phase 2 commit

 

Cause

There were numerous files in /home/oracle/jboss/server/default/data/tx-object-store/HashedActionStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction that were 0 bytes in size amd this was preventing the startup from being successful.

The WARNING message from the log file appears when the writes of a transactional object state to the persistent store have failed. This will create corrupt records in the persistent store, probably zero length files. The recovery process tries to retrieve the transactional object state so that it can recreate the relevant object in memory. This transactional object may be a transaction, a participant record or some other type of object. Since the records are corrupt this is going to keep failing and the recovery manager will keep retrying and failing.

The most common cause for this problem is the disk is full. 


 

Resolution

There are two parts to this solution:

1. Free up disk space.

Because the most common cause for this problem is that the disk is full, first verify and free up disk space before proceeding with clearing the transaction store. An article that can help with this effort is article 000027944 - How to check RSA Identity Governance & Lifecycle local file system disk space usage.

2. Clear the transaction store.

1. Stop the application server:

$> sudo service aveksa_server stop

2. Execute the following commands in the  /home/oracle/jboss/server/default/data DIRECTORY to list the zero-length files:

$> cd /home/oracle/jboss/server/default/data
$> find . -size 0 -type f -ls

3. Execute the following to delete the zero-length files (Note:  This only deletes files in this directory that have a zero-length (0 bytes) file size):

find . -size 0 -type f -delete

After these commands are run, the errors in the server log will stop once the application server is re-started.

4. Restart the application server

sudo service aveksa_server start

Notes