Persistence layer issue in RSA Identity Governance & Lifecycle degrades performance and potentially leads to system crash
2 years ago
Originally Published: 2019-10-03
Article Number
000041636
Applies To
RSA Product Set: Identity Governance & Lifecycle
RSA Version/Condition: 7.1.0
 
Issue
RSA Identity Governance & Lifecycle performance degrades to the point where a system crash may occur unless the RSA Identity Governance & Lifecycle application is restarted. Symptoms include:
  • Illegal TXN State during transaction rollback.
  • Blocking sessions on the database caused by Idle sessions. 
  • Row lock contention.
  • AFX Server stops running
  • Performance degradation
  • Possible system crash

Below are examples of symptoms that indicate you are having this issue:
  • Illegal TXN State in the user interface (Admin > System > Server Nodes)
Error - Illegal TXN State: Attempt to start new transaction during rollback.
 
User-added image
  • An increasing number of blocking sessions are seen in Oracle Enterprise Manager (OEM):
User-added image
 
  • Illegal TXN State errors and PersistenceServiceProvider references are seen in the aveksaServer.log:
 
07/03/2018 12:01:30.698 ERROR (default task-26) [com.aveksa.gui.core.filters.LoginFilter] java.lang.IllegalStateException: have sessions (SessionImpl(PersistenceContext
...
at com.aveksa.server.db.persistence.PersistenceServiceProvider.checkAllClosed(PersistenceServiceProvider.java:3925) at com.aveksa.gui.core.MainManager.doGet(MainManager.java:135)

and 
 
07/03/2018 12:04:29.857 ERROR (default task-8) [com.aveksa.server.core.AdminServiceProvider] Error cleaning up database context java.lang.RuntimeException: Illegal TXN State: Attempt to start new transaction during rollback. Txn count=3 at com.aveksa.server.db.persistence.PersistenceServiceProvider.beginTransaction(PersistenceServiceProvider.java:2633)

 
Please refer to RSA Knowledge Base Article 000030327 -- Artifacts to gather in RSA Identity Governance & Lifecycle to find the location of the log files for your specific deployment.
 
  • The AFX Server shuts down 
Please refer to RSA Knowledge Base Article  000038027 -- AFX Server intermittently shuts down in RSA Identity Governance & Lifecycle for additional information.
Cause
When something failed or was interrupted during database interaction (for example a malformed, failed or killed database statement), the code sometimes did not cleanup resources and revert transactions properly. This would lead to memory leaks which would waste memory and keep database locks until the application server was restarted. 

 
Resolution
This issue is resolved in the following RSA Identity Governance & Lifecycle patches:
  • RSA Identity Governance & Lifecycle 7.1.0 P03
  • RSA Identity Governance & Lifecycle 7.1.1.
Workaround
Restart RSA Identity Governance & Lifecycle using the method for your specific platform deployment.