000031343 - RSA Identity Governance & Lifecycle XARecoveryModule generates an XAException when attempting to roll back changes in failed workflows

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support on Dec 16, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000031343
Applies ToRSA Product Set: Identity Management & Governance, RSA Identity Governance & Lifecycle
RSA Product/Service Type: Enterprise Software
RSA Version/Condition: 6.8.1.,6.9, 6.9.1, 7.0.0, 7.0.1, 7.0.2
Platform: WebSphere, WebLogic, Wildfly
IssueWebSphere generates a large number of errors associated with the XA transaction rollback tables in the WebSphere system.out log.

[9/2/15 14:31:28:262 GMT] 00000073 XARecoveryDat A   WTRN0146I: Obtained 0 xid(s) from xa recover on XAResource: WPDS of which 0 will be processed by this server
[9/2/15 14:31:28:263 GMT] 00000073 XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: WPDS
[9/2/15 14:31:28:370 GMT] 00000073 WSRdbXaResour E   DSRA0304E:  XAException occurred. XAException contents and details are: The cause is               : null.
[9/2/15 14:31:28:371 GMT] 00000073 WSRdbXaResour E   DSRA0302E:  XAException occurred.  Error code is: XAER_RMERR (-3).  Exception is: <null>
[9/2/15 14:31:28:371 GMT] 00000073 XARminst      E   WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@dfcc68de. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException
                at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)
                at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1116)
                at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141)
                at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:716)
                at com.ibm.tx.jta.impl.PartnerLogTable.recover(PartnerLogTable.java:431)
                at com.ibm.tx.jta.impl.RecoveryManager.resync(RecoveryManager.java:1534)
                at com.ibm.tx.jta.impl.RecoveryManager.performResync(RecoveryManager.java:2267)
                at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:117)
                at com.ibm.tx.jta.impl.RecoveryManager.run(RecoveryManager.java:2220)
                at java.lang.Thread.run(Thread.java:777)

Wildfly generates a large number of errors associated with the XA transaction rollback tables in the server.log located in /home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log (6.9.1)  or /home/oracle/wildfly-8.2.0.Final/standalone/log (7.0.0 and later).

2017-04-05 02:35:24,266 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.recover(XAManagedConnection.java:362)
at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.recover(XAResourceWrapperImpl.java:185)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:548) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:187) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]

In addition systems where the transaction rollback is not configured correctly the system may experience slow performance  and high CPU loads when a large number of failed transactions are pending such as may occur if there is a workflow that is failing due to a fault.  This can lead to other workflows that are configured correctly to fail. 
CauseThis issue may occur on if the RSA Identity Governance and Lifecycle user AVUSER is not configured with permission to access the Oracle XA transaction rollback tables.  This can also cause the transaction tables to fill up.
XA transaction database grants are already present if you are using a hardware appliance with local database, but these grants need to be manually added for WebSphere and WebLogic servers or Wildfly appliances with external databases.  The 6.8.1 and 6.9.1 installation guides for WebSphere and WebLogic do not include this information.   This is corrected in the RSA Via L&G 7.0 installation guide.   
Resolution1.  For Hardware and Software appliances with a local database the setup will install the correct grants. 
2. For Hardware and Software Appliances based on Wildfly the instructions for applying the correct database grants for a remote database are covered starting on page 28 of the Database Setup and Management Guide for 7.0.1 in the chapter “Configure the User Schema Privilege Grants”.
3.  For older Oracle or WebSphere instances apply the following Oracle grants for AVUSER to the following XA transaction tables:
grant select on pending_trans$ to avuser; 
grant select on dba_2pc_pending to avuser;
grant select on dba_pending_transactions to avuser;
grant execute on dbms_xa to avuser;

On WebSphere the transaction logs can also take up considerable disk space.  If the grants were not applied correctly you should also clean up the old transaction logs.
The transaction logs for WebSphere may be located in different locations depending on the location where WebSphere is installed.  For example:

To remove the transaction logs, do the following:

  1. Stop the server.
  2. Delete the '\transaction' folder and all subdirectories.
  3. Restart the server. 
For WebLogic the transaction logs are in binary format and cannot be deleted. 

See the following IBM article for more information for XA transaction configuration for WebSphere.

See the following Oracle Article for more information for XA transaction configuration for WebLogic. 
The following query confirms the current setting if you prefer to verify if one or more grants are missing. It needs to be executed using SYS user. 

select * from table_privileges 
WHERE grantee = 'AVUSER' 
ORDER BY owner, table_name;


The result bellow is the expected result from the query above.