000039095 - Change Requests stuck in the AFX Fulfillment Handler Workflow Node and Workflows Stalled in RSA Identity Governance & Lifecycle

Document created by RSA Customer Support Employee on Jul 6, 2020•Last modified by RSA Customer Support Employee on Aug 25, 2020
Version 26Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000039095
Applies ToRSA Product Set: RSA Identity Governance & Lifecycle
RSA Version/Condition: 7.0.x, 7.1.x, 7.2.0
 
IssueChange Request workflows intermittently get stuck in the AFX fulfillment handler node and do not progress.

SYMPTOMS:
  • Change requests that provision using AFX are stuck in the fulfillment phase
  • The change request workflows are stuck in the AFX fulfillment handler node
  • Workflows are flagged as stalled
  • AFX requests are stuck in the V state
For more information on the AFX request V state, see RSA Knowledge Base Article 000038781 -- What is 'AFX fulfillment state V' that is sometimes seen in the AFX log in RSA Identity Governance & Lifecycle?
 
CauseThis is a known issue reported in engineering ticket ACM-101553.

Not enough memory is available to the ActiveMQ queue during heavy processing of bulk Change Request items which causes messages to get stuck in the ActiveMQ queue intermittently impacting AFX processing and Change Request completion.
 
ResolutionThis issue is resolved in the following RSA Identity Governance & Lifecycle patches:
  • RSA Identity Governance & Lifecycle 7.1.1 P07
  • RSA Identity Governance & Lifecycle 7.2.0 P02
WorkaroundThere are two workarounds needed:
  1. Restart the stuck AFX requests, and
  2. Manually modify the ActiveMQ memory configuration to prevent future occurrences of this issue.
 

Restart stuck AFX requests:


Open a support ticket with RSA Identity Governance & Lifecycle Support for a workaround and provide the following information in the case:

  1. Login to a SQL tool such as SQL*Plus or SQL Developer as avuser, and
  2. Execute the following queries:

These queries will determine the AFX request state of the specific stuck change request:



select * from T_AV_CHANGE_REQUESTS CR where NAME like '%stuck-change-request-name%';


Note the values of CR.ID and CR.CURRENT_STATE returned in the query results and substitute them into the next two queries:



select * from T_AV_CHANGE_REQUEST_DETAILS CRD where CHANGE_REQUESTS_ID=<CR.ID>
select * from  T_AV_AFX_REQUEST where change_item_id=<CRD.ID>


This query will identify any AFX requests in the V state:




select * from t_av_afx_request where state ='V'



Manually modify the ActiveMQ memory configuration:



  1. Follow the instructions in the attached ActiveMQ_Memory_Changes.docx document.
  2. Restart AFX as the afx user:


afx restart


 

Attachments

Outcomes