Change Requests stuck in the AFX Fulfillment Handler Workflow Node and Workflows Stalled in RSA Identity Governance & Lifecycle

[#1]

Article Number

000039095

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle
RSA Version/Condition: 7.0.x, 7.1.x, 7.2.0
 

Issue

Change Request workflows intermittently get stuck in the AFX fulfillment handler node and do not progress.

SYMPTOMS:

• Change requests that provision using AFX are stuck in the fulfillment phase.
• The change request workflows are stuck in the AFX fulfillment handler node.
• Workflows are flagged as stalled.
• AFX requests are stuck in the V state.

For more information about the AFX request V state, see RSA Knowledge Base Article 000038781 -- What is 'AFX fulfillment state V' that is sometimes seen in the AFX log in RSA Identity Governance & Lifecycle?
 

Cause

This is a known issue reported in engineering ticket ACM-101553.

Not enough memory is available to the ActiveMQ queue during heavy processing of bulk Change Request items which causes messages to get stuck in the ActiveMQ queue intermittently impacting AFX processing and Change Request completion.
 

Resolution

This issue is resolved in the following RSA Identity Governance & Lifecycle patches:

• RSA Identity Governance & Lifecycle 7.1.1 P07
• RSA Identity Governance & Lifecycle 7.2.0 P02
• RSA Identity Governance & Lifecycle 7.2.1
• RSA Identity Governance & Lifecycle 7.5.0

Workaround

There are two workarounds needed:

1. Restart the stuck AFX requests, and
2. Manually modify the ActiveMQ memory configuration to prevent future occurrences of this issue.

 

Restart stuck AFX requests:

Open a support ticket with RSA Identity Governance & Lifecycle Support for a workaround and provide the following information in the case:

1. Login to a SQL tool such as SQL*Plus or SQL Developer as avuser, and
2. Execute the following queries:

These queries will determine the AFX request state of the specific stuck change request:

select * from T_AV_CHANGE_REQUESTS CR where NAME like '%stuck-change-request-name%';

Note the values of CR.ID and CR.CURRENT_STATE returned in the query results and substitute them into the next two queries:

select * from T_AV_CHANGE_REQUEST_DETAILS CRD where CHANGE_REQUESTS_ID=<CR.ID>
select * from  T_AV_AFX_REQUEST where change_item_id=<CRD.ID>

This query will identify any AFX requests in the V state:

select * from t_av_afx_request where state ='V'

Manually modify the ActiveMQ memory configuration:

1. Follow the instructions in the attached ActiveMQ_Memory_Changes.docx document.
2. Restart AFX as the afx user:

afx restart