|Applies To||RSA Product Set: RSA Identity Governance & Lifecycle|
RSA Version/Condition: 7.1.0, 7.1.1
|Issue||Change requests get randomly and unexpectedly cancelled in RSA Identity Governance & Lifecycle and generate up to thousands of unrelated activities and/or continue to provision even though they have been cancelled.|
Additional symptoms include:
Errors potentially seen in the log files include:
This issue can cause a single change request to be associated with a huge number (hundreds of thousands or millions) of unrelated change request items all with workflow tasks that are not relevant. These workflow tasks typically are not actionable, but they are on the Workpoint task list and are processed by Workpoint. This may result in errors similar to but not identical to the following in the WorkPoint.log:
|Cause||This is a known issue reported in engineering tickets ACM-97445, ACM-101694 and ACM-101765. |
This issue occurs when a select query used to expand (create) Workpoint items for change request details by Change Request ID faults at the database abstraction layer. We fail to trap the exception leaving the query to match all change request details for other change requests. This causes Workpoint to instantiate a very large number (millions) of irrelevant jobs related to the change request but related to other change request details.
|Resolution||This issue is resolved in the following RSA Identity Governance & Lifecycle versions and patch levels which introduces additional error trapping around the database connection layer. |
Customers who have encountered this issue should be strongly encouraged to patch to a version where this is resolved. Even if a particular event self resolves the severity of future events cannot be predicted.
NOTE: Until you are able to upgrade, you can use the tools provided in RSA Knowledge Base Article 000038546 -- Queries to detect workflows and change requests that could overload the Workpoint Server in RSA Identity Governance & Lifecycle to monitor the system for change requests or workflows that may potentially cause this issue.
|Workaround||This is not a workround. These are instructions for resolving the situation once it has occurred. Please patch asap.|
There are two parts to this resolution:
As avuser run the following SQL detection scripts:
The output gives you a list of change requests with corrupted information from unrelated change requests that are currently being processed.
NOTE: This same issue can occur with an aberrant workflow that is not associated with a change request such as custom tasks or rules. In this case the cr_id will be null.
The output gives you a list of cancelled CRs that are still provisioning:
If you have output from either script, you have this issue.
|Notes||See related RSA Knowledge Base Article 000038545 -- How to detect and rectify a workflow stalled and/or Workpoint server down situation in RSA Identity Governance & Lifecycle for additional information.|