New feature for RSA Identity Governance & Lifecycle 7.1: Workflow System Status

[#1]

Article Number

000036324

Applies To

RSA Product Set: Identity Governance & Lifecycle
RSA Version/Condition: 7.1.0 and higher
 

Issue

Introduction and References

A new feature has been implemented in RSA Identity Governance & Lifecycle 7.1 to aid in identifying problems in workflow that might impact the system.

There are several sites that will introduce you to that new feature:

• The RSA Link blog entitled New Feature:  Workflow System Status.
• Search for Workflow System Status in the online help for RSA Identity Governance & Lifecycle 7.1.
• Watch a video on RSA Link about WorkFlow Priority Queues.

This article will not repeat any information contained in the above referenced material, we therefore recommend that you consume the information provided in above links.

Problem Observation

After an upgrade to RSA Identity Governance & Lifecycle 7.1, a customer might receive an enormous amount of admin emails with the following text:

A new admin error has been generated.
The error requires the attention of AveksaAdmin.
Admin error Details:
Type: SystemStatusEvent
Description: Queue Depth
Date Generated: 2/28/18 1:47 PM
Priority: High
View Error
You are receiving this email because an admin error that requires your attention has been created.
Error:
OK: Monitor[alertq#Normal] has taken 2724 ms to process an item.

Notifications like this will also show up in the /home/oracle/wildflystandalone/log/aveksaServer.log (this is a different example):
 

02/25/2018 13:36:44.555 WARN (Worker_eventq#Event Queue#WPDS_23) [com.aveksa.server.workflow.statistics.QueueDepthProcessor] WARNING: Monitor[actionq#Normal] has taken 7339 ms to process an item. 
please check logs for details.
<?xml version="1.0" encoding="UTF-8"?>

<monitor_statistics version="1.0" type="Java">

<event_date>2018-02-25T13:36:42</event_date>
<event_uuid>ea8e2351-4597-43aa-93e8-c3225d01535c</event_uuid>
<queue_monitor_statistics>
<queue_monitor>
<dsn>WPDS</dsn>
<resc>-1</resc>
<monitor_type>actionq</monitor_type>
<name>actionq#Normal</name>
<active>true</active>
<time_initiated>1519587404516</time_initiated>
<latest_service_date_time>1519587317022</latest_service_date_time> GMT: Sunday, February 25, 2018 7:35:17.022 PM
<longest_queued_time>7339</longest_queued_time>
<queue_worker_statistics>
<name>actionq#Normal#1180</name>
<longest_service_time_detail>ScriptID=3:WPDS, ProcessRef='WF_RR_2', JobID=25689:WPDS, JobRowVersion=19, ProcessNodeUUID=1bc0ac70-25a0-4e19-90ad-3b136d9e79b9, NodeName='Form Approvals', 
JobNodeID=167616:WPDS, WorkItem=103652:WPDS:1, Script='Node - Available Asynchronous'. Duration=1683 milliseconds.
</longest_service_time_detail>
<time_started>1519585601626</time_started> GMT: Sunday, February 25, 2018 7:06:41.626 PM
<current_task_time>1519587404516</current_task_time> GMT: Sunday, February 25, 2018 7:36:44.516 PM
<num_active_workers>1</num_active_workers>
<longest_service_millis>1664</longest_service_millis>
<longest_service_date_time>1519587317022</longest_service_date_time>
<total>2000</total>
<num_items_serviced>5</num_items_serviced>
</queue_worker_statistics>
</queue_monitor>
</queue_monitor_statistics>
</monitor_statistics>

Task

Adjust the WARNING threshold level.

Resolution

These messages are genreated because there is always data in the queues, and due to the upgrade, the queue times are higher than expected.

If these messages persists, they may be due to possible system load and/or performance issues, and may require customer specific values.  Very few customers run into these messages, and an old or undersized system is often the cause.

To remedy the issue, and appropriately lower the amount of emails and messages in the aveksaServer.log, look at their high numbers and find a comfortable number and change the thresholds.

Thresholds in 7.1 are set (see online doc link above) to

Warning - 1000 milliseconds

For example, if your warnings show a value such as the example below, you may increase the WARNING value to 7500:

02/25/2018 13:36:44.555 WARN (Worker_eventq#Event Queue#WPDS_23) [com.aveksa.server.workflow.statistics.QueueDepthProcessor] 
WARNING: Monitor[actionq#Normal] has taken 7339 ms to process an item. please check logs for details.

To change the WARNING level threshold

1. Go to Admin > System 
2. Click on the Settings tab.
3. Click Edit.
4. At the bottom of the page (Custom), add the following parameter (from the online doc link) and set it to an appropriate threshold:

WF_QUEUE_DEPTH_DURATION_WARNING
 

As mentioned above, if your warnings usually stop below 7500, you can use 7500 as new threshold.

 

These threshold settings depend on each customer environment and are expected to be adjusted if needed.