RSA Governance & Lifecycle 8.0.0
High CPU utilization spikes are being observed on the application server. After gathering a thread dump and multiple top outputs, the results point to a workflow issue.
The output in the top command used to collect the CPU usage information shows an output similar to the below where %CPU is showing as 94.1 for multiple threads with name "Worker_actionq#":
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20954 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:30.57 Worker_actionq# 20956 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:07.16 Worker_actionq# 20958 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:27.04 Worker_actionq# 20959 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:24.21 Worker_actionq# 20960 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:05.98 Worker_actionq# 20961 oracle 20 0 23.3g 20.2g 37784 R 94.1 64.7 90:24.80 Worker_actionq# 20955 oracle 20 0 23.3g 20.2g 37784 R 88.2 64.7 89:29.21 Worker_actionq#
The output in the thread dumps generated to analyze the execution stack traces show an output similar to the mentioned below showing "StatementEngineConditionWizard" belonging to "workpoint":
"Worker_actionq#Normal#jdbc/avdb_2" #342 daemon prio=5 os_prio=0 tid=0x00005558d39ae800 nid=0x51d9 runnable [0x00007f49e522a000] java.lang.Thread.State: RUNNABLE at com.workpoint.server.script.StatementEngineConditionWizard.execute(Unknown Source) at com.workpoint.server.script.ScriptEngine.A(Unknown Source) at com.workpoint.server.script.ScriptEngine.executeCondition(Unknown Source) at com.workpoint.services.impl.ScriptExecAsyncServiceImpl.executeCondition(Unknown Source) at com.workpoint.client.Script.executeCondition(Unknown Source) at com.aveksa.server.workflow.scripts.action.ConditionAction.evaluateWorkflowVariables(ConditionAction.java:1164) at com.aveksa.server.workflow.scripts.action.ConditionAction.evaluateChangeRequestConditions(ConditionAction.java:167) at com.aveksa.server.workflow.scripts.action.ConditionAction.evaluate(ConditionAction.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.workpoint.server.script.StatementEngineJava.execute(Unknown Source) at com.workpoint.server.script.ScriptEngine.A(Unknown Source) at com.workpoint.server.script.ScriptEngine.execute(Unknown Source) at com.workpoint.server.script.ScriptEngine.execute(Unknown Source) at com.workpoint.server.job.JobNode.changeState(Unknown Source) at com.workpoint.server.job.JobNode.isComplete(Unknown Source) at com.workpoint.server.job.Job.changeWorkItemState(Unknown Source) at com.workpoint.server.job.Job.changeWorkItemState(Unknown Source) at com.workpoint.services.impl.WorkItemUpdateServiceImpl.changeState(Unknown Source) - locked <0x0000000313819830> (a java.lang.Object) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at com.sun.proxy.$Proxy89.changeState(Unknown Source) at com.workpoint.client.WorkItemEntry.A(Unknown Source) at com.workpoint.client.WorkItemEntry.openAndComplete(Unknown Source) at com.aveksa.server.workflow.scripts.action.WorkflowActions.completeWorkItem(WorkflowActions.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.workpoint.server.script.StatementEngineJava.execute(Unknown Source) at com.workpoint.server.script.ScriptEngine.A(Unknown Source) at com.workpoint.server.script.ScriptEngine.execute(Unknown Source) at com.workpoint.server.monitor.ActionMonitorHelper.A(Unknown Source) at com.workpoint.server.monitor.ActionMonitorHelper.execute(Unknown Source) at com.workpoint.services.impl.ScriptExecAsyncServiceImpl.executeScriptMonitor(Unknown Source) at com.workpoint.client.Monitor.executeScriptMonitor(Unknown Source) at com.workpoint.queue.work.ActionQWorker.A(Unknown Source) at com.workpoint.queue.work.ActionQWorker.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Locked ownable synchronizers: - <0x0000000313819f98> (a java.util.concurrent.ThreadPoolExecutor$Worker)
The 'Manual Activities' Fulfillment Workflow contains a Decision Node with the following configuration:
The above is configured to have two comparison statements, but the Connector is set to 'NONE' for both. This is causing the Decision Node to remain stuck and not able to move forward.
To prevent this issue from happening again, update the Decision Node configuration to define Connectors for the second and subsequent rows. For example, the above shown Decision Node should be updated as follows (where we are using the OR connector for second condition):
If you are currently experiencing high CPU usage due to this issue, the affected Change Request(s) must be cancelled.
To find affected Change Requests:
1- Go to RSA Governance & Lifecycle UI and review each outstanding (in non-terminal state) request.
2- Navigate to the "Workflow tab" for each of those requests, confirm if the workflow has a Decision Node in active state, then cancel the Change Request by going back to the Change Request details page, and cancel the Change Request using 'Cancel Request' button.
3- Restart of the RSA Governance & Lifecycle application server is required to complete the steps.
To gather thread dumps from the RSA Governance & Lifecycle Wildfly server component and top command outputs, please follow below steps:
- Identify the RSA Governance & Lifecycle (G&L) Wildfly server component process id (first numeric value after the process owner name). To do this, run the below command as the Linux user owning the G&L application process (oracle user):
ps -ef |grep java|grep "Server:" - To generate top command output, run the following as the Linux user owning the G&L application process (oracle user), replace the string
${PID}with the G&L server component PID obtained from the above command:top -p ${PID} -H -n1 -b > /tmp/top_output1 - To generate a thread dump output, run the following as the Linux user owning the G&L application process (oracle user), replace the string
${PID}with the G&L server component PID obtained from the command in step 1:jstack -l ${PID} >/tmp/thread_dump_output
