000035770 - Event Stream Analysis service becomes unresponsive under heavy load  in RSA Security Analytics

Document created by RSA Customer Support Employee on Dec 9, 2017•Last modified by RSA Customer Support Employee on Apr 10, 2019
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000035770
Applies ToRSA Product Set: NetWitness Logs & Network, Security Analytics
RSA Product/Service Type: Event Stream Analysis (ESA), Advanced Threat Detection
RSA Version/Condition: 10.6.x, 11.1.x, 11.2.x
Platform: CentOS
O/S Version: EL6
IssueThe Event Stream Analysis may become unresponsive due to heavy resource usage, and the configuration for the wrapper may need to be adjusted.

The following error is found in the /opt/rsa/esa/wrapper.log:

DEBUG | wrapperp | 2016/10/19 21:02:27 | send a packet PING : ping
ERROR | wrapper | 2016/10/19 21:02:28 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2016/10/19 21:02:28 | JVM did not exit on request, terminated
DEBUG | wrapperp | 2016/10/19 21:02:29 | server listening on port 32001.
DEBUG | wrapper | 2016/10/19 21:02:29 | Waiting 42 seconds before launching another JVM.
DEBUG | wrapper | 2016/10/19 21:02:32 | Signal trapped. Details:
DEBUG | wrapper | 2016/10/19 21:02:32 | signal number=17 (SIGCHLD), source="unknown"
DEBUG | wrapper | 2016/10/19 21:02:32 | Received SIGCHLD, checking JVM process status.
STATUS | wrapper | 2016/10/19 21:02:32 | JVM exited in response to signal SIGKILL (9).
DEBUG | wrapper | 2016/10/19 21:02:32 | JVM process exited with a code of 1, however the wrapper exit code was already 1.
ERROR | wrapper | 2016/10/19 21:02:32 | Unable to start a JVM
STATUS | wrapper | 2016/10/19 21:02:32 | <-- Wrapper Stopped
CauseHeavy resource usage may be caused by enabling features such as Automated Threat Detection for Logs in the ESA or when the source concentrators also have Group Aggregation configured on them.
ResolutionThis issue is resolved in RSA NetWitness 11.3 where the ESA service wrapper is no longer used. 
WorkaroundYou may need to change the ping time settings in the wrapper.conf file.

Perform the following:
  1. Connect to the ESA host via SSH as the root user.
  2. Edit the wrapper.conf file.

    vi /opt/rsa/rsa/conf/wrapper.conf

  3. Change the following setting (or add to bottom of file if line doesn't exist.

    wrapper.ping.timeout=300


     

  4. Add the following lines at the end of the file:

    wrapper.restart.delay=40
    wrapper.ping.timeout.action=RESTART

  5. Restart the Event Stream Analysis service.

    service rsa-esa restart

     

Attachments

    Outcomes