000038369 - Esper metrics collection can impact performance in some environments with ESA rules that consume large amounts of memory for RSA NetWitness Platform 11.4.x

Document created by RSA Customer Support Employee on Jan 24, 2020•Last modified by RSA Customer Support Employee on Apr 6, 2020
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038369
Applies ToRSA Product Set: NetWitness Platform
RSA Product/Service Type: ESA host/ESA Correlation service
RSA Version/Condition: 11.4.x
IssueMetric collection in Esper 8.2.0 is different than in the previous 7.1.0 version.
 

11.4.0.x



For an Event Stream Analysis (ESA) Correlation server with rules that consume a lot of memory, the gathering of metrics can consume significant CPU resources leading to a drop in Events Per Second (EPS) being processed when the metrics are being collected. To avoid the drop in EPS, the default interval to collect metrics in RSA NetWitness Platform 11.4.0.x is set to a large value (999999 days). This prevents the Esper metrics from being collected.


 

11.4.1 and higher



In a typical deployment, rule metrics calculations finish quickly, usually within seconds. If a rule uses a significant amount of memory, it may take a long time to calculate the metrics. During this time, the Event Stream Analysis (ESA) Correlation server does not analyze events and will result in an overall Events Per Second (EPS) processing drop. The ESA Correlation server will attempt to calculate metrics for a maximum of 15 seconds (default) and if any rules have metrics that cannot be calculated in this time, an error is shown in the logs. The ESA Correlation server will then cancel the calculation to avoid further EPS drop. This results in a maximum of 15 seconds of analysis lost every 5 minutes (background-metrics-frequency).

CauseEsper upgrade from 7.1.0 to 8.2.0 with RSA NetWitness Platform 11.4 and higher on the Event Stream Analysis server.
Workaround

11.4.0.x


If you need metrics that are collected at a more frequent interval, you can update the background-metrics-frequency parameter on the ESA Correlation service.

Do not set the metrics collection interval lower than five minutes.



  1. In the RSA NetWitness Platform UI,
    1. Go to Admin > Services.
    2. Select the ESA Correlation service.
    3. Select Action (Red Gear) > View > Explore. 
  2. In the Explore view node list on the left side, select Correlation > Esper.
  3. In the right panel, enter a new metrics collection interval value for background-metrics-frequency.
  4. Restart the ESA Correlation service.
    1. From the UI, go to Admin > Services, select the ESA Correlation service, and then select Action (Red Gear) > Restart. 
    2.  From the command line:



systemctl restart rsa-nw-correlation-server



 



11.4.1 and higher


If you need metrics collected at a more frequent interval, you can update the background-metrics-frequency and metrics-timeout parameters on the ESA Correlation service.
For example, if you have a rule that is using a lot of memory and it cannot be optimized, you can reduce the overall EPS drop by increasing the frequency and/or lowering the timeout.
  1. In the RSA NetWitness Platform UI,
    1. Go to Admin > Services.
    2. Select the ESA Correlation service.
    3. Select Action (Red Gear) > View > Explore. 
  2. In the Explore view node list on the left side, select Correlation > Esper.
  3. In the right panel, you can change the background-metrics-frequency and/or the metrics-timeout parameter value.
  4. Restart the ESA Correlation service.
    1. From the UI, go to Admin > Services, select the ESA Correlation service, and then select Action (Red Gear) > Restart. 
    2.  From the command line:



systemctl restart rsa-nw-correlation-server

Attachments

    Outcomes