000034580 - Aggregation fills up memory due to large queries from investigations and reports in RSA NetWitness Logs & Packets

Document created by RSA Customer Support Employee on Dec 20, 2016Last modified by RSA Customer Support on Jul 24, 2018
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000034580
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Server
RSA Version/Condition: 10.4, 10.5, 10.6, 11.0
 
IssueA large query or several large queries run from a report, a dashboard, a chart or investigation and then aggregation fails to start and sessions fall behind.

Unable to allocate memory messages from aggregation and failed to allocate memory from MemPages is seen in the logs:

/var/log/messages:

[MemPages] [info] compacting memory
[MemPages] [info] memory compacted
[MemPages] [warning] Failed to allocate memory of size 2621440000 error Cannot allocate memory retrying in 1s
[MemPages] [warning] Failed to allocate memory of size 2621440000 error Cannot allocate memory retrying in 1s
[MemPages] [warning] Failed to allocate memory of size 2621440000 error Cannot allocate memory
[Aggregation] [failure] Failed to start aggregation thread because Unable to allocate any memory in MemPages constructor
[Aggregation] [failure] Unable to allocate any memory in MemPages constructor
Cause- query.timeout configuration may be set to 0.  Which, if set, a large query will never timeout and the query will wait until it finishes; potentially consuming memory until the service is reset.

- Too many dashboards or reports executing large queries from charts consume memory.

- Or, reports and charts are scheduled to run at high load times and need to be rescheduled.
ResolutionReconfigure query.timeout and confirm that it is not set to 0

In Administration > Services - select device > go to Explore and set:

Users >Accounts > [username] > query.timeout

[username] - The user account that is running the report or query.

Manage the user role and configure query.timeout with the procedures from the documentation below:
https://community.rsa.com/docs/DOC-84326

Enter an acceptable low timeout value for query.timeout in minutes preferably not 0.

Check dashboards, charts, and reports:

Where to disable charts:

Reports > Click on the Manage tab then Charts > Disable the charts by selecting the chart and clicking the disable blank circular icon.  Or, modify the rule used for the chart that would result in a more efficient query and use less memory: Reports > Click on the Manage tab then Rules > Select the associated rule for the chart whose query you would like to change > Click the Action tool icon and select Edit

Where to modify or delete reports:

The same location as Charts is for Reports but under the Manage tab then Reports > Select the report > Click the Actions tool icon and select Schedule Report or modify the Reports by clicking on the edit icon or delete the Report by clicking the delete icon.

Where to disable the corresponding dashboards for these charts:

Dashboards > Click on the Edit drop down > Select Manage Dashboards > Start selecting the dashboard you would want to disable and check the disable select box.

Attachments

    Outcomes