000034576 - RSA NetWitness Event Source Monitoring (ESM) Beta is alerting and can cause server issues with operating environment in large deployments

Document created by RSA Customer Support Employee on Feb 23, 2018
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000034576
Applies ToRSA Product Set: NetWitness
RSA Product/Service Type: SA Event Source Monitoring
RSA Version/Condition: 10.6.1.1
 
IssueThe RSA NetWitness Event Source Monitoring (ESM) Beta is alerting and can cause server issues with operating environment in large deployments

This is caused by a new automatic alerting feature. This is a beta feature at present and turning it off will ONLY turn off the advanced baselining and automatic alerting; however regular ESM policy-based alerts will still work as usual (this issue does not affect SMS  policy-based alerting). Keeping it on, on the other hand, will cause all sorts of memory stability issues (given the number of resources needed to create and maintain baseline data for automatic monitoring and notifications) and the kernel may keep invoking the OOM killer to kill processes since it runs out of swap space.  

 
CauseThis is caused by the ESM alerting being enabled by default.
ResolutionTo resolve the issue,
  1. Disable this feature by navigating to Event Sources > Settings and remove all check marks.  For example, 

User-added image


  1. Perform the following tasks with the assumption that Automatic Monitoring is disabled):
    1. Stop the puppet agent:


service puppet stop


  1. Stop Jetty:


stop jettysrv


  1. Stop SMS:


service rsa-sms stop


  1. Run the following commands from a ssh session on the SA Head server:


[root@SERVER92 ~]# echo 'db.esmbaselinedata.drop()' | mongo esm
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: esm
true
bye
[root@SERVER92 ~]# echo 'db.aggregatedata.drop()' | mongo esm
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: esm
true
bye
[root@SERVER92 ~]# echo 'db.esmbaselineanalytics.drop()' | mongo esm
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: esm
true
bye
[root@SERVER92 ~]#


  1. Start SMS (service rsa-sms start):


service rsa-sms start


  1. Wait for 30 seconds
  2. Start Jetty:


start jettysrv


  1. Lastly, restart the puppet agent:


service puppet start


The baseline aggregation collections may come back in Mongo (get re-created), however, the automatic monitoring should remain disabled and the collections will not have any data in them.

Attachments

    Outcomes