000034576 - Event Source Monitoring (ESM) Beta is alerting and can cause server issues with operating environment in large RSA NetWitness deployments

Document created by RSA Customer Support Employee on Feb 23, 2018•Last modified by RSA Customer Support Employee on Sep 15, 2018
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000034576
Applies ToRSA Product Set: RSA NetWitness Logs & Network, Security Analytics
RSA Product/Service Type: Event Source Monitoring (ESM)
RSA Version/Condition: 10.6.x, 11.0.x, 11.1.x
IssueThe RSA NetWitness Event Source Monitoring (ESM) feature is causing RSA NetWitness Head Unit server performance issues in large NetWitness deployments.

Symptoms on the RSA NetWitness Head Unit server include:
  • High memory usage (which can result in Health & Wellness alarms of "High Swap Utilization")
  • RSA NetWitness UI unavailability on port 443 due to the rsa-sms service crashing/becoming unresponsive

    NOTE: The rsa-sms service is a prerequisite of the jettysrv service.


  • High disk utilization on tokumx service volumes (e.g. /var/lib/netwitness/database/tokumx) due to large collection sizes in MongoDB

The following symptoms have also been reported in the RSA NetWitness 11.x user interface: (both may have other causes)
  • All hosts showing red status on the Admin > Health & Wellness > Monitoring tab (suggesting an issue with the rsa-sms service)
  • Large number of services showing as Offline in Admin > Services (suggesting that carlos is having trouble monitoring the service status)
CauseThis is caused by the ESM Alarms based on calculated baseline being enabled by default. (This feature is known as as ESM Automatic Monitoring.)
 
On large environments, a large amount of resources are required to create and maintain baseline data for automatic monitoring and notifications.  This is a beta feature at present and turning it off will ONLY turn off the advanced baselining and automatic alerting.
 
Features that are unaffected by disabling this include:
  • Regular ESM policy-based alerts
  • Health & Wellness policy-based alerting
ResolutionTo resolve this issue you must disable ESM Automatic Monitoring and remove some of the large ESM collections in the MongoDB by performing the following steps:
  1. Disable the feature in RSA Security Analytics 10.x by navigating to the Administration > Event Sources > Settings tab and deselect all checkboxes as shown in the example below.

User-added image


Disable this feature in RSA NetWitness 11.x by navigating Admin > Event Sources > Settings tab and deselect all checkboxes.



  1. Perform the following tasks after connecting to the RSA NetWitness Head Unit server via SSH.

    RSA Security Analytics 10.x



    1. Stop the puppet agent service to avoid service restart while performing maintenance.


    service puppet stop


    1. Stop the Jetty service.
       

      This will cause the RSA NetWitness UI to become temporarily unavailable.


        


    stop jettysrv


    1. Stop the SMS service.


    service rsa-sms stop

     

    RSA NetWitness 11.x



    1. Stop the nginx service.


    systemctl stop nginx.service      # could also use: service nginx stop


    1. Stop the Jetty service.


    systemctl stop jetty.service      # could also use: service jetty stop


    1. Stop the SMS service.


    systemctl stop rsa-sms.service    # could also use: service rsa-sms stop


    1. Make a backup of the ESM collections (optional).
       

      This requires that the rsa-sms service be stopped in order to obtain exclusive access to the MongoDB collection.


        


    backup_loc=~/$(date +"%Y%m%d").esm.backup
    mkdir -p "$backup_loc"



    mongodump -d esm -o "$backup_loc"    # NW 10.X
    mongodump -d esm -o "$backup_loc" -u deploy_admin -p <deploy_password> --authenticationDatabase admin   # NW 11.X


    Example Output:




    connected to: 127.0.0.1
    Thu Sep 13 03:17:42.235 DATABASE: esm    to     /root/20180913.esm.backup/esm
    Thu Sep 13 03:17:42.266         esm.system.indexes to /root/20180913.esm.backup/esm/system.indexes.bson
    Thu Sep 13 03:17:42.267                  92 objects
    Thu Sep 13 03:17:42.267         esm.esmalarm to /root/20180913.esm.backup/esm/esmalarm.bson
    Thu Sep 13 03:17:42.295                  0 objects
    Thu Sep 13 03:17:42.295         Metadata for esm.esmalarm to /root/20180913.esm.backup/esm/esmalarm.metadata.json
    Thu Sep 13 03:17:42.296         esm.eventsources to /root/20180913.esm.backup/esm/eventsources.bson
    Thu Sep 13 03:17:42.328                  18 objects
    Thu Sep 13 03:17:42.328         Metadata for esm.eventsources to /root/20180913.esm.backup/esm/eventsources.metadata.json
    Thu Sep 13 03:17:42.329         esm.esmbaselinedata to /root/20180913.esm.backup/esm/esmbaselinedata.bson
    Thu Sep 13 03:17:42.455                  432 objects
    Thu Sep 13 03:17:42.456         Metadata for esm.esmbaselinedata to /root/20180913.esm.backup/esm/esmbaselinedata.metadata.json
    Thu Sep 13 03:17:42.456         esm.esmaggregatedata to /root/20180913.esm.backup/esm/esmaggregatedata.bson
    Thu Sep 13 03:17:42.574                  3351 objects
    Thu Sep 13 03:17:42.574         Metadata for esm.esmaggregatedata to /root/20180913.esm.backup/esm/esmaggregatedata.metadata.json
    Thu Sep 13 03:17:42.575         esm.esmbaselineanalytics to /root/20180913.esm.backup/esm/esmbaselineanalytics.bson
    Thu Sep 13 03:17:42.698                  0 objects
    Thu Sep 13 03:17:42.698         Metadata for esm.esmbaselineanalytics to /root/20180913.esm.backup/esm/esmbaselineanalytics.metadata.json
    Thu Sep 13 03:17:42.699         esm.esmgroup to /root/20180913.esm.backup/esm/esmgroup.bson
    Thu Sep 13 03:17:42.777                  6 objects
    Thu Sep 13 03:17:42.777         Metadata for esm.esmgroup to /root/20180913.esm.backup/esm/esmgroup.metadata.json
    Thu Sep 13 03:17:42.777         esm.esmpolicy to /root/20180913.esm.backup/esm/esmpolicy.bson
    Thu Sep 13 03:17:42.777                  5 objects
    Thu Sep 13 03:17:42.777         Metadata for esm.esmpolicy to /root/20180913.esm.backup/esm/esmpolicy.metadata.json



    cd "$backup_loc"



    tar cvjpf $(date +"%Y%m%d").esm.backup.tar.bz2 "$backup_loc"/esm


    Example Output:




    tar: Removing leading `/' from member names
    /root/20180913.esm.backup/esm/
    /root/20180913.esm.backup/esm/esmbaselineanalytics.bson
    /root/20180913.esm.backup/esm/esmaggregatedata.metadata.json
    /root/20180913.esm.backup/esm/esmaggregatedata.bson
    /root/20180913.esm.backup/esm/esmgroup.metadata.json
    /root/20180913.esm.backup/esm/eventsources.metadata.json
    /root/20180913.esm.backup/esm/esmbaselinedata.metadata.json
    /root/20180913.esm.backup/esm/eventsources.bson
    /root/20180913.esm.backup/esm/esmbaselineanalytics.metadata.json
    /root/20180913.esm.backup/esm/esmpolicy.bson
    /root/20180913.esm.backup/esm/esmalarm.metadata.json
    /root/20180913.esm.backup/esm/esmgroup.bson
    /root/20180913.esm.backup/esm/esmbaselinedata.bson
    /root/20180913.esm.backup/esm/system.indexes.bson
    /root/20180913.esm.backup/esm/esmalarm.bson
    /root/20180913.esm.backup/esm/esmpolicy.metadata.json


    Clean up by removing the uncompressed files.




    rm -rf "$backup_loc"/esm


    1. Drop the large collections in the ESM MongoDB database
       

      NOTE: If you don't receive the output of true then there is likely a mistake in the collection name.


       
     

    RSA Security Analytics 10.x




    # echo 'db.esmbaselinedata.drop()' | mongo esm
    TokuMX mongo shell v1.4.2-mongodb-2.4.10
    connecting to: esm
    true
    bye
    # echo 'db.esmaggregatedata.drop()' | mongo esm
    TokuMX mongo shell v1.4.2-mongodb-2.4.10
    connecting to: esm
    true
    bye
    # echo 'db.esmbaselineanalytics.drop()' | mongo esm
    TokuMX mongo shell v1.4.2-mongodb-2.4.10
    connecting to: esm
    true
    bye

     

    RSA NetWitness 11.x




    # mongo esm -u deploy_admin -p <deploy_password> --authenticationDatabase admin
    MongoDB shell version v3.6.4
    connecting to: mongodb://127.0.0.1:27017/esm
    MongoDB server version: 3.6.4
    > db.esmbaselinedata.drop()
    true
    > db.esmaggregatedata.drop()
    true
    > db.esmbaselineanalytics.drop()
    true
    > exit
    bye

     

    RSA Security Analytics 10.x



    1. Restart the SMS service.


    service rsa-sms start


    1. Wait for 30 seconds.
    2. Restart the Jetty service.


    start jettysrv


    1. Restart the puppet agent service.


    service puppet start

 

RSA NetWitness 11.x




systemctl start rsa-sms.service    # could also use: service rsa-sms start



systemctl start nginx.service      # could also use: service nginx start



systemctl start jetty.service      # could also use: service jetty start


The dropped ESM collections may be recreated after the restart of the rsa-sms service. However, automatic monitoring should remain disabled and the collections will remain empty.


To avoid obtaining messages similar to the example below after disabling ESM Automatic monitoring, it is recommended that you restart the collectd service on all hosts running the logcollector service.


/var/log/messages:
Sep  4 04:13:22 logdecoder1 collectd[3960]: NgEsmReader_all: error getting ESM data for field "source" for device=ciscorouter. Reason: uninitialized

 
For RSA Security Analytics 10.x you can restart the collectd service on all hosts in the environment that are managed by puppet by issuing the command below on the RSA NetWitness Head Unit server.


mco service collectd restart

Example Output


Do you really want to operate on services unfiltered? (y/n): y
 * [ ============================================================> ] 12 / 12

Summary of Service Status:
   running = 12

Finished processing 12 / 12 hosts in 23682.16 ms
NotesFor RSA NetWitness 11.x and any 10.x hosts where the node id does not appear in the mco ping output, you may need to have the collectd service restarted manually using one of the commands below.

service collectd restart              # NW 10.X
systemctl restart collectd.service    # NW 11.X

Attachments

    Outcomes