000031246 - Concentrator service is taking a long time to initialize in RSA Security Analytics

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 14, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000031246
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Concentrator, Hybrid Concentrator, Security Analytics UI
RSA Version/Condition: 10.5.x,10.4.2
Platform: CentOS
O/S Version: EL6
IssueWhen attempting to start or restart the Concentrator service (nwconcentrator) running RSA Security Analytics 10.5.x or 10.4.2, it takes a long time to fully initialize. In some situations it may take 15-20 minutes or longer for the REST port (50105) to initialize before the service is fully functional.  In other cases, meta may take upwards of several minutes to load in Investigator.
CauseThis issue occurs because the Concentrator service at version 10.5.x or 10.4.2 performs more work during index save and load operations in order to avoid having to perform the same work during each query.  
While this change results in shorter query times, it also contributes to longer initial load times.
This can be more noticeable on hybrid appliances, as the indexes reside on drives that are much slower than the solid state drives on standalone Concentrator appliances.
WorkaroundIf the index load operations take longer than 10 minutes, this could be due to one of the following reasons:
  • The Concentrator service isn't shutting down cleanly.
  • The number of index slices is much higher than it needs to be.
    NOTE:  On a hybrid appliance, this si only possible if the event rate is low and the retention time is long (i.e. greater than 6 months).
To ensure that the Concentrator service is able to shut down properly, perform the steps below.
  1. Connect to the Concentrator appliance via SSH as the root user.
  2. Use the vi editor to edit the startup/shutdown script for the nwconcentrator service.
    vi /etc/init/nwconcentrator.conf

  3. Modify the kill timeout value to be 300 rather than 60, as shown below.
    Screenshot of the /etc/init/nwconcentrator.conf contents, identifying the "kill timeout 60" line.
  4. Save the changes by typing :wq! and pressing the Enter key.
To reduce the number of index slices, perform the steps below.
  1. In the Security Analytics UI, navigate to the Administration -> Services page.
  2. Click on the red Actions button for the Concentrator service and select View -> Explore.
  3. In the directory tree on the left, expand the index node and click on config.
  4. In the right pane, change the value for save.session.count to be 600000000 and then click out of the field to apply the change.
    Screenshot of the save.session.count value in the Concentrator Explore view.
    NOTE:  This is the default for new installs at version 10.5.x.
  5. Click on Explore in the black menu bar and select Config to navigate to the Concentrator Configuration page.
  6. Click on the Files tab and select scheduler from the drop down menu.
  7. Comment out the following line from the text, as the change in Step 4 removes the need to schedule index saves:  hours=8 pathname=/index msg=save
    The scheduler file on the Concentrator Config page.
  8. Click on the blue Apply button.
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.
NotesIndex Rebuild

In rare cases, a Core service might benefit from an index rebuild. Examples:

  • The Security Analytics Core service has index slices created by a very old version of the product and has not rolled out any data in more than six months.
  • The index was configured incorrectly, and the customer wants to re-index all meta with a new index configuration.
  • The traffic load into the Core service was very light, and the save interval was large, causing more slices than needed to be generated.

In these cases, an index rebuild may provide performance improvements. To do so, you must send the message reset with the parameter index=1 to the /decoder folder on a Decoder, the /concentrator folder on a Concentrator, or the /archiver folder on an Archiver.

Be aware that a full re-index takes days to complete on a fully loaded Concentrator, and possibly weeks on a full Archiver.
In instances where data roll is low, and a large number of indexes of small amounts of data has occurred (Iow bandwidth consumption systems), performance will only increase after a re-index to take advantage of the larger data chunk sizes.  See the article How to perform a manual data or index reset on an RSA NetWitness appliance for how to perform a manual index reset.

The index save/load times has been improved significantly in Security Analytics 10.6, but there will still be an unavoidable index load time on the initial boot of the service.
This is also noted in the Optimization Techniques section of the Security Analytics 10.5 User Guide.