|Applies To||RSA Product Set: Security Analytics|
RSA Product/Service Type: Concentrator, Hybrid Concentrator, Security Analytics UI
RSA Version/Condition: 10.5.x,10.4.2
O/S Version: EL6
|Issue||When attempting to start or restart the Concentrator service (nwconcentrator) running RSA Security Analytics 10.5.x or 10.4.2, it takes a long time to fully initialize. In some situations it may take 15-20 minutes or longer for the REST port (50105) to initialize before the service is fully functional. In other cases, meta may take upwards of several minutes to load in Investigator.|
|Cause||This issue occurs because the Concentrator service at version 10.5.x or 10.4.2 performs more work during index save and load operations in order to avoid having to perform the same work during each query. |
While this change results in shorter query times, it also contributes to longer initial load times.
This can be more noticeable on hybrid appliances, as the indexes reside on drives that are much slower than the solid state drives on standalone Concentrator appliances.
|Workaround||If the index load operations take longer than 10 minutes, this could be due to one of the following reasons:|
To ensure that the Concentrator service is able to shut down properly, perform the steps below.
- The Concentrator service isn't shutting down cleanly.
- The number of index slices is much higher than it needs to be.
NOTE: On a hybrid appliance, this si only possible if the event rate is low and the retention time is long (i.e. greater than 6 months).
To reduce the number of index slices, perform the steps below.
- Connect to the Concentrator appliance via SSH as the root user.
- Use the vi editor to edit the startup/shutdown script for the nwconcentrator service.
- Modify the kill timeout value to be 300 rather than 60, as shown below.
- Save the changes by typing :wq! and pressing the Enter key.
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.
- In the Security Analytics UI, navigate to the Administration -> Services page.
- Click on the red Actions button for the Concentrator service and select View -> Explore.
- In the directory tree on the left, expand the index node and click on config.
- In the right pane, change the value for save.session.count to be 600000000 and then click out of the field to apply the change.
NOTE: This is the default for new installs at version 10.5.x.
- Click on Explore in the black menu bar and select Config to navigate to the Concentrator Configuration page.
- Click on the Files tab and select scheduler from the drop down menu.
- Comment out the following line from the text, as the change in Step 4 removes the need to schedule index saves: hours=8 pathname=/index msg=save
- Click on the blue Apply button.
In rare cases, a Core service might benefit from an index rebuild. Examples:
- The Security Analytics Core service has index slices created by a very old version of the product and has not rolled out any data in more than six months.
- The index was configured incorrectly, and the customer wants to re-index all meta with a new index configuration.
- The traffic load into the Core service was very light, and the save interval was large, causing more slices than needed to be generated.
In these cases, an index rebuild may provide performance improvements. To do so, you must send the message reset with the parameter index=1 to the /decoder folder on a Decoder, the /concentrator folder on a Concentrator, or the /archiver folder on an Archiver.
Be aware that a full re-index takes days to complete on a fully loaded Concentrator, and possibly weeks on a full Archiver.
In instances where data roll is low, and a large number of indexes of small amounts of data has occurred (Iow bandwidth consumption systems), performance will only increase after a re-index to take advantage of the larger data chunk sizes. See the article How to perform a manual data or index reset on an RSA NetWitness appliance for how to perform a manual index reset.
The index save/load times has been improved significantly in Security Analytics 10.6, but there will still be an unavoidable index load time on the initial boot of the service.
This is also noted in the Optimization Techniques section of the Security Analytics 10.5 User Guide.