000017823 - 'Too many open files' error is logging against the concentrator service in RSA Security Analytics

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000017823
Applies ToRSA Security Analytics
RSA Security Analytics Concentrator
RSA Security Analytics Hybrid
RSA Security Analytics All-in-One
RSA NetWitness NextGen
Issue"Too many open files" error is logging against the concentrator service in RSA Security Analytics
The following log message is found in the /var/log/messages file:  nw[5254]: [Index] [failure] Unexpected Query Exception of type N5boost12interprocess22interprocess_exceptionE: Too many open files
CauseThe default value for max open file for concentrator service default of 1024 is too small.
Resolution

In order to resolve the issue, follow the instructions below.

For appliances running CentOS 6:

  1. Add the following line to the /etc/init.d/nwconcentrator file after the line that says "limit core unlimited unlimited": limit nofile 100000 100000
  2. Stop the concentrator service by issuing the following command:  stop nwconcentrator
  3. Start the concentrator service by issuing the following command:  start nwconcentrator

 

For appliances running CentOS 5:

  1. Add the following line to the top of the /etc/init.d/nwconcentrator file after the opening comment block: ulimit -n 100000
  2. Stop the concentrator service by issuing the following command:  monit stop nwconcentrator
  3. Wait approximately two minutes, and then start the concentrator service by issuing the following command:  monit start nwconcentrator

 

If you still see the "too many open files" error in the log after making the above changes, please verify that the process has the correct configuration by looking at /proc/<pid>/limits where <pid> is the PID for the NwConcentrator process, or whatever process is issuing a "too many open files" error. If /proc/<pid>/limits still shows a low Max open files limit, then recheck your configuration and, if necessary, restart the nwconcentrator service or the whole appliance.

 

If you are unsure of any of the steps above or experience any issues, contact RSA Support for further assistance.

Notes

Below is an example of a concentrator appliance running CentOS 5 and NetWitness NextGen 9.8.5.17.

[root@concentrator ~]# ps -ef | grep Nw
root      7515  7513  0 Mar26 ?        00:00:00 /bin/bash -c ulimit -S -c unlimited >/dev/null 2>&1 ; /usr/sbin/NwAppliance >/dev/null
root      7516  7515  0 Mar26 ?        00:02:11 /usr/sbin/NwAppliance
root     16032 16030  0 12:33 ?        00:00:00 /bin/bash -c ulimit -S -c unlimited >/dev/null 2>&1 ; /usr/sbin/NwConcentrator >/dev/null
root     16033 16032 33 12:33 ?        00:00:03 /usr/sbin/NwConcentrator
root     16055 15955  0 12:33 pts/0    00:00:00 grep Nw
[root@concentrator ~]# cat /proc/16032/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             1187840              1187840              processes
Max open files            100000               100000               files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       1187840              1187840              signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

 


System-wide defaults can be seen by issuing the command sysctl fs.file-max on the appliance, as shown in the example below.

[root@concentrator ~]# sysctl fs.file-max
fs.file-max = 14663326

Legacy Article IDa64950

Attachments

    Outcomes