'Too many open files' and other Unix errors seen in the log files after deploying P15 on version 6.9.1 of RSA Identity Governance & Lifecycle

[#1]

Article Number

000034242

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle 
RSA Version/Condition: 6.9.1 P15
 

Issue

After upgrading from RSA Identity Governance & Lifecycle 6.9.1 Pxx to 6.9.1 P15, the system is unresponsive and keeps crashing with the following  symptoms:

• User interface page is not loading properly.  The welcome page is initially distorted and cannot load/run javascripts or images. After multiple screen refreshes, it eventually works.
• ​The issue is occurring even though there is a very minimal number of active users (e. g., five active users).
• The application keeps crashing, even after the application and operating system are restarted.
• There are multiple too many open files errors in the aveksaServer.log.
• If you are on WebSphere, the SystemOut log also has multiple too many open file errors as well as other UNIX errors.
• The following message may be displayed in the aveksaServer.log for a number of files when the process has exhausted the file handle limit:

java.io.FileNotFoundException: <file location/file name> (Too many open files)

• The following message may be displayed in the SystemOut.log for a number of files when the process has exhausted the file handle limit:

java.io.FileNotFoundException: <file location/file name> (Too many open files)
[07/09/16 11:41:31:679 EST] 0000001f prefs         
W   Could not lock User prefs. Unix error code 24.
[07/09/16 11:41:31:680 EST] 0000001f prefs         
W   Couldn't flush user prefs: java.util.prefs.BackingStoreException: 
Couldn't get file lock.

Please refer to RSA Knowledge Base Article 000030327 -- Artifacts to gather in RSA Identity Governance & Lifecycle to find the log files for your specific deployment. Certain log files such as the aveksaServer.log may also be downloaded from the RSA Identity Governance & Lifecycle user interface (Admin > System > Server Nodes tab > under Logs.)

The below commands performed as the root user may be used to confirm if the number of open files limit is low in relation to the full list of open files and if the number of open files seems to be growing over time.

• cat /proc/sys/fs/file-max to determine the maximum number of file handles for the entire system. 65536 is the minimum.
• ulimit -Sn to report the soft limit of the maximum number of open file descriptors. Soft limits are simply the currently enforced limits.
• ulimit -Hn to report the hard limit of the maximum number of open file descriptors. Hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
• lsof –p <PID> > lsof.out to get the full list of open files for the Java process.
• lsof –p <PID> -r <interval in seconds, 900 for 15 minutes> > lsof_interval.out to determine which files are opened and which are growing over a period of time.

Here is a sample session:

root# cat /proc/sys/fs/file-max
65536
root# su - oracle
oracle$ ulimit -Sn
1024 
oracle$ ulimit -Hn   
8192
root# lsof –p 32666 > lsof.out 
root# lsof –p 32666 -r 900 > lsof_interval.out

 

Cause

This is a known issue reported in engineering ticket ACM-67378.  

The too many open files error message that is written to the log files indicates that all available file handles for the process have been used (this includes sockets as well) due to system configuration limitations. This issue was introduced by one of the fixes in RSA Identity Governance & Lifecycle 6.9.1 P15 where the property file loading implementation did not handle the file stream properly, keeping multiple file descriptors in an open state for some time.
 

Resolution

This issue is resolved in the following RSA Identity Governance & Lifecycle versions and/or patch levels: 

• RSA Identity Governance & Lifecycle 6.9.1 P19
• RSA Identity Governance & Lifecycle 7.0.0
• RSA Identity Governance & Lifecycle 7.0.1 P01
• RSA Identity Governance & Lifecycle 7.0.2

 

Workaround

The workaround for this issue is to increase the ulimits at the Operating System level. To do this, login as root and edit the /etc/security/limits.conf  to increase the limits and then restart RSA Identity Governance & Lifecycle.

1. As the root user, edit /etc/security/limits.conf and add the following lines or edit the following lines if they already exist. Set soft nofile and hard nofile as indicated below:

oracle soft nofile 16384
oracle hard nofile 65536

2. Change to the oracle user and confirm the new values have been set:

su - oracle
ulimit -Sn
ulimit -Hn

3. Restart RSA Identity Governance & Lifecycle as the oracle user.

acm restart

EXAMPLE SESSION:

root# vi /etc/security/limits.conf
oracle soft nofile 16384
oracle hard nofile 65536
root# su - oracle
oracle$ ulimit -Sn
16384
oracle$ ulimit -Hn
65536
oracle$ acm restart  

Please refer to RSA Knowledge Base Article 000038069 -- What are the recommended ulimit settings in /etc/security/limits.conf for use with RSA Identity Governance & Lifecycle for more information on setting ulimits.