000034242 - "Too many open files" and other Unix errors seen in RSA Identity Governance & Lifecycle  6.9.1 logs after deploying P15

Document created by RSA Customer Support Employee on Dec 21, 2016Last modified by RSA Customer Support on Oct 18, 2019
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000034242
Applies ToRSA Product Set: Identity Governance & Lifecycle 
RSA Version/Condition: 6.9.1 P15
 
IssueAfter upgrading from 6.9.1 Pxx to 6.9.1 P15, the system is unresponsive and keeps crashing with the following  symptoms:
  • User interface page is not loading properly.  The welcome page is initially distorted and cannot load/run javascripts or images. After multiple screen refreshes it eventually works.
  • The issue is occurring even though there is a very minimal number of active users (e. g., five active users).
  • The application keeps crashing, even after the application and operating system are restarted.
  • There are a lot of too many open files errors in the aveksaServer.log and Unix errors in the WebSphere SystemOut log. When the too many open files error message is written to the logs, it indicates that all available file handles for the process have been used (this includes sockets as well).
  • The following message may be displayed in the aveksaServer.log for a number of files when the process has exhausted the file handle limit:


java.io.FileNotFoundException: <file location/file name> (Too many open files)


  • The following message may be displayed in the SystemOut.log for a number of files when the process has exhausted the file handle limit:


java.io.FileNotFoundException: <file location/file name> (Too many open files)
[07/09/16 11:41:31:679 EST] 0000001f prefs        
W   Could not lock User prefs. Unix error code 24.
[07/09/16 11:41:31:680 EST] 0000001f prefs        
W   Couldn't flush user prefs: java.util.prefs.BackingStoreException:
Couldn't get file lock.


Please refer to RSA Knowledge Base Article 000030327 -- Artifacts to gather in RSA Identity Governance & Lifecycle to find the location of the log files for your specific deployment. 


The below commands may be used to determine if the number of open files limit is low in relation to the full list of open files and if the number of open files seems to be growing over time.
  • cat /proc/sys/fs/file-max to determine the maximum number of file handles for the entire system. 65536 is the minimum.
  • ulimit -Sn to report the soft limit of the maximum number of open file descriptors. Soft limits are simply the currently enforced limits.
  • ulimit -Hn to report the hard limit of the maximum number of open file descriptors. Hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
  • lsof –p <PID> > lsof.out to get the full list of open files for the Java process.
  • lsof –p <PID> -r <interval in seconds, 900 for 15 minutes> > lsof_interval.out to determine which files are opened and which are growing over a period of time.

Here is a sample session:



root# cat /proc/sys/fs/file-max
65536
root# su - oracle
oracle$ ulimit -Sn
1024 
oracle$ ulimit -Hn  
8192
root# lsof –p 32666 > lsof.out 
root# lsof –p 32666 -r 900 > lsof_interval.out


     
    CauseThis issue is caused by system configuration limitations indicating that all available file handles for the process have been used.

    This was brought about by one of the fixes in 6.9.1 P15, where the property file loading implementation did not handle the file stream properly, keeping multiple file descriptors in an open state for some time.
    ResolutionThis issue is resolved in the following RSA Identity Governance & Lifecycle patches:
    • RSA Identity Governance & Lifecycle 6.9.1 P19
    • RSA Identity Governance & Lifecycle 7.0.1 P02
    • RSA Identity Governance & Lifecycle 7.0.2
     
    WorkaroundThe workaround for this issue is to increase the ulimits at the operating system level.  To do this, login as root and edit the /etc/security/limits.conf  to increase the limits and then restart RSA Identity Governance & Lifecycle.
    1. Edit /etc/security/limits.conf, adding the following lines or editing them if they already exist. Set soft nofile to 16384 and hard nofile to 65536.
    2. Change to the oracle user and confirm the new values:


    ulimit -Sn
    ulimit -Hn


    1. Restart RSA Identity Governance & Lifecycle as the oracle user.


    root# vi /etc/security/limits.conf
    oracle soft nofile 16384
    oracle hard nofile 65536

    root# su - oracle
    oracle$ ulimit -Sn
    16384
    oracle$ ulimit -Hn
    65536
    oracle$ acm restart 


    Please refer to RSA Knowledge Base Article 00003869 -- What are the recommended ulimit settings in /etc/security/limits.conf for use with RSA Identity Governance & Lifecycle for more information on setting ulimits.

    Attachments

      Outcomes