After upgrading from RSA Identity Governance & Lifecycle 6.9.1 Pxx to 6.9.1 P15, the system is unresponsive and keeps crashing with the following symptoms:
- User interface page is not loading properly. The welcome page is initially distorted and cannot load/run javascripts or images. After multiple screen refreshes, it eventually works.
- The issue is occurring even though there is a very minimal number of active users (e. g., five active users).
- The application keeps crashing, even after the application and operating system are restarted.
- There are multiple too many open files errors in the aveksaServer.log.
- If you are on WebSphere, the SystemOut log also has multiple too many open file errors as well as other UNIX errors.
- The following message may be displayed in the aveksaServer.log for a number of files when the process has exhausted the file handle limit:
java.io.FileNotFoundException: <file location/file name> (Too many open files)
- The following message may be displayed in the SystemOut.log for a number of files when the process has exhausted the file handle limit:
java.io.FileNotFoundException: <file location/file name> (Too many open files)
[07/09/16 11:41:31:679 EST] 0000001f prefs
W Could not lock User prefs. Unix error code 24.
[07/09/16 11:41:31:680 EST] 0000001f prefs
W Couldn't flush user prefs: java.util.prefs.BackingStoreException:
Couldn't get file lock.
Please refer to RSA Knowledge Base Article
000030327 -- Artifacts to gather in RSA Identity Governance & Lifecycle to find the log files for your specific deployment. Certain log files such as the
aveksaServer.log may also be downloaded from the RSA Identity Governance & Lifecycle user interface (
Admin >
System >
Server Nodes tab > under
Logs.)
The below commands performed as the
root user may be used to confirm if the number of open files limit is low in relation to the full list of open files and if the number of open files seems to be growing over time.
- cat /proc/sys/fs/file-max to determine the maximum number of file handles for the entire system. 65536 is the minimum.
- ulimit -Sn to report the soft limit of the maximum number of open file descriptors. Soft limits are simply the currently enforced limits.
- ulimit -Hn to report the hard limit of the maximum number of open file descriptors. Hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
- lsof –p <PID> > lsof.out to get the full list of open files for the Java process.
- lsof –p <PID> -r <interval in seconds, 900 for 15 minutes> > lsof_interval.out to determine which files are opened and which are growing over a period of time.
Here is a sample session:
root# cat /proc/sys/fs/file-max
65536
root# su - oracle
oracle$ ulimit -Sn
1024
oracle$ ulimit -Hn
8192
root# lsof –p 32666 > lsof.out
root# lsof –p 32666 -r 900 > lsof_interval.out
