000034242 - "Too many open files" and other Unix errors seen in RSA Identity Management and Governance 6.9.1 logs after deploying P15

Document created by RSA Customer Support Employee on Dec 21, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000034242
Applies ToRSA Product Set: Identity Management and Governance
RSA Product/Service Type: Enterprise Software
RSA Version/Condition: 6.9.1 P15
Platform: WebSphere
IssueAfter upgrading from 6.9.1 P13 to 6.9.1 P15, the system is unresponsive and keeps crashing with the following  symptoms:
  • UI page is not loading.  The welcome page is not loading properly initially (distorted UI), and then it works after couple of refreshes. It is not able to load *.js or images, etc.
  • The issue is occurring even though there is a very minimal number of active users (e. g., five active users).
  • The application keeps crashing, even after the application and OS are restarted.
  • There are a lot of "too many open files" errors in the aveksaServer.log and Unix errors in the WebSphere SystemOut log. When the "too many open files" error message is written to the logs, it indicates that all available file handles for the process have been used (this includes sockets as well).
  • The following messages could be displayed in the aveksaServer.log for a number of files when the process has exhausted the file handle limit:
java.io.FileNotFoundException: <file location/file name> (Too many open files)

  • The following messages could be displayed in the SystemOut.log for a number of files when the process has exhausted the file handle limit:
java.io.FileNotFoundException: <file location/file name> (Too many open files)
[07/09/16 11:41:31:679 EST] 0000001f prefs         W   Could not lock User prefs. Unix error code 24.
[07/09/16 11:41:31:680 EST] 0000001f prefs         W   Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.

  • It is determined that the number of open files limit is low in relation to the full list of open files, and the number of opened files seems to be growing over time.  To get this information, run the following commands:
    • cat /proc/sys/fs/file-max to determine the maximum number of file handles for the entire system, 65536 is the minimum.
    • ulimit -Sn to report the soft limit of the maximum number of open file descriptors. Soft limits are simply the currently enforced limits.
    • ulimit -Hn to report the hard limit of the maximum number of open file descriptors. Hard limits mark the maximum value which cannot be exceeded by setting a soft limit.
    • lsof –p <PID> > lsof.out to get the full list of open files for the Java process.
    • lsof –p <PID> -r <interval in seconds, 900 for 15 minutes> > lsof_interval.out to determine which files are opened and which are growing over a period of time.
A sample of the commands is below:

acm-691:~ # cat /proc/sys/fs/file-max
65536
acm-691:~ # su - oracle
oracle@acm-691:~> ulimit -Sn
1024 
oracle@acm-691:~> ulimit -Hn  
8192
acm-691:~ # lsof –p <PID> > lsof.out 
acm-691:~ # lsof –p <PID> -r <interval in seconds, 900 for 15 minutes> > lsof_interval.out

  • An excerpt from aveksaServer.log shows the following errors:
09/07/2016 11:50:18.330 ERROR (WebContainer : 1) [com.aveksa.UI] com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:107) -
com.ibm.ws.webcontainer.webapp.WebAppErrorReport: javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif &#40;Too many open files&#41;
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:153)
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:194)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:592)
     at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl.handleRequest(StaticFileServletWrapperImpl.java:176)
     at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:752)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
     at com.aveksa.gui.core.filters.CacheFilter.doFilter(CacheFilter.java:39)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:67)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:964)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1104)
     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3901)
     at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
     at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
     at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
     at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
     at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
     at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
     at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
     at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
     at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1865)
Caused by:
javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif (Too many open files)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:740)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:538)
... 29 more
Caused by:
java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif (Too many open files)
     at java.io.FileInputStream.open(Native Method)
     at java.io.FileInputStream.<init>(FileInputStream.java:137)
     at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapper.getInputStream(StaticFileServletWrapper.java:49)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeByBytes(FileServletWrapper.java:1057)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper._writeResponseToClient(FileServletWrapper.java:1024)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeResponseToClient(FileServletWrapper.java:933)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:712)
... 30 more

09/07/2016 11:50:18.330 ERROR (WebContainer : 1) [com.aveksa.UI] com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:107) -
com.ibm.ws.webcontainer.webapp.WebAppErrorReport: javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif &#40;Too many open files&#41;
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:153)
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:194)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:592)
     at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl.handleRequest(StaticFileServletWrapperImpl.java:176)
     at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:752)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
     at com.aveksa.gui.core.filters.CacheFilter.doFilter(CacheFilter.java:39)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:67)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:964)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1104)
     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3901)
     at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
     at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
     at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
     at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
     at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
     at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
     at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
     at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
     at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1865)
Caused by:
javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif (Too many open files)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:740)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:538)
... 29 more
Caused by:
java.io.FileNotFoundException: /
opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/images/table/tbl-header.gif
(Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:137)
at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapper.getInputStream(StaticFileServletWrapper.java:49)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeByBytes(FileServletWrapper.java:1057)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper._writeResponseToClient(FileServletWrapper.java:1024)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeResponseToClient(FileServletWrapper.java:933)
at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:712)
... 30 more


  • Excerpt from WebSphere's SystemOut.log:
[07/09/16 11:41:28:040 EST] 000001e3 SystemOut     O ERROR (WebContainer : 0) [UI] com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:107) -
com.ibm.ws.webcontainer.webapp.WebAppErrorReport: javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/js/Utils.js &#40;Too many open files&#41;
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:153)
     at com.ibm.ws.webcontainer.webapp.WebAppErrorReport.constructErrorReport(WebAppErrorReport.java:194)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:592)
     at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl.handleRequest(StaticFileServletWrapperImpl.java:176)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
     at com.aveksa.gui.core.filters.CacheFilter.doFilter(CacheFilter.java:39)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.aveksa.gui.core.filters.LoginFilter.doFilter(LoginFilter.java:67)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:964)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1104)
     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3901)
     at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
     at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
     at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
     at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
     at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
     at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
     at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
     at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
     at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1865)
Caused by:
javax.servlet.ServletException: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/js/Utils.js (Too many open files)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:740)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:538)
... 28 more
Caused by:
java.io.FileNotFoundException: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/PRODCell01/aveksa.ear/aveksa.war/js/Utils.js
(Too many open files)
     at java.io.FileInputStream.open(Native Method)
     at java.io.FileInputStream.<init>(FileInputStream.java:137)
     at com.ibm.ws.webcontainer.servlet.StaticFileServletWrapper.getInputStream(StaticFileServletWrapper.java:49)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeByBytes(FileServletWrapper.java:1057)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper._writeResponseToClient(FileServletWrapper.java:1024)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeResponseToClient(FileServletWrapper.java:933)
     at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:712)
... 29 more
[07/09/16 11:41:31:679 EST] 0000001f prefs         W   Could not lock User prefs. Unix error code 24.
[07/09/16 11:41:31:680 EST] 0000001f prefs         W   Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.


CauseThis issue was caused by system configuration limitations indicating that all available file handles for the process have been used.
This was brought about by one of the fixes in 6.9.1 P15, where the property file loading implementation did not handle the file stream properly, keeping multiple file descriptors in opened state for some time.
ResolutionFixed in Identity Governance and Lifecycle versions 6.9.1 P19, 7.0.0 and 7.0.1.
 
WorkaroundThe workaround for this issue is to increase the ulimits at the operating system level.  To do this, login as root and edit /etc/security/limits.conf  to increase limits and then restart ACM:
  1. Edit /etc/security/limits.conf, adding the following lines or editing them if they already exist. Set soft nofile to 16384 and hard nofile  to 65536.
  2. Change to the oracle user and confirm the new values.
  3. Exit the oracle user.
  4. These settings won't take effect until the application server and nodeagent are restarted, so restart ACM. 
root-691:~ # vi /etc/security/limits.conf
oracle soft nofile 16384
oracle hard nofile 65536

root-691:~ # su - oracle
oracle@acm-691:~> ulimit -Sn
16384
oracle@acm-691:~> ulimit -Hn
65536
oracle@acm-691:~> exit
logout
root-691:~ # su - oracle
oracle@acm-691:~ # acm restart 

Attachments

    Outcomes