Article Number
000034136
Applies To
RSA Product Set: Access Manager
RSA Version/Condition: 6.1 SP4, 6.2, 6.2 SP1, 6.2 SP2, 6.2 SP3
Datastore: SQL
Issue
After a restart of SQL (Oracle) users experienced Access Denied messages and the RSA Access Manager aserver was maintaining bad connections to SQL.
Example of errors from the aserver.log file:
sequence_number=1541,2015-03-19 10:19:49:355 GMT-05:00,messageID=-2,internal_error,description='An error occurred in connecting to the database server.
[SQL Error : ORA-01033: ORACLE initialization or shutdown in progress
Code: (1033) State: (72000)]',details='sirrus.da.exception.DataStoreException: An error occurred in connecting to the database server.
[SQL Error : ORA-01033: ORACLE initialization or shutdown in progress
Code: (1033) State: (72000)]'
sequence_number=1543,2015-03-19 10:20:04:357 GMT-05:00,messageID=-2,internal_error,description='Unable to send data to receiver.',details='java.io.IOException: Unable to send data to receiver.'
Example of errors from aserver out file.
sirrus.da.exception.DataStoreException: SQLException : Code:17002;TextIo exception: Connection reset by peer: socket write error
at sirrus.da.sql.util.SQLEntityHelper.getUser(SQLEntityHelper.java:473)
at sirrus.da.sql.util.SQLConnectionImpl.get(SQLConnectionImpl.java:458)
at sirrus.da.sql.auth.factory.SQLEntityFactory.getUserByName_aroundBody2(SQLEntityFactory.java:136)
at sirrus.da.sql.auth.factory.SQLEntityFactory$AjcClosure3.run(SQLEntityFactory.java:1)
sirrus.da.exception.DataStoreException: SQLException : Code:17410;TextNo more data to read from socket
at sirrus.da.sql.util.SQLResourceHelper.getWebServer(SQLResourceHelper.java:1555)
at sirrus.da.sql.util.SQLConnectionImpl.get(SQLConnectionImpl.java:440)
at sirrus.da.sql.auth.factory.SQLWebServerFactory.getWebServerByName(SQLWebServerFactory.java:111)
at sirrus.da.auth.cache.factory.CachingWebServerFactory.getWebServerByName(CachingWebServerFactory.java:131)
at sirrus.da.auth.WebServer.getByName(WebServer.java:63)
at sirrus.authserver.AuthorizationAPI.addApplicationData(AuthorizationAPI.java:3593)
at sirrus.authserver.AuthorizationAPI.isUrlProtected(AuthorizationAPI.java:3550)
at sirrus.authserver.AuthorizationAPI.doCheckResource(AuthorizationAPI.java:1599)
at sirrus.authserver.AuthorizationAPI.checkResourceStatus(AuthorizationAPI.java:554)
at sirrus.authserver.DebugAuthorizationAPI.checkResourceStatus(DebugAuthorizationAPI.java:110)
at sirrus.authserver.TCPServerAPIAdaptor.checkResourceStatus(TCPServerAPIAdaptor.java:69)
Cause
With these defaults, the aserver on startup opens and keeps 10 connections open to oracle:
Reclaimconnections: true
Idletimeout: 300000 (5 minutes)
maxidleconnections : 10
keepalive : true
If the Oracle CT instance is restarted, these connections become bad, but are still kept in the connection pool until the watcher thread is run. (keepalive validation)
If the connections get leased and returned, that restarts the clock on them being idle.
In an environment under light load where the connections get removed until reduced to the minimum connections maintained. (determined by .maxidleconnections:)
These minimum connections only get removed if they have been idle for 5 minutes.
Because it is the watcher thread that closes these minimum connections that have been idle for 5 minutes, these bad connections never get closed if a user keeps trying to logon.
This is because the watcher thread never runs on them, as they are getting leased every time a user tries to logon.
Resolution
This can be resolved by restarting the aserver or by patching to version 6.2 Service Pack 3 Hotfix 04 or later.
