The RSA Access Manager servers are opening connections to the datastore but not closing any connections, eventually running out of connections.
In the aserver.log, the error "Data Store Error" will be followed by either "Unable to send data to receiver" or "This stream has been closed due to Socket Timeout."
Examples:
sequence_number=29,date=2015-10-28 10:24:07:642 EDT,messageID=9,user=62testuser,client_ip_address=192.168.221.128,client_port=63224,browser_ip_address=192.168.221.128,result_code=101,user=62testuser,user_dn=null,result_action=Data Store Error
sequence_number=30,date=2015-10-28 10:24:07:646 EDT,messageID=-2,internal_error,description='Unable to send data to receiver.',details='java.io.IOException: Unable to send data to receiver.'
sequence_number=45,date=2015-10-28 10:26:22:924 EDT,messageID=9,user=62testuser,client_ip_address=192.168.221.128,client_port=63299,browser_ip_address=192.168.221.128,result_code=101,user=62testuser,user_dn=null,result_action=Data Store Error
sequence_number=46,date=2015-10-28 10:26:22:924 EDT,messageID=-2,internal_error,description='This stream has been closed due to Socket Timeout.',details='java.io.IOException: This stream has been closed due to Socket Timeout.'
A defect was introduced in Access Manager Server Version 6.2 SP3. This defect occurs during the watcher thread when idle connections get validated. The watcher thread leases the connection to see if valid, but does not return the connection back to the "free" connections in the pool. The next request will have no free connections to use and a new connection will get opened.
This new connection eventually gets validated during the watcher thread and subsequently does not get returned to the free connections in the pool. This continues until the aserver reaches the Max connections configured to the data store (default 100).
Once reaching max connections, the aserver will no longer open new connection and since all leased connection are not getting freed, this will cause a catastrophic failure of the aserver.
A hotfix has been released to remedy this defect, which is 6.2.3.04.
Please open a case with
RSA Customer Support and reference this article number to request the latest patch that includes this fix.
