RSA Access Manger is unable to open new sockets
Originally Published: 2011-10-03
Last Modified: 2023-10-06
Article Number
000047910
Applies To
RSA Product Set: Access Manager
RSA Product/Service Type: Access Manager
RSA Version/Condition: 6.1, 6.2​​​​​​​
 
Issue
RSA Access Manager is unable to open new sockets.
Error in aserver out.
Error connecting to the dispatcher at:server.com:5607
Too many open files
Retry in 10 seconds

Error in dispatcher.log
sequence_number=567151,remote_client=dispatcher1,2011-08-30 10:20:21:188 EDT,messageID=2001,event_type=Ping Failure,event_description=Ping failure for server,server=AuthServerDetails( Addr = aserver1, Port = 5615, Exp host = 192.168.2.10, Location Class = class1, Failure Cnt = 3 

When attempting to start the server instance the following error is generated:
Starting RSA Access Manager 6.1.4 Servers: /server-61/bin/aserver.sh: fork: retry: Resource temporarily unavailable
/server-61/bin/aserver.sh: fork: retry: Resource temporarily unavailable

When attempting to start the server instance the aserver.out shows the following exception:
20:36:30:863 [*] [main] - Resolved hostname to following IP(s) : 192.168.0.10
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Unknown Source)
        at netscape.ldap.LDAPConnThread.connect(Unknown Source)
Cause
The dispacher.log message indicates that the Manager failed to update its status with the dispatcher at the scheduled interval.  If the failure persists the aserver will be marked as being down. The error in the aserver.out indicates that the aserver was up and was responsive but was unable to open a TCP socket to the dispatcher to send the status update.   The failure was due to a resource problem on the physical machine hosting the aserver process.
Resolution
Run netstat or lsof -a and determine the number of sockets in use on the machine.  Audit the sockets opened by Access Manager and determine if they are logical based on the number of clients of the aserver.  Typically the aserver will require several sockets per agent.  Some agents and monitoring programs may require substantially more sockets.  Determine the actual number of sockets in use during peak loads and set the operating system value for "maximum file handles" to a reasonable value based on your requirements.
The UNIX command ulimit -n will list the current number of file handles allocated to the process.  The ulimit can be set per user, so the command should be executed using the account used to launch the aserver.  The ulimit command can also be used to set the values for the maximum number of file handles.  On UNIX each TCP socket requires a file handle.  File handles are also required for processes and log files.  The default setting for max file handles on some UNIX systems may be 1024 which is insufficient for an RSA Access Manager Aserver.  Set this limit to 2048 or higher depending on the number of actual sockets in use.

If the ulimit cannot be set high enough, the hard limits may need to be raised by editing the following in the  /etc/security /limits.conf file. 
 
* soft nofile 256000
* hard nofile 256000
Don't see what you're looking for?