All RSA Authentication Manager 8.2 servers in a deployment do not respond to authentication requests at the same time

[#1]

Article Number

000035928

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2 SP1 up to 8.3 base
O/S Version: 11.4
 

Issue

• All RSA Authentication Manager 8.2 servers do not respond to authentication requests and stopped authenticating at the same time.
• No users can authenticate.
• All services are running on all Authentication Manager servers and all replica servers are synchronized.
• Downloading the troubleshooting logs from Operations Console fails at the end while downloading system logs. However, when using an SSH connection, the RSA admin is able to provide the logs.
• The Authentication Manager server will appear to be running.
• The UDP socket will be up, TCP  is in the LISTEN state.
• The OS-level network receive queue byte count value (obtained from running netstat) will be non-zero.
• No UDP authentication requests of any kind will be processed (i. e., authentication agents will be unable to initialize communications).
• The syslog shows the following message:

ACEAGENT: The message entry does not exist for Message ID: 1002.

• If Error level tracing is enabled, the server will report the following:

Unexpected error while receiving datagram packet. java.lang.IllegalArgumentException: unsupported address type.

• Restarting the server restores the authentication for that specific server.

Cause

An as yet unknown network packet or event triggered an exception on the authentication service port.  Exception handing code changes caused the connection on which this exception was received to stop listening for network traffic on that connection, specifically 5500 UDP, the authentication service port. When this exception was encountered, instead of being handled it caused a second exception which broke authentications by making 5500 UDP unresponsive.

The current hot fix (as of 18 January 2018) which will be added to Authentication Manager 8.2 SP1 and Authentication Manager 8.3 patches now handles this network exception on 5500 UDP instead of triggering the second exception that broke authentications.

Resolution

This issue has been documented in defects AM-31699 and AM-31708 and it has been resolved in a hot fix. The fix detects this network condition/event along with a change to the code that handles the exception. Contact RSA Technical Support to obtain the hot fix.

Applying the hot fix

The hot fix is contained in the common-am-8.2.1.6.0.jar file. On all Authentication Manager 8.2 servers depending upon the patch level, the corresponding version of common-am-8.2.1.x.0.jar needs to be replaced. In this example, the server is running Authentication Manager 8.2 SP1 patch 6; hence, the common-am-8.2.1.6.0.jar file name is being used.

1. Connect to the Authentication  Manager server va SSH or direct conection.
2. Navigate to /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib

cd /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib

3. Rename the file to a backup:

mv common-am-8.2.1.6.0.jar common-am-8.2.1.6.0.jar.BAK

4. Copy the jar file from the solution to this directory from /tmp.  Please note the dot at the end of the command (if it is copied and saved in /tmp)

cp /tmp/common-am-8.2.1.6.0.jar .

5. Make sure there are two files in this directory: one BAK file and the one copied from /tmp.
6. Replace the same file in the following other directories:

cd /opt/rsa/am/server/servers/console/tmp/_WL_user/console-shared-library/t5l98w/WEB-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .
cd /opt/rsa/am/server/servers/AdminServer/tmp/_WL_user/console-shared-library/8hkrcb/WEB-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .
cd /opt/rsa/am/server/servers/radiusoc/tmp/_WL_user/am-radius-app/cbsd0y/APP-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .

7. Once the files are in place, restart the servers:

cd /opt/rsa/am/server
./rsaserv restart all

Verifying the hot fix

A server that encounters the condition and handles it using the updated logic will create a Warning trace message as follows: 

No address returned after receive() from channel.

Note that the trace level must be configured to the Warning level or this message will not be generated. 

Reverting the hot fix

1. Copy the backup file to the /tmp directory

cp /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib/common-am-8.2.1.6.0.jar.BAK /tmp/common-am-8.2.1.6.0.jar

2. Use the same command to replace the new .jar file with old one in the following directories:

cd /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .
cd /opt/rsa/am/server/servers/console/tmp/_WL_user/console-shared-library/t5l98w/WEB-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .
cd /opt/rsa/am/server/servers/AdminServer/tmp/_WL_user/console-shared-library/8hkrcb/WEB-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .
cd /opt/rsa/am/server/servers/radiusoc/tmp/_WL_user/am-radius-app/cbsd0y/APP-INF/lib
cp /tmp/common-am-8.2.1.6.0.jar .

3. Once the reverted files are in place, restart the Authentication Manager services:

cd /opt/rsa/am/server
./rsaserv restart all

Workaround

Reboot all Authentication Manager servers.

Note that rebooting a primary or replica server will correct the behavior of only that server.

Notes

• In order to apply this hot fix, RSA Authentication Manager must be upgraded to RSA Authentication  Manager 8.2 SP1 patch 6.
• If you have applied this hot fix, do not upgrade until this fix is included in a patch. 
• If installing patch 7 on a server with this hot fix, it will break the fix.  A new hot fix corresponding to patch 7 must be obtained from RSA and applied.
• This issue has been resolved in RSA Authentication Manager 8.2 SP1 patch 8. The release notes include: 

​​AM-31699 – In specific network environments, RSA Authentication Manager sometimes stopped responding to authentication requests on certain network ports, which prevented successful authentication until the server was restarted.

• If upgrading to RSA Authentication Manager 8.3, you must obtain a hot fix from RSA Customer Support for that version.
• This fix will be included in Authentication Manager 8.3 patch 1.