RSA RADIUS Server Operations Console service for RSA Authentication Manager 8.2 and higher fails to start after upgrade

[#1]

Article Number

000036059

Applies To

RSA Product Set: SecurID  
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2, 8.3

Issue

The RSA RADIUS Server Operations Console service fails to start after an upgrade.

A search of the /opt/rsa/am/server/logs/radiusoc.log shows:   

weblogic.application.ModuleException: java.lang.OutOfMemoryError: Java heap space

Cause

A WebLogic Diagnostic Framework file has grown in size exceeding the limits defined in the configuration that Authentication Manager uses to run WebLogic.

The processing of this large file size causes the aforementioned OutOfMemory Error.  This has mostly been seen when patching, where the file has grown large and the patch process performs a stop and start of services.  It can happen any time a stop and start is done.  When the file is too large in size to impact the start up process varies. 
 

Resolution

1. Remove the excessively large file WLS_DIAGNOSTICS000000.DAT
2. Login to the RSA Authentication Manager server via SSH as rsaadmin. 
3. Navigate to /opt/rsa/am/
4. Run a find command as follows:

find . -iname WLS_DIAG*.DAT -exec ls -lh {} \+

5. Output will look similar to the following:

-rw------- 1 rsaadmin rsaadmin 86M Mar  7 16:00 ./server/servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
-rw------- 1 rsaadmin rsaadmin 86M Mar  7 16:00 ./server/servers/biztier/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
-rw------- 1 rsaadmin rsaadmin 86M Mar  7 15:59 ./server/servers/console/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
-rw------- 1 rsaadmin rsaadmin 1.4G Mar  7 15:59 ./server/servers/radiusoc/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT

6. Delete the file shown above, which is large.

cd server/servers/radiusoc/data/store/diagnostics/
rm WLS_DIAGNOSTICS000000.DAT

7. Navigate back to /opt/rsa/am/server:

cd /opt/rsa/am/server

8. Start all services:

​./rsaserv stop all
./rsaserv start all

Workaround

At this time the cause of the file size growth is unknown.  It is a benign file for the way RSA uses WebLogic.  Engineering is investigating the cause and remedial steps to constrain the growth of the file. 

This issue has been addressed in AM-31774 (Huge file WLS_DIAGNOSTICSxxxxxx.DAT preventing 3 services from starting).  The fix is included in  RSA Authentication Manager 8.3 Patch 1 or later (requires RSA Link registration).