Authentication Manager Administration Server with Operations Console Fails to Start – Error Keywords: CertificateExpiredException / BEA-000365 / BEA-000383 / Console Services Failed
17 hours ago
Originally Published: 2015-06-30
Article Number
000063056
Applies To
RSA Product Set:  SecurID
RSA Product/Service Type:  Authentication Manager 
RSA Version/Condition: 8.x
 
Issue

When attempting to connect to the RSA Authentication Manager administration consoles, all services fail to start and the Security Console, Operations Console, and Self-Service Console are inaccessible.

 

Observable symptoms:

  • RSA Authentication Manager services do not start
  • The Security Console, Operations Console, and Self-Service Console are not accessible
  • Attempting to restart services with ./rsaserv restart all fails at the same stage
  • Rebooting the server does not resolve the issue
  • The machine's hostname is resolvable and the IP address is correct
  • Date, time, and time zone on the server are all correct

When running ./rsaserv start console via SSH, vSphere, or direct connection, the following output is seen:

 

rsaadmin@am.primary:/opt/rsa/am/server> ./rsaserv start console
Starting RSA Administration Server with Operations Console:
Starting RSA Database Server: - RSA Database Server                    [RUNNING]
******* 
RSA Administration Server with Operations Console                      [FAILED]
Starting RSA Console Server ****** 
RSA Console Server                                                      [FAILED]

 

Cause

The console certificate installed on the RSA Authentication Manager server has expired, preventing the Administration Server with Operations Console service from starting.

 

This commonly occurs when a third-party console certificate is installed but its expiry date is not actively monitored. When the certificate expires, the WebLogic server fails to initialize the SSL context and shuts itself down, causing all dependent services to fail.

To confirm this cause, review the AdminServerWrapper.log file located at /opt/rsa/am/server/logs/.

 Look for the following key error lines:

 

Caused by: java.security.cert.CertificateExpiredException:
Checked date: <current date> is after Certificate notAfter date: <expiry date>

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>


<Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>
Resolution

 

Step 1: Connect to the RSA Authentication Manager server via SSH, vSphere, or direct connection.

NOTE: For SSH connection instructions, refer to  000038244 - How to SSH to an RSA Authentication Manager server.

 

Step 2: Navigate to the /opt/rsa/am/utils directory: 

cd /opt/rsa/am/utils

 

Step 3: Run the following command to revert the console certificate to the default RSA-supplied certificate:

./rsautil reset-server-cert -u <Operations Console username> -p <Operations Console password>

 

Step 4: Navigate to the /opt/rsa/am/server directory:

cd /opt/rsa/am/server

 

Step 5: Start all RSA Authentication Manager services:

./rsaserv start all

 

Step 6 (Verification):  Open a browser and confirm the Security Console and Operations Console are accessible and loading correctly.

NOTE: After services are restored, import a new valid console certificate via the Operations Console under Deployment Configuration > Certificates > Console Certificate Management to replace the expired certificate.

 

 

Notes

Expired Certificate Status: After reverting to the default certificate, the expired certificate will be listed as Inactive in the Operations Console under Deployment Configuration > Certificates > Console Certificate Management. 

Verified in RSA Labs icon.png