000030676 - Administration server with Operations Console service fails to start for RSA Authentication Manager 8.x

The following is seen when trying to connect to the RSA Authentication Manager administration consoles:

• RSA Authentication Manager services do not start.
• The Security Console, Operations Console, and Self-Service Console are not accessible.
• When connected to the RSA Authentication Manager server using SSH, vSphere or a direct connection, the RSA Administration server with Operations Console service is failing to start, causing all other services to fail, except for the RSA Database Server service.

rsaadmin@am83p:/opt/rsa/am/server> ./rsaserv start console
Starting RSA Administration Server with Operations Console:
Starting RSA Database Server: - RSA Database Server                                        [RUNNING]

*******
RSA Administration Server with Operations Console         [FAILED]
Starting RSA Console Server ******
RSA Console Server                                         [FAILED]
rsaadmin@am83p:/opt/rsa/am/server>

In addition:

• Attempting to restart services with the command ./rsaserv restart all fails at the same stage.
• Rebooting the server does not resolve the issue.
• The machine’s hostname is resolvable and the IP address is correct.
• Date, time, and time zone on the server are all correct.

The /opt/rsa/am/server/logs/AdminServerWrapper.log information shows that the console certificate has expired. In the example below, the date stamp on the log is 30 June 2015, but the certificate expired on 21 May 2015. The error message is called out in red.      

6 7a b3 [.5.E.?....ey..z.]
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |    00f0: 5c 2a a8 f1 16 38 c9 3c c8 a9 8c db 6d d
6 96 e2 [\*...8.>....m...]
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |
INFO | jvm 1      | main    | 2015/06/30 6:18:24 | ]
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.security.utils.SSLContext Manager
.fail(SSLContextManager.java:703)
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.security.utils.SSLContext Manager
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.security.utils.SSLContext Manager
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.security.utils.SSLContext Manager
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.security.utils.SSLContext Manager
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      at weblogic.server.channels.DynamicJSSLEListe
nThread.<init>(DynamicJSSLEListenThread.java:50)
INFO | jvm 1      | main    | 2015/06/30 6:18:24 |      ...7 more
INFO | jvm 1      | main    | 2015/06/30 6:18:24 | Caused by: java.security.cert.CertificateExpired
Exception: Checked date:  Tue Jun 30 06:18:23 EDT 2015 is after Certificate notAfter date: Thu May 21 
22:28:48 EDT 2015.
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     at com.rsa.cryptoj.c.pk.a(UnknownSource)
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     at com.rsa.cryptoj.c.pj.checkValidity(Unknown Source)
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     at weblogic.security.utils.SSLContextManager.checkIdentity
(SSLContextManager.java.508)
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     ... 11 more
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 | >
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     <Jun 30, 2015 6:18:24 AM EDT> <Notice> <Weblogic 
Server> <BEA-000365> <Server state changed to FAILED.>
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     <Jun 30, 2015 6:18:24 AM EDT> <Error> <WeblogicS
erver> <BEA-000383> <A critical service failed. The server will shut itself down.>
INFO   | jvm 1    | main    | 2015/06/30 06:18:24 |     <Jun 30, 2015 6:18:24 AM EDT> <Error> <WeblogicS
erver> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>
STATUS | wrapper  | main    | 2015/06/30 06:18:26 | <-- Wrapper Stopped
rsaadmin@am83p:/opt/rsa/am/server/logs     

To resolve this issue, revert to the default certificate that ships with RSA Authentication Manager and then import a new console certificate. The steps to revert to the original certificate supplied by RSA are shown below:

1.  Connect to the RSA Authentication Manager server using SSH, vSphere, or direct connection. Instructions can be found in article 000038244 - How to SSH to an RSA Authentication Manager server.
2.  Go to /opt/rsa/am/utils.
3.  To change the console certificate from the third-party certificate to the original certificate, run the command below:

./rsautil reset-server-cert -u <Operations Console user> -p <Operations Console password>

4. After reverting the default certificate, go to /opt/rsa/am/server and start the RSA Authentication Manager services:

./rsaserv start all

After reverting to the default certificate, the expired certificate will be listed as Inactive in the Operations Console under Deployment Configuration > Certificates > Console Certificate Management.

Screenshot of error in logs