Upgraded 8.4.0.13.0 SecurID web tier unavailable.

[JamesClark3]

The RSA Authentication Manager 8.4 Patch 13 was applied successfully along with RSA Authentication Agent for Web for IIS to 8.0.4.22. Remote web servers required node secrets resynced but all IIS web sites are operating normally after upgrades.

The web tier is inexplicably unavailable though. It was also updated and reinstalled and the process seemed to go normally including the Operations Console showing the updated 8.4.0.13.0 version now online. In the webtier install log, it finishes with 'Configuration step WebtierBootstrapper:installAll [SUCCESS]' and all else looks normal.

Web browsers to the expected domain/IP address at port 443 find nothing responsive. Doing a 'netstat -an' on the server shows no process opening port 443 on any IP address or local interface.

The webtier imsTrace.log shows both the Spring Resource and Webtier Bootstrapper Started and the wrapper-3.2.3rsa4.exe process open twice on the system. However, those PIDs are open on ports 32002 (Bootstrapper) and 32001 (RSA_WEBTIER_devfull).

How can I tell what went wrong and how to fix it? I've uninstalled and reinstalled, trying going back a version and reinstalled the latest version and all with the same result. I've looked for any other web tier log files with relevant info to no avail. Thanks for any help!

-Jim

[JamesClark3]

I'm following up on this with some more troubleshooting info that may be helpful. To eliminate the Web Tier deployment as the issue, I deleted the existing Web Tier Deployment from the Operations Console and created a new Web Tier with a new Deployment Name but with everything else the same. I then reinstalled the web tier using the new deployment package. The same issue is resulting with the Webtier and Webtier_Bootstrapper processes starting at ports 32001 and 32002 but nothing opens anything on port 443. The Operations Console shows the web tier online at version 8.4.0.13.0.

Also, in Console Certificate Management, there are two Signer Certificates with a recent expiration of 5/30/20. Both Issued By ' CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE ' and one has a subject of 'CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE' and the other with a subject 'CN=USERTrust RSA Certification Authority, O=The USERTRUST Network, L=Jersey City, ST=New Jersey, C=US'.

Could the expired Signer Certificates be the reason the web tier isn't starting? If so, can the AddTrust External CA Root certificates be safely deleted? Sorry this is starting to drift a bit from the original post topic but considering that an expired cert will prevent the Authentication Manager from starting, perhaps the expired AddTrust certs are impacting the web tier start up? 

Hope this helps and thanks for any guidance on how to get the web tier listening again on port 443.

~Jim

[EdwardDavis]

If it's the cert:

In the webtier ../../server/logs directory, it would log a certificate issue, adminserver*.log, java.io.IOException: Identity certificate has expired

Certs for Webtier are in the Operations Console Virtual Host Certificate Management

Remember to wait after the webtier installs, for it to finalize. The installer will say completed but the webtier still needs to finish setup, and create the server directory, and create the services...that can take from 5 to 25 minutes depending on the server and other things. While waiting you will see traffic between the webtier and the primary on port 7022. You will not have port 443 up until this finalize completes.

[JamesClark3]

Edward,

I completely understand that it takes a little while after the web tier starts for port 443 to normally be opened up. As for last night's troubleshooting, port 443 is still not open many hours later.

Thanks for pointing me to AdminServer.log. I looked through it and found several of these errors:

####<Jun 11, 2020 2:20:28,512 AM PDT> <Error> <Server> <SecurID-AM> <AdminServer> <[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0419273e-2183-4366-9e78-a1ca31970562-00000017> <1591867228512> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002606> <The server is unable to create a server socket for listening on channel "VirtualHostChannel[2]". The address 127.0.0.1 might be incorrect or another process is using port 443: java.io.IOException: Identity certificate has expired: [
Version: V3
Serial Number: 26471149583208131559647911801012699958
SignatureAlgorithm: SHA384withRSA (1.2.840.113549.1.1.12)
Issuer Name: CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE
Validity From: Tue May 30 03:48:38 PDT 2000
To: Sat May 30 03:48:38 PDT 2020
Subject Name: CN=USERTrust RSA Certification Authority, O=The USERTRUST Network, L=Jersey City, ST=New Jersey, C=US
Key: RSA (1.2.840.113549.1.1.1)
Key value: ....

If I understand this correctly, it looks like the process is trying to open up port 443 and encounters a certificate error with the expired AddTrust certs. 

I did more research and found the following blog article at ssl.com that discusses replacement AddTrust certficates. I can add the new ones from AddTrust External CA Root Expired May 30, 2020 - SSL.com to the Virtual Host Certificate Management console. Would adding these certs and deleting the expired AddTrust certs rectify this? How do I delete the existing AddTrust certs since there is no option to do so in the Operations Console?

Thanks,

Jim

[AhmedAbouelnaga · Accepted solution]

Hello James,

Yes a certificate issue would not allow the server to listen on port 443 and thanks to find the below article to delete the old expired cert and add a new valid cert:

000035095 - How to delete old or pending certificate signing requests for RSA Authentication Manager... 

Then you need to create a new CSR from the Operations Console >> Deployment Configuration > Certificates > Virtual Host Certificate Management and import the certificates once received.

Once the certificates are imported, you will need to update the Webtiers from the Operations Console >> Deployment Configuration > Web-Tier Deployments > Manage Existing, then hit the update button for each Webtier and wait for completion. it would take few minutes for the webtier to start listening after adding the update and become accessible.

Hope this helps

Best Regards,
Ahmed Abouelnaga

[EdwardDavis]

In Operations Console, you choose any valid cert, then reinstall webtier, and the new cert is inside the webtier package file.

[JamesClark3]

Thank you! I cleaned out all certificates except the RSA ones which I marked as the active certificate. The Operations Console indicated a update was required of the web tier. I clicked the Update button and within 10 minutes, port 443 was opened and the web tier available again, no reinstall required.

I'll generate a new CSR and import a new virtual host certificate next so the cert is trusted again.

Thanks to both you and Edward for helping me figure this out and getting the web tier back online.

-Jim