Replica promotion for maintenance fails with certificate not verified error in RSA Authentication Manager 8.4

[#1]

Article Number

000039248

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.4.0.0, 8.3.0.0, 8.2.1.0.0
Platform: Linux

Issue

Replica promotion when primary has replaced console certificate from Public CA but replica has default RSA self-signed console certificate. Replica needs to Trust primary's replacement console cert, but replica promotion looks in /opt/rsa/am/server/security/trust.jks not /opt/rsa/am/server/security/webserver-inactive.jks

• Attempting promotion for maintenance in the replica Operations Console fails Pre-Promotion checks:

Image description

Task status
Pre-promotion checks

Checking that services are running on this instance….
SUCCESS. 

Checking Replication status on this instance….
SUCCESS.
ERROR: The Operations Console on the primary instance is not reachable to check replication status or reachability with other instances.
ERROR: Could not access HTTP invoker remote service at [https://RSAprimary.abccompany.com:7072/operations-console/dispatcher/HttpInvokerPlannerPromotion]; nested exception
javax.net.ssl.SSLException: Certificate not verified
SUCCESS: The software version of this instance matches the primary instance...

• Checking that the primary instance is reachable and healthy….
• Attempting to reach the Operations Console on the primary instance: am83p.vcloud.local….
• Checking that all instances are reachable and healthy….
• Checking continueonerror replication state on: am83p.vcloud.local...
• Checking replication status of replica instances and reachability to other replica instances….

ERROR: The Operations Console on the primary instance is not reachable to check replication status or reachability with other replica instances.

Checking the replication status of all RADIUS servers….
ERROR: Could not access HTTP invoker remote service at [https://am83p.vcloud.local:7072/operations-console/dispatcher/HttpInvokerPlannedPromotion]; nested exception is javax.net.ssl.SSLException: Certificate not verified.

Cause

The original RSA Authentication Manager primary server has a replacement console certificate, while the replica being promoted has RSA self-signed console certificate. Because of this the replica does not trust the primary replacement console certificate.

Resolution

To resolve this issue,

1. Use WinSCP or FileZilla to copy the primary replacement console root CA signing certificate file to the /tmp directory on the replica.  

If you need to obtain the RootCA file from the original primary, refer to KB 000032384 - Obtain the RSA root CA certificate from RSA Authentication Manager 8.x.

2. Obtain the SSL Trust Store File Password on the replica.
   1. First, enable Enable Secure Shell on the Appliance.
   2. Then log On to the Appliance Operating System with SSH.
   3. Go to /opt/rsa/am/utils.
   4. Run the following command:

./rsautil manage-secrets -a listall

Image description

5. Scroll down the list to find the SSL Trust Store File Password. This value is different in each deployment of RSA Authentication Manager.

 

3. Make a backup of your /opt/rsa/am/server/security/trust.jks file on the replica. 
   1. In SSH,

cd ../server/security
cp trust.jks trust.jks.bak_aug17

4. List the contents of the /opt/rsa/am/server/security/trust.jks on the replica:

   ../../appserver/jdk/bin/keytool -list -keystore ./trust.jks

   No password needed for list
   Image description
5. Import the Primary replacement Console Root CA signing Certificate file into trust.jks Java Key Store file on the replica with keytool -importcert
       ../../appserver/jdk/bin/keytool -importcert -keystore ./trust.jks -file /tmp/am81p_2020_RootCA2.cer
   Enter Keystore password: s6TD7qb7M91kYWa5YoIdey8vvjPIMC  DOES NOT DISPLAY
   Trust  this certificate? [no]:  yes
   Certificate was added to keystore
   Image description
6. Verify Primary's Root CA cert imported successfully by listing the contents of the /opt/rsa/am/server/security/trust.jks on the replica again.
       ../../appserver/jdk/bin/keytool -list -keystore ./trust.jks
   No password needed for list
   Your keystore contains 9 entries.
   Alias name: mykey
   Creation date: Aug 17, 2020
   Entry type: trustedCertEntry
   Owner: CN=2k8r2-vcloud-2K8R2-DC1-CA, DC=2k8r2-vcloud, DC=local
   Issuer: CN=2k8r2-vcloud-2K8R2-DC1-CA, DC=2k8r2-vcloud, DC=local
   Valid from: Tue May 21 16:48:38 EDT 2019 until: Mon May 20 16:58:37 EDT 2024
   
    

Workaround

Revert the original Primary replace console certificate back to RSA self-signed.
  /opt/rsa/am/utils/rsautil reset-server-cert

See https://community.rsa.com/docs/DOC-46747 KB 000017506 - Reverting to the RSA self-signed default certificates on Authentication Manager 8.1 for details. 

Notes

1. RSA Support strongly recommends making backup copies of any Java Key Store, .JKS that your edit
2. RSA Support also strongly recommends against deleting any certificate or keys with keytool, as you could make your AM server inoperable.  BE VERY CAREFUL with keytool.  Open a Support Case for Assistance.
3. SSL Trust Store File Password is only displayed with ./rsautil manage-secrets -a listall 
        ./rsautil manage-secrets -a list com.rsa.ssl.trust.store.password does not provide anything of use, it only displays some default passwords
   
   Related info: https://community.rsa.com/docs/DOC-76463 KB 000035095 - How to delete old or pending certificate signing requests for RSA Authentication Manager console or virtual host replacement certificates.