Backup restored from different RSA Authentication Manager 8.x deployment breaks Agent API 8.5 agents including RSA SecurID Access

[#1]

Article Number

000032022

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x

Issue

After restoring a backup from another RSA Authentication Manager server in the Operations Console, existing RSA Authentication Agent API 8.5 agents, which uses TCP port 5500, do not work and nothing shows in the authentication logs, no authentication request is sent from the Authentication Agent API 8.5 to the Authentication Manager server.

With the /opt/rsa/am/server/logs/imsTrace.log file set to verbose, the following messages are found:

2015-11-17 19:34:50,954, [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'],
(AgentConfigPolicyAdministrationImpl.java:17),
trace.com.rsa.authmgr.internal.admin.configmgt.impl.AgentConfigPolicyAdministrationImpl,
DEBUG, rsa8.<domain>.com.local,,,,
Processing request for agent configuration.

2015-11-17 19:34:50,954, [[ACTIVE] ExecuteThread: '0' for queue:
'weblogic.kernel.Default (self-tuning)'],
(AgentConfigPolicyAdministrationImpl.java:17),
trace.com.rsa.authmgr.internal.admin.configmgt.impl.AgentConfigPolicyAdministrationImpl,
DEBUG, rsa8.<domain>.com.local,,,,
Agent with the given name: 192.168.1.251 was found.

2015-11-17 19:34:50,955, [[ACTIVE] ExecuteThread: '0' for queue:
'weblogic.kernel.Default (self-tuning)'], (DataObjectAccessSql.java:932),
trace.com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql, INFO, rsa8
<domain>.com.local,,,,
Executing HQL: select dataObject from com.rsa.authmgr.internal.admin.configmgt.dal.AgentConfigPolicy
as dataObject where dataObject.realmDefault = :param1

2015-11-17 19:34:50,956, [[ACTIVE] ExecuteThread: '0' for queue:
'weblogic.kernel.Default (self-tuning)'], (AgentConfigPolicyAdministrationImpl.java:17),
trace.com.rsa.authmgr.internal.admin.configmgt.impl.AgentConfigPolicyAdministrationImpl,
DEBUG, rsa8.<domain>.com.local,,,,
Agent configuration update is required for agent 192.168.1.251 as a new copy was found.

Cause

Before importing and restoring a database from another RSA Authentication Manager 8.x server, note the following system changes:

1. Change the internal database super admin account and password to the RSA Authentication Manager 8.x instance from which the backup came.
2. Change the Operations Console administrator account and password to the one from which the RSA Authentication Manager 8.x instance the backup came.
3. Add an inactive console certificate from the the RSA Authentication Manager 8.x instance from which the backup came, including the root CA. In this way, the consoles will work, but the accounts have changed.
4. Change the agent cert in Security Console:
   1.  Go to Setup > System Settings.
   2. Under Authentication Settings, click the Agents link.
   3. Click the here link to configure agents using IPV6.
   4. Change the agent certs to be the same as the instance from which the back up came.
5. Any existing API 8.5 agents (including RSA SecurID Access) that existed before the database restore will no longer work because they are encrypting with the original RSA Authentication Agent certs, so the agent will not be able to build the SSL connection and therefore, will not send any authentication requests
6. Any sdconf.rec generated from now on will contain the root CA from the RSA Authentication Manager 8.x instance from which the backup came.

Resolution

To get the old agent API 8.5 agents to work,

1. Generate and download a new sdconf.rec (Access > Authentication Agents > Generate Configuration File) and replace the existing sdconf.rec on each agent.
2. Go to Setup > System Settings. Under Authentication Settings, click the Agents link.
3. Click the here link to configure agents using IPV6. 
4. Review the cert imported, make sure if it's correct, or change it otherwise.
5. Click Save.

You may need to rebalance (Access > Authentication Agents > Authentication Manager Contact List > Automatic Rebalance and click the Rebalance button) to be safe.

Workaround

If you do not want the new API 8.5 agents from the imported/restored database, import the original agent certificate back into the RSA Authentication Manager 8.x server.

Notes

The root CA cert is included in the sdconf.rec file and is used by Agent API 8.5 agents; that is, the TCP 5500 authentication traffic (including RSA SecurID Access agents), instead of the old UDP 5500 authentication traffic.

When an RSA Authentication Manager 8.x database is restored to another RSA Authentication Manager 8.x server, there is a confusion over which root CA to use in the sdconf.rec:

• The one originally created at deployment, or
• The one in the restored backup.

This means that restoring an RSA Authentication Manager 8.x backup from another system breaks any TCP 5500 port Agent API 8.5 type traffic.

To fix this situation, you must extract the root CA cert from the original RSA Authentication Manager 8.x backup and import it into the new RSA Authentication Manager 8.x server using the steps below:

1. Through the Security Console (Setup > System > Agents).
2. Under Authentication Settings, click the Agents link then click the here link to configure agents using IPV6.

Image description

3. Scroll down to bottom of page.
4. Under Existing Certificate Details, you can see the current root CA information from the original deployment of this RSA Authentication Manager server:

Image description

5. You can import the certificate of the new primary server by clicking Choose File to import the original root CA. The UI states "Use for migrating the agents to the new deployment." This is vaguely documented on page 463 of the RSA Authentication Manager 8.1 SP1 Administrator's Guide, under "Configure an IPv4/IPv6 Agent," where it says:

(Optional) In the Import Certificate of the New Primary Server field, click Browse to locate and import a new root certificate.

Note: You are required to import a new root certificate only if you are migrating agents to a new deployment. This feature supports migrations from RSA Authentication Manager 8.0 to future versions of Authentication Manager.

6. To obtain a root CA from the original RSA Authentication Manager 8.x server, use openssl with the -showcerts option from within SSH:

rsaadmin@am82p:~> openssl s_client -showcerts -connect <name_IPaddr>:7004
CONNECTED (00000003)
depth=1 /CN=RSA root CA for am82p.vcloud.local/serialNumber=65c1331972306315d86e29174fbf3dd1eab360e47790154d5ba69ce884dab90d
verify error:num19:self signed certificate in certificate chain
verify return:0

---

7. Running the openssl command shows the server certificate chain in red and the root CA certificate chain in blue:

Certificate chain
 0 s:/CN=am82p.vcloud.local/serialNumber=7ff6116758773d4bdf96d336fa8dfd338dbb72131fe27a55f8700774800d69d
   i:/CN=RSA root CA for am82p.vcloud.local/serialNumber=65c1331972306315d86e26174fbf3dd1eab360e47790154d5ba69ce884dab90d
-----BEGIN CERTIFICATE-----
MIIEijCCA3KgAwIBAgIQSb2hk3XXoLOGXMHyu2zGtzANBgkqhkiG9w0BAQsFADBj
MQswCQYDVQQGEwJVUzEYMBYGA1UEChMPRU1DIENvcnBvcmF0aW9uMSUwIwYDVQQL
ExxHbG9iYWwgU2VjdXJpdHkgT3JnYW5pemF0aW9uMRMwEQYDVQQDEwpFTUMgU1NM
IENBMB4XDTE3MDYyMTE5Mzc1M1oXDTIwMDYyMTE5Mzc1M1owgYUxCzAJBgNVBAYT
AlVTMRYwFAYDVQQIEw1NYXNzYWNodXNldHRzMRAwDgYDVQQHEwdCZWRmb3JkMRkw
FwYDVQQKExBSU0EgU2VjdXJpdHkgTExDMRQwEgYDVQQLEwtSU0EgU3VwcG9ydDEb
MBkGA1UEAxMSYW04MnAudmNsb3VkLmxvY2FsMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAm58FfKsyY2uLfduyxm4HxaPvfySb06WRH5iTt+2RIGBvVWKO
mEhDvYOwkDLkFlhRrbWqf6xlfkIqY/lrdtFav1m42GLR18mwVQQ78SsABGbH+Kl8
An7EEqOumuYYviqhTx4EecZejqixDixyl2966/csr+EvAwG/8IoTGIYnx2U8dGRj
BuEShYTTROqNj4HxkAjnuNn+SxG6b2UkWALBwpeCTD0aYNHzVHbpKNUiBTFMwTOP
9IZSPIsEw96ex9Ojc1AQn50BbwiAfJXJ9JvSmulHGdY6DLaMK/ReOcW1gva58vK5
agOFYU0r07nrj6Ku0xS3Dhx2Kp/DKMDQMr1ruQIDAQABo4IBFTCCAREwDgYDVR0P
AQH/BAQDAgO4MBEGCWCGSAGG+EIBAQQEAwIGwDAdBgNVHREEFjAUghJhbTgycC52
Y2xvdWQubG9jYWwwHwYDVR0jBBgwFoAU7setF5AcQb3XE7/jvnrQxKLuxwUwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGC1sGAQUFBwMCMB0GA1UdDgQWBBSTUwcIocMETsUh
PN+6AK5l8SWalDBuBgNVHR8EZzBlMDOgMaAvhi1odHRwOi8vZW50ZXJwcmlzZWNh
LmNvcnAuZW1jLmNvbS9FTUNTU0xDQS5jcmwwLq11oCqGKGh0dHA6Ly9wa2kuY29y
cC5lbWMuY29tL2NybC9FTUNTU0xDQS5jcmwwDQYJKoZIhvcNAQELBQADggEBADAL
wLyjYb7illE0q+knng7sZso4LRuD4RgxYF/f50IEg8MdZlTVvH1QWA3H9g/zQ2i5
2S107cRwx3paxgSXb9zfX1Krgojx6I4fFJPxE6aIt7MUI+oTbjaClnr/dA58XoET
/zzs2cZ0G1Ho40tQ78lJZpd6nPWuhCgzcqw4/d6EnshjYqoseS1GGNE9ZDjT1m6N
eUdyGAt9Q/55T3WQI0KmszaYgo/mgWEFyM9hbzHgCudSq9qT7lxwT1hHAYjfbWqU
g4ApcZnsud9rhxR+UahD8lYwEExo2yfqmDFLVa5hxFtqgFwnP5QF1r1QV8zEqBAO
vDhttxjT59IRStKJQTU=
-----END CERTIFICATE-----
 1 s:/RSA root CA for am82p.vcloud.local/serialNumber=7ff6116758773d4bdf96d336fa8dfd338dbb72131fe27a55f8700774800d69d
   i:/CN=RSA root CA for am82p.vcloud.local/serialNumber=7ff6116758773d4bdf96d336fa8dfd338dbb72131fe27a55f8700774800d69d
----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgIQDnpJf/sai2ikg8QrEDRcejANBgkqhkiG9w0BAQUFADA9
MQswCQYDVQQGEwJVUzEYMBYGA1UEChMPRU1DIENvcnBvcmF0aW9uMRQwEgYDVQQD
EwtFTUMgUm9vdCBDQTAeFw0xMTAzMDgwMjM1MThaFw0yNjAzMDgwMjM1MThaMD0x
CzAJBgNVBAYTAlVTMRgwFgYDVQQKEw9FTUMgQ29ycG9yYXRpb24xFDASBgNVBAMT
C0VNQyBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwEV0
QaykbhIOVKj1BunB8pXsISlXgiv10QSGSxG2Dnbwoli0WSgPpLqPD8bsQuwjReg0
ERGXTXpxDEpb4Kya+YcIr4KGMd+EIdLjogXnrKv1/EWa54UNNjNLU6tkwEnVQ79p
Sbx2weCxEi+VG755+Bbb5AJKDcgk4ss5hXjI8tOzAgHe+tReNQamMSOgCO+4bZJ1
RBalcYHmGxVz2TbK0qrKKC7Um4ALQfRQejB+TuvYMoTZHD8Wm/e3Hdq7wwTOmQUL
/hG4+J+k4fl8WUtf4M6CzmeYVnEpZ34wk4H/1bRmFI9jvEQlmu/uKmFZ8DPOvK8j
YJCPft/fWOLkCZO205IDAQABo2YwZDAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/
BAgwBgEB/wIBAzAdBgNVHQ4EFgQUjyKad6YrWTr8z+fAlE5VRpSg/zQwHwYDVR0j
BBgwFoAUjyKad6YrWTr8z+fAlE5VRpSg/zQwDQYJKoZIhvcNAQEFBQADggEBALaL
B5rAo9GLri9vvYMIkMwtI4SFYeftNr205YA4o49sbCVlgdmzUXWk48aevoUZRl6/
rEPFbTxaZUbmjOv+XO+bGFA3T57RS6rAFeGBai/UirrckJhGgusAVU5lFtO31Mgm
W3cPXqV+PXwwHKbgLRCeTJFK3Rw68TxBqazMjNp4WufdnPC379Fg/zeKrCLwgsa4
AVFHmeIadvijSQBpY0bFzsSZGF/205h+NiYJpWRdDXfeeQStdZWxPESbWoXPu/Qg
0dIifLaHr2Nugkg8eTcp+F2rl2YIjnQcEFqOUNhyI8kPzzsWinYel47tC9kDL7qR
s34MLubs2L1iMIk7fJ4=
-----END CERTIFICATE-----

8. Copy and paste the cert into a text editor such as Notepad and save as a .cer file.
9. Make sure the file includes the -----BEGIN CERTIFICATE----- as the first line at the top of the file and -----END CERTIFICATE----- at the bottom, as the last line, like this:
   1. Edit the sdconf.rec with a text editor, such as Notepad++.
   2. Scroll down past the encrypted characters to the bottom.
   3. Enable word wrap.
   4. Look for the delimiter with X509 in it, as shown:

xmlns:bootstrap="http://www.rsa.com/schemas/2008/05/CommonAPI/bootstrap"><bootstrap:ServiceKeys><bootstrap:X509Certificate>

5. In between the start <bootstrap:X509Certificate>and stop </bootstrap:X509Certificate> is the certificate. Copy all of the text to a buffer. For example,

MIIDnjCCAoagAwIBAgIQi8nZ9+145MSlQkXpRicAjANBgkqhkiG9w0BAQUFADBMT
IwMAYDVQQDDClSU0Egcm9vdCBDQSBmb3Igam11bHNtYW4tdm0xMTUubmEucnNhLm
5ldDFJMEcGA1UEBRNANjVjMTMzMTk3MjMwNjMxNWQ4NmUyNjE3NGZiZjNkZDFlYW
IzNjBlNDc3OTAxNTRkNWJhNjljZTg4NGRhYjkwZDAeFw0xNDA4MjcxNDUzMjRaFw
0zNzAxMDEwNTAwMDBaMH8xMjAwBgNVBAMMKVJTQSByb290IENBIGZvciBqbXVsc2
1hbi12bTExNS5uYS5yc2EubmV0MUkwRwYDVQQFE0A2NWMxMzMxOTcyMzA2MzE1ZD
g2ZTI2MTc0ZmJmM2RkMWVhYjM2MGU0Nzc5MDE1NGQ1YmE2OWNlODg0ZGFiOTBkMI
IBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0Sh9lh1nCH6eAAt4C+buA3
YCd+WPyPwSym8bi2Xt9la0llEMqd1XuQknQQRdOU8A9Mlr8T8y2Imj8NCkTgcXcs
eoFnRoPImmL6Ch4f175xc61569boZIXa7MoGj7tIHG6TQ54xfiGoTSF1OMTqUJ8K
h1ysFfioORFPcX2llsBdZoLrMTjSsvaGzNVPp684822W4I1Peiaa0o1ppimZICF3
fIa0jAQUlXUDI2mo0ULayPgl3czQgOIZ5z13hM3CV0buTnUqJTu5mEvqASIZripM
tZw13eNkUhC7b3185zCXHVlVd1fE3QMGyaMi6cJdKhu8JZwJRYGlbRNywIDAQABo
xYwFDASBgNVHRMBAf8ECDAGAQHAgEBMA0GCSqGSIb3DQEBBQUAA4IBAQCbcxIUdp
cUVgkeq4qEmagFaOavzDhpBbiAGxPv4F3dWOsHSyq4sykWRe4simwTSxEQnXO5vg
Ob9Fsed6Fb5qxIdCpYkoweURVeJ6aBhqzmWq0nSb7X2nT7Ft26tydVWRD+YYkPOJ
YLGrMxthtaU0zwjBtHs+IThfxLR1cLRFIpJCbHa1FtVSQetoxAYryZuXjh8FkktB
gW6BNR7lfwK6Bzcyjflya9OPkwOUoWNqC1YkbgsEYkLGmKaYEt2a698IrwrzfRR8
hsN346XFMVhFd+CIz2Fn37NyCE0BMF3bq7h3x8QPDcZochs0PUn2DRTeFHyWF7M7
LNVj3oJ3JOp

6. Paste the text to a text editor.
7. Add -----BEGIN CERTIFICATE----- to the top line of the file, ensuring that it is the only text on the line.
8. Add -----END CERTIFICATE----- at the bottom, as the last line, ensuring that is the only text on the line.
9. When done, the file should look as what is shown here:

-----BEGIN CERTIFICATE-----
MIIDnjCCAoagAwIBAgIQi8nZ9+145MSlQkXpRicAjANBgkqhkiG9w0BAQUFADBMT
IwMAYDVQQDDClSU0Egcm9vdCBDQSBmb3Igam11bHNtYW4tdm0xMTUubmEucnNhLm
5ldDFJMEcGA1UEBRNANjVjMTMzMTk3MjMwNjMxNWQ4NmUyNjE3NGZiZjNkZDFlYW
IzNjBlNDc3OTAxNTRkNWJhNjljZTg4NGRhYjkwZDAeFw0xNDA4MjcxNDUzMjRaFw
0zNzAxMDEwNTAwMDBaMH8xMjAwBgNVBAMMKVJTQSByb290IENBIGZvciBqbXVsc2
1hbi12bTExNS5uYS5yc2EubmV0MUkwRwYDVQQFE0A2NWMxMzMxOTcyMzA2MzE1ZD
g2ZTI2MTc0ZmJmM2RkMWVhYjM2MGU0Nzc5MDE1NGQ1YmE2OWNlODg0ZGFiOTBkMI
IBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0Sh9lh1nCH6eAAt4C+buA3
YCd+WPyPwSym8bi2Xt9la0llEMqd1XuQknQQRdOU8A9Mlr8T8y2Imj8NCkTgcXcs
eoFnRoPImmL6Ch4f175xc61569boZIXa7MoGj7tIHG6TQ54xfiGoTSF1OMTqUJ8K
h1ysFfioORFPcX2llsBdZoLrMTjSsvaGzNVPp684822W4I1Peiaa0o1ppimZICF3
fIa0jAQUlXUDI2mo0ULayPgl3czQgOIZ5z13hM3CV0buTnUqJTu5mEvqASIZripM
tZw13eNkUhC7b3185zCXHVlVd1fE3QMGyaMi6cJdKhu8JZwJRYGlbRNywIDAQABo
xYwFDASBgNVHRMBAf8ECDAGAQHAgEBMA0GCSqGSIb3DQEBBQUAA4IBAQCbcxIUdp
cUVgkeq4qEmagFaOavzDhpBbiAGxPv4F3dWOsHSyq4sykWRe4simwTSxEQnXO5vg
Ob9Fsed6Fb5qxIdCpYkoweURVeJ6aBhqzmWq0nSb7X2nT7Ft26tydVWRD+YYkPOJ
YLGrMxthtaU0zwjBtHs+IThfxLR1cLRFIpJCbHa1FtVSQetoxAYryZuXjh8FkktB
gW6BNR7lfwK6Bzcyjflya9OPkwOUoWNqC1YkbgsEYkLGmKaYEt2a698IrwrzfRR8
hsN346XFMVhFd+CIz2Fn37NyCE0BMF3bq7h3x8QPDcZochs0PUn2DRTeFHyWF7M7
LNVj3oJ3JOp
-----END CERTIFICATE-----

10. Save the file with an extension of .cer.

9. Open the file with Microsoft Crypto Shell Extensions to read this cert info or optionally, paste into a certificate tool web site.