000030556 - RSA Authentication Manager 8.1 sync process fails at "Downloading database dump to replica"

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 14, 2017
Version 6Show Document
  • View in full screen mode

Article Content

Article Number000030556
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.1 SP1, 8.2
IssueDuring the manual database sync between a primary and replica instance the synchronization fails at Downloading database dump to replica.
 
User-added image

Messages found in the RSA Authentication Manager troubleshooting log files (obtained from the Operations Console):  
321889 2015-06-15 15:36:03,753 ERROR: am81r.vcloud.local,,,,Database dump download failed on attempt 1
com.rsa.authmgr.internal.replication.TransportClientException: Hash verification failed for dump file resume
at com.rsa.authmgr.internal.replication.TransportClient.isDumpFileContentReady(TransportClient.java:767)
at com.rsa.authmgr.internal.replication.TransportClient.tryToDownloadDump(TransportClient.java:258)
at com.rsa.authmgr.internal.replication.TransportClient.getDatabaseDump(TransportClient.java:229)
at com.rsa.authmgr.internal.replication.TransportClient$getDatabaseDump.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at SetupReplica.prepareForSynchronize(SetupReplica.groovy:380)

 
403671 2015-06-15 15:37:25,535 ERROR: am81r.vcloud.local,,,,IOException caught.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at com.rsa.sslj.x.aG.d(Unknown Source)
at com.rsa.sslj.x.ap.a(Unknown Source)
at com.rsa.sslj.x.ap.b(Unknown Source)
at com.rsa.sslj.x.ap.b(Unknown Source)
at com.rsa.sslj.x.al.read(Unknown Source)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.commons.httpclient.ContentLengthInputStream.read(ContentLengthInputStream.java:170)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:107)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:126)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1465)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1441)
at com.rsa.authmgr.internal.replication.TransportClient.copy(TransportClient.java:806)
at com.rsa.authmgr.internal.replication.TransportClient.writeFileAndHash(TransportClient.java:789)
at com.rsa.authmgr.internal.replication.TransportClient.tryToDownloadDump(TransportClient.java:275)
at com.rsa.authmgr.internal.replication.TransportClient.getDatabaseDump(TransportClient.java:229)
at com.rsa.authmgr.internal.replication.TransportClient$getDatabaseDump.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at SetupReplica.prepareForSynchronize(SetupReplica.groovy:380)
at SetupReplica$prepareForSynchronize.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at SetupReplica.synchronize(SetupReplica.groovy:350)

 
498671 2015-06-15 15:39:00,535 ERROR: am81r.vcloud.local,,,,Failed to send Replica Synchronization Heartbeat. Retry time: 2
com.rsa.authmgr.internal.replication.TransportClientConnectionException: Unexpected IOException
at com.rsa.authmgr.internal.replication.TransportClientBase.translateHttpIOException(TransportClientBase.java:255)
at com.rsa.authmgr.internal.replication.TransportClientBase.executeMethodExpectAvailable(TransportClientBase.java:197)
at com.rsa.authmgr.internal.replication.TransportClientBase.executeMethod(TransportClientBase.java:217)
at com.rsa.authmgr.internal.replication.TransportClientBase.executeMethodExpectOK(TransportClientBase.java:164)
at com.rsa.authmgr.internal.replication.TransportClient.executeMethodWithHostname(TransportClient.java:379)
at com.rsa.authmgr.internal.replication.TransportClient.sendSynchronizationHeartbeat(TransportClient.java:725)
at com.rsa.authmgr.internal.replication.ReplicaSynchronizationHeartbeat.run(ReplicaSynchronizationHeartbeat.java:87)
Caused by: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
ResolutionThe release of RSA Authentication Manager 8.1 Service Pack 1 Patch 3 allows for additional parameters to manually configure the sync settings in an RSA Authentication Manager 8.1 instance. These updates are as follows:
ParameterUpdate Value
Primary Sync Retry6
Replica Sync Retry6
Primary Sync Chunk Size (in kilobytes; KB)1024
Replica Sync Heartbeat Interval (seconds)90

It is required that these parameters be set manually on the primary Authentication Manager and Any Replica Instances you need to sync
This works for manually syncing large Authentication Manager databases between a primary and replica instance, or where a large database is being used in either a planned promotion for maintenance or replica instance promotion for disaster recovery.
For information on promotions refer to Chapter 15, "System Maintenance and Disaster Recovery" of the RSA Authentication Manager 8.1 Administrator's Guide.


Steps for usage where RSA Authentication Manager 8.1 Service Pack 1 Patch 3 or later is in use:
  1. Login with the rsaadmin account to the Authentication Manager primary server with an SSH session or at the local console.  
  2. Navigate to /opt/rsa/am/utils.  
  3. Add the following new parameters to the primary instance with the following rsautil store commands.  When running these commands, replace <OC_admin> with a valid Operations Console administrator name and <OC_password> with the Operations Console administrator's password.
    1. Create a primary synchronization retry parameter and set the retry value to 6  
./rsautil store -o <OC_admin> -p <OC_password> -a add_config auth_manager.synchronization.primary_sync.retry 6 GLOBAL 501

  1. Create a replica synchronization retry parameter and set the retry value to 6  
./rsautil store -o <OC_admin> -p <OC_password> -a add_config auth_manager.synchronization.replica_sync.retry 6 GLOBAL 501

  1. Create a primary synchronization chunksize parameter and set the value to 1MB 
./rsautil store -o <OC_admin> -p <OC_password> -a add_config auth_manager.synchronization.primary_sync.chunksize.kilobytes 1024 GLOBAL 501

  1. Create a replica heartbeat interval and set the value to 90 seconds 
./rsautil store -o <OC_admin> -p <OC_password> -a add_config auth_manager.synchronization.replica_sync.heartbeat_interval.seconds 90 GLOBAL 501

  1. Should the parameters already exist in the RSA Authentication Manager 8.1 instance, change the action in the rsautil store command to use update_config instead of add_config.  For example, 
./rsautil store -o <OC_admin> -p <OC_password> -a update_config auth_manager.synchronization.replica_sync.heartbeat_interval.seconds 90 GLOBAL 501

  1. For the new parameters, or for updates made to existing parameters, to take effect, the RSA Authentication Manager services must be stopped and restarted using one of the sets of commands below:
/opt/rsa/am/server/rsaserv stop
/opt/rsa/am/server/rsaserv start

/opt/rsa/am/server/rsaserv restart

  1. After performing the steps above on the primary Authentication Manager server, repeat steps 1 - 4 on each replica one-by-one.
Notes

Manual database synchronization between a primary and its replica instance(s)


IMPORTANT NOTE: Of the two examples below, one will set just a single replica to require a sync and the second will set all replicas to require a sync. In large environments if you have a primary and five replicas but only one is out of sync, it's best to just set the single replica out of sync versus all of them. This saves time and prevents forcing syncs on replicas that are not out of sync.
  1. Login with the rsaadmin account to the Authentication Manager primary server with an SSH session or at the local console.  
  2. Navigate to /opt/rsa/am/utils.  
  3. Retrieve the password for the rsa_dba user using the following command:
  4. When prompted, enter the Operations Console administrator's name and password. It is recommended to create a read-only user for database access. 
  5. Note the value of the com.rsa.db.dba.password.
  6. Connect to the SQL database with the command ./psql -h localhost -p 7050 -d db -U rsa_dba.
  7. When prompted, enter the com.rsa.db.dba.password.
login as: rsaadmin
Using keyboard-interactive authentication.
Password:
Last login: Mon Jan  9 13:07:55 2017 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am81p:~> cd /opt/rsa/am/utils
rsaadmin@am81p:/opt/rsa/am/utils> ./rsautil manage-secrets -a get com.rsa.db.dba.password
Please enter OC Administrator username: <enter Operations Console administrator name>
Please enter OC Administrator password: <enter Operations Console administrator password>
com.rsa.db.dba.password: rSKD5bGguLGNL9uGvFWnJoxIcHJah2
@am81p:/opt/rsa/am/utils> cd ../pgsql/bin
rsaadmin@am81p:/opt/rsa/am/pgsql/bin> ./psql -h localhost -p 7050 -d db -U rsa_dba
Password for user rsa_dba: <enter com.rsa.db.dba.password from above>
psql.bin (9.2.4)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.
db=#

  1. At the db=# prompt, enter the following SQL statement to review the contents of the table:
db=# SELECT * FROM rsa_rep.IMS_INSTANCE_NODE;
                id                |           instance_id            |  name  |        host         |      ip
   | product_patch |     activation_time     |     last_updated_on
----------------------------------+----------------------------------+--------+---------------------+-----------
---+---------------+-------------------------+-------------------------
 43dbd0751e02a8c01aa389a608c8d329 | 1666addb1e02a8c008016d234bd2b1d7 | am81p  | am81p.vcloud.local  | 192.168.2.
30 |               | 2016-11-22 22:39:48.537 | 2017-01-24 21:48:09.929
 6b4a94581f02a8c01a7da293c61f4850 | 1055445d1f02a8c00801c4db3d79d286 | am81r1 | am81r1.vcloud.local | 192.168.2.
31 |               | 2016-10-18 17:49:19.185 | 2017-01-02 13:17:45.624
(2 rows)

  1. To set one replica as out of sync, capture the name value of the replica from the output above.  In this example, it is am81r1.  Run the following command:
db=# UPDATE RSA_REP.IMS_INSTANCE SET deployed_state = 'out_of_sync' WHERE name='am81r1';
UPDATE 1

  1. To set all of the replicas out of sync, use the following command:
    db=# UPDATE RSA_REP.IMS_INSTANCE SET deployed_state = 'out_of_sync' where is_primary='FALSE';
    UPDATE 1

    1. To exit the database, type \q:
    db=# \q
    rsaadmin@am81p:/opt/rsa/am/utils>

    1. Now in the RSA Operations Console of the primary, the Sync link is available for use:
    User-added image

    1. Click Sync to perform a database sync between the primary and replica instance.

    Attachments

      Outcomes