The Replication Status Report found in the primary instance Operations Console is reporting a status of 'Internal Replication Error' for one or more replica instances.

The likely causes for an 'Internal Replication Error' being reported is as follows:

• Check the primary and replica instances are running the same version of Authentication Manager software before troubleshooting an 'Internal Replication Error'.

NOTE: Primary and replica instance must be running the same version of Authentication Manager software for replication to work.

• Communication failure between the Authentication Manager instances in the Authentication Manager deployment.
• Time differences between the primary and replica instances in the Authentication Manager deployment. 
• Primary and/or replica instances are unable to forward and reverse lookup the fully qualified hostnames and IP addresses of all primary and replica instances in the Authentication Manager deployment.
• There is a database violation with a duplicate user ID in the Authentication Manager deployment.
• There is low disk space on the primary and/or replica instances.

The Force_Sync_Link.sh shell script attached to this article will force the replica instance to have an Out of Sync status in the Replication Status Report found in the primary instance Operations Console. Once the Out of Sync status is seen an administrator can perform a replica instance synchronization.

Installation

1. Download and copy the Force_Sync_Link.sh shell script into the /tmp folder on the Authentication Manager instance.  You must log in to the RSA Community to download the Force_Sync_Link.sh file. Where SSH has been enabled  for Authentication Manager a secure FTP client (e.g. WinSCP) can be used to copy the shell script into the /tmp folder.
2. Change the permissions of the Force_Sync_Link.sh so it can be executed at the command line e.g. 

chmod 755 /tmp/Force_Sync_Link.sh

Usage

1. Logon to the Authentication Manager instance with the rsaadmin account, either in an SSH session or at the local console.

Note that during Quick Setup another user name may have been selected. Use that user name to login.

2. Change the privileges of the rsaadmin account:

sudo su -

  If you do not change the privileges of the rsaadmin account the following message appears:

You must be the root user to use this program; exiting...

3. Navigate to the /tmp folder:

cd /tmp

4. The shell script can be executed in one of two ways, as Operations Console user credentials are required.

• Option 1 is to run the command ./Force_Sync_Link.sh <Operations Console admin user name> <Operations Console admin password> in one step:

rsa01:/tmp # ./Force_Sync_Link.sh ocadmin password
Checking OC credentails..
OC credentials validated... continuing..

• Option 2 is a two step process:

rsa01:/tmp # ./Force_Sync_Link.sh
Checking OC credentails....missing OC credentials!

Please enter OC Administrator username: <enter Operations Console admin user name> 
Please enter OC Administrator password: <enter Operations Console admin user password> 

OC credentials validated... continuing..

RSA Customer Support Force Sync Link for Replica Instance

Local Hostname: rsa01.local.net

This Authentication Manager instance is a primary with hostname: rsa01.local.net  continuing..

Enter the replica hostname to Sync:

5. Enter the replica instance hostname that you wish to force to be Out of Sync. 

In the following examples the primary hostname is rsa01.local.net and the replica instance is rsa02.local.net. Note that when you enter the replica's FQDN, it is case-sensitive and has to match exactly to what is seen in the primary Operations Console's replica list. 

RSA Customer Support Force Sync Link for Replica Instance

Local Hostname: rsa01.local.net

This Authentication Manager instance is a primary with hostname: rsa01.local.net  continuing..

Enter the replica hostname to Sync: rsa02.local.net

Hostname: rsa02.local.net was found to be a REPLICA

Setting SYNC for rsa02.local.net

Update made where the REPLICA rsa02.local.net is in a SYNC state.

Done!

• Should the replica instance already be in an Out of Sync state, then the following is seen:

RSA Customer Support Force Sync Link for Replica Instance

Local Hostname: rsa01.local.net

This Authentication Manager instance is a primary with hostname: rsa01.local.net  continuing..

Enter the replica hostname to Sync: rsa02.local.net

Hostname: rsa02.local.net was found to be a REPLICA

Setting SYNC for rsa02.local.net

Found REPLICA rsa02.local.net was already in a SYNC state.. exiting!

• Should the hostname not exist in the Authentication Manager deployment, then the following is seen:

RSA Customer Support Force Sync Link for Replica Instance

Local Hostname: rsa01.local.net

This Authentication Manager instance is a primary with hostname: rsa01.local.net  continuing..

Enter the replica hostname to Sync: rsa02.loca.net

Hostname rsa02.loca.net was not found in the Authentication Manager database.. exiting!

IMPORTANT: check the primary and replica instances are running the same version of Authentication Manager software before troubleshooting an 'Internal Replication Error.'

1. Communication failure between the Authentication Manager instances in the Authentication Manager deployment.
   • Refer to an article called 'Testing TCP ports on RSA Authentication Manager 8.x instances' at URL https://community.rsa.com/s/article/Testing-TCP-ports-on-RSA-Authentication-Manager-8-x-instances
2. Time differences between the primary and replica instances in the Authentication Manager deployment. 

• The time between the primary and replica instances in an Authentication Manager deployment must not be different by more than 1-2 minutes. Refer to 'Accurate System Date and Time Settings' at URL https://community.rsa.com/s/article/Accurate-System-Date-and-Time-Settings-f1aaf3a3 to check the date/time on the primary and replica instances.

3. Primary and/or replica instances are unable to forward and reverse lookup the fully qualified hostnames and IP addresses of all primary and replica instances in the Authentication Manager deployment.
   • Check DNS (and/or local) resolution of the hostnames and IP addresses of all primary and replica instances on each primary and replica instance in the Authentication Manager deployment.
   • The commcheck.sh program does provide DNS lookup information on the hostname and IP addresses for the primary and replica instances listed in the Authentication Manager. Alternatively, the Operations Console of an Authentication Manager instance has a feature called Network Tools that will allow an administrator to perform a Name Server Lookup (Operations Console > Administration > Network > Network Tools)
4. There is a database violation with a duplicate user ID in the Authentication Manager deployment.

• Check the /opt/rsa/am/server/logs/PrimaryReplication.log on the primary instance or /opt/rsa/am/server/logs/ReplicaReplication.log on replica instances for any reference to a duplicate key violation in the Authentication Manager database or duplicate User ID exceptions.

5.  There is low disk space on the primary and/or replica instances.

• Refer to this article on Examining the Disk Space Usage on Authentication Manager 8.x  to assist with checking what may have consumed disk space.