Prerequisite : You need to verify you have the proper Master Password.  From the (rsahome)/utils directory run the following command:

./rsautil manage-replication -a list

You will be prompted for the Master Password.  If the command fails, do NOT continue with this process until you have the proper Master Password.

This fix covers all scenarios; one replica broken vs. all replicas broken

1. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup . You can download the backup off the primary following the procedures in: a45460)
 

2. If Radius is configured, delete the Replica Radius server from the primary Operations Console (Deployment Configuration>Radius>Manage Existing)
Note: if you are running at least SP4, and the RADIUS Server is operating properly, and you will be re-adding the existing appliance, you can skip step 2  and 16

  2a. If there is at least one functional Replica, and  problem replica is not running at least SP4,  then apply the SP4 factory reset patch to the problem Replica and factory reset it. This will make it an unconfigured SP4 appliance.
 

3- Delete the Replica Server from the Primary Operations Console (Deployment Configuration>Instances>Manage Existing)
   IMPORTANT- If more than one Replica is to be removed from the Operations Console, don't chose multiple Replicas, choose ONE AT A TIME

If this hangs with --Status:Stopping propagation process at [<instance>] - Stop RSA Services on this replica.  May also need to reboot Primary
4.On the Primary run the following command:

 ./rsautil setup-replication -a list

You will be prompted for the Master password and/or SuperAdmin  Password to run all rsautil commands.

5. If the failed replica is in the list, run the following command:

./rsautil setup-replication -a remove-replica -n <name of replica to be removed>

6. On the Primary, run the following command:

 ./rsautil setup-replication -a remove-unreg-replicas

***NOTE: Steps 7 applies ONLY if there are no working replicas. If you have any working replicas skip step 7.*** 

 7. On the Primary, run the following commands:

 ./rsautil setup-replication -a remove-primary

 ./rsautil manage-rep-error -a run-script -o cleanup_propagation.sql

 ./rsautil setup-replication -a set-primary

Confirm and answer Y to all questions 

***NOTE: Steps 7 applies ONLY if there are no working replicas. If you have any working replicas skip step 7.*** 

8. On the Primary, Issue the following command:

 ./rsautil manage-rep-error -a run-script -o cleanup_propagation.sql

9. Restart the Authentication Manager database services TWICE including reboot

      Switch from the (rsahome)/utils directory to the /server directory:

      cd ../server 

      ./rsaam stop all

   ./rsaam start db   (this is the first db restart)

wait 2 minutes after the database start finishes

./rsaam stop all

exit   (leaving rsaadmin)

sudo su - (to get in to root mode, use same password as emcsrv)

reboot     (this will also do the second database restart, and free up locked files)

The reboot normally takes approximately 10 minutes to finish. If it has been more than 6 months since the last reboot with fsck (disk check), the system will do a Linux fsck , which  increases the reboot time to approximately 20 minutes

10. On the Primary, log into the Security Console and click the Setup->Instances menu. Verify that replication status is "Running"

11. Logon to the Security Console of the Primary and update the Authentication Manager Contact List: 

    Access/Authentication Agents/Authentication Manager Contact List/Automatic Rebalance, Rebalance

12. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup).

13. On the Primary, generate a new Replica Package/Dump file from the Operations Console (Deployment Configuration>Instances>Generate Replica Package. Always use the "Manual" Option)

14. Logon to the Replica's Operations Console and ATTACH to the primary using the new Replica Package/Dump files

NOTE: The Replica Operations Console should only give you the option to ATTACH to the Primary. In the event the ATTACH option is not available on the Replicas Operations Console, the command below must be run on the Replica Server to prepare it for Attachment to the Primary.

CD to " /usr/local/RSASecurity/RSAAuthenticationManager/utils " and run the following command:

 ./rsautil manage-replication -a cleanup-offline-site

Once the command finishes successfully, logon to the Replicas Operations Console and ATTACH to the Primary using the new Replication Package/dump files.

15. Once the replica attaches to the primary, logon to the Primary Operations Console and check the replication Status report (Deployment Configuration>Instances>Status Report)

  The Data Transfer Status Should show "COMPLETE" both ways.

16. Logon to the Security Console of the Primary and update the Authentication Manager Contact List:

    Access/Authentication Agents/Authentication Manager Contact List/Automatic Rebalance, Rebalance

17. If you use RADIUS in your environment, and removed the Replica RADIUS Server as part of Step 2, Reconfigure RADIUS on the Replica server. Open the Operations Console, navigate to Deployment Configuration>RADIUS>Configure Server and enter the required data to configure RADIUS.

It's highly recommended that all Appliance instances are upgraded to the latest Appliance 3.0 Service Packs/Patches. Follow the readme instructions of each Service Pack/Patch to apply the updates.