000016220 - How to clean up an RSA Authentication Manager 7.1 primary and reattach a replica after a replication failure

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000016220
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 7.1 SP2 or later
Platform: Windows 2003 Server, Windows 2008 Server R2
IssueTHis article provides steps on how to:
  • Clean up an RSA Authentication Manager primary and reattach a replica after a replication failure on a Windows server.
  • Clean up an RSA Authentication Manager primary.
  • Detach and reattach a replica.
  • Handle replication failure.
  • Recover from the following errors:
    • The Oracle alert_<server_name>.log shows the message:
Found the stuck propagation process to <SID>
remote_apply_error => ORA-26714: User error encountered while applying
ORA-01280: Fatal LogMiner Error. The capture process on local site is ABORTED

Archive log deleted from Primary and Change not applied to Replica - must clean-up and re-attach

  • The info_replica.html shows the message:
remote_log_apply_time => <not current date>
Archived Log Status is not current date, and Deleted = NO
<date>     ../backup/<SID>/archivelog/<date>/<name>_.arc
e.g. \Program Files\RSA Security\RSA Authentication Manager\backup\IKAGWYZR\archivelog\2014_ 03_30\o1_mf_1_5712_9mhdq1pr_.arc     NO

  • The info_primary.html shows the message:
archive log .arc files not deleted since <not current date>
Local Apply TIMEOUT <not current date>

  • The alert_<replica_server_name>.log shows the message:
Errors in file ../db/admin/bijqumds/bdump/<anme>.trc:
ORA-07445: exception encountered: core dump [kghbshrt()+112] [SIGBUS] [Non-existent physical address] [0xB714050C] [] []
<not current date>
- IMS Trace - 0 rows deleted in rsa_logrep.ims_log_audit_adm for .01 seconds
Resolution

How to clean up an RSA Authentication Manager primary and reattach a replica after a replication failure on a Windows server


Before you implement this solution, you must delete any excess database files from the system's temp directory on the primary and all replicas.  To do this, open a command prompt on the servers and navigate to C:\Temp.  Run the following command:


del db*.sql

  • Depending on the build up of these .sql files, this command can take quite some time to run.  
  • There have been customers who had so many of these files on their servers that it took hours to complete.  
  • Be sure to run this command through command line rather than in Windows Explorer because if there is a large amount of files in the directory it will take a long time for the UI to paint.


Note: All RSA utilities (rsautil) commands, are run from the RSA_HOME\utils directory, where RSA_HOME is the RSA installation path. For example: C:\Program Files\RSA Security\RSA Authentication Manager\utils. Also, these utilities require the master password for your deployment, which you created when you installed the server.


To clean up the primary and reattach a replica


  1. On the primary, do the following:
    1. Create a backup using the RSA Operations Console (Maintenance > Backups > Create Backup).
    2. If RSA RADIUS is configured, delete the replica RADIUS server using the Operations Console (Deployment Configuration > RADIUS > Manage Existing).  If RSA RADIUS is functioning properly on the replica, and you plan to reattach this replica, skip this step.
IMPORTANT: If more than one replica will be deleted through the Operations Console,  don't choose multiple replicas, choose ONLY ONE AT A TIME.

  1. Delete the replica using the Operations Console (Deployment Configuration > Instances > Manage Existing).    If this hangs with --Status:Stopping propagation process at [<instance>], stop RSA services on this replica.  You may also need to reboot the primary.
    1. Open a command prompt, navigate to RSA_HOME\utils, and run the following command.  Enter the master password when prompted.
rsautil setup-replication -a list


  1. If the failed replica is in the list, run:
rsautil setup-replication -a remove-replica -n <name of replica to be removed>

  1. Run 
rsautil setup-replication -a remove-unreg-replicas

  1. If there are NO functioning replicas in your deployment, run the following commands, answering Y to all questions.  If you have one or more functioning replicas, skip this step.
rsautil setup-replication -a remove-primary  
rsautil manage-rep-error -a run-script -o cleanup_propagation.sql
rsautil setup-replication -a set-primary

  1. Run:
rsautil manage-rep-error -a run-script -o cleanup_propagation.sql

  1. On the primary Authentication Manager server,
    1. From the Windows services.msc, stop and restart the RSA Authentication Manager services twice. RSA recommends that after you stop the services the first time, reboot the primary to unlock files, and wait for all RSA services to start. Then stop and restart all RSA services again. Another reboot is not required, but it allows all RSA services to start in the regular sequence.
    2. Log on to the RSA Security Console and go to Setup > Instances. Verify that replication status is Running. 
    3. Update the RSA Authentication Manager Contact List using the Security Console (Access > Authentication Agents > Authentication Manager Contact List > Automatic Rebalance).
    4. Create a backup using Operations Console (Maintenance > Backups > Create Backup).
    5. Generate a new replica package using the Operations Console (Deployment Configuration > Instances > Generate Replica Package).

  1. On the replica Authentication Manager server,
    1. If you are building a new replica, use the replica package to build the new replica.
    2. For either a new replica that needs to be attached or an existing replica that needs to be reattached, log on to the Operations Console and attach the replica to the primary using the new replica package.  When logging on to the replica's Operations Console it should only give you the option to attach to the primary. If the Attach option is not available, open a command prompt on the replica, set the current directory to RSA_HOME\utils, and run:
rsautil manage-replication -a cleanup-offline-site

  1. After the command runs, log on to the Operations Console and attach the replica to the primary using the new replication package. If the Attach option is still not available, you must uninstall and reinstall the replica.
  1. On the primary do the following:
    1. Check replication status using the Operations Console (Deployment Configuration > Instances > Status Report). The Data Transfer Status should show "COMPLETE" both ways.
    2. Update the RSA Authentication Manager Contact List using the Security Console (Access > Authentication Agents > Authentication Manager Contact List > Automatic Rebalance).
    3. On the replica, reconfigure RSA RADIUS.
       
      • Log on the Operations Console, go to Deployment Configuration > RADIUS > Configure Server.  
      • Enter the required data to configure RSA RADIUS.
       
NotesRSA recommends upgrading all primary and replica instances in your deployment to the latest available Service Pack and Patch. Versions before Authentication Manager 7.1 SP4 are no longer supported.
It is not recommended to pause replication in Authentication Manager 7.1 SP4, as any communication problems between primary and any replica could make bigger problems
Legacy Article IDa51068

Attachments

    Outcomes