000027600 - How to rebuild an Appliance V3 after replication failure on a pre-SP2 installation

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000027600
Applies ToSecurID Appliance 3.0
IssueHow to rebuild an Appliance V3 after replication failure on a pre-SP2 installation
Cleanup Primary after replica installation failure.
Failed appliance 3.0 install
Failed runtime replication scenario

To perform the following, SSH connectivity must be enabled on the Primary.

In the Operations Console, go to Administration->Networking->Configure Connectivity using SSH

Check the box entitled "Enable SSH".

 

If the Operations Console is not available, a keyboard and monitor must be attached to the appliance.

Log in as 'emcsrv' using the OS password configured during Quick Setup

sudo su - root

Enter the password when prompted

       ***NOTE***: In the following command ?service? is not a directory, it is the command

 /sbin/service sshd start        // you should see: Starting sshd  [  OK  ]


To perform this procedure services on the Primary will be need to be stopped.

Resolution

IMPORTANT NOTE: A non-SP2 appliance cannot be installed after 12/31/2009, the installation process will fail. If you will be trying to add (or re-add) a Replica after this date, the Primary (and any working Replicas) must be upgraded to the SP2 version of the Appliance software (3.0.2.x) , and the new Replica must have a SP2 factory image, before trying to add it.

Preparation for the cleanup and rebuilding:

There are certain patches that will be required later in this solution. These downloads are large; it is helpful to download and check them first, so they will be available when required.  The database Patch 3.0.0.2  (~3GB) is a prerequisite for all later patches including 3.0.0.9 or SP2  (~2.4GB).  

Download the 3.0.0.2 database patch, the readme is VERY important.  It is recommended to use the Download Central website, please see A47708 for details.   It can also be download from the FTP site, but this is a large download (~3GB), and the FTP site is slower than Download Central. If you still prefer to use FTP: 

ftp.rsasecurity.com/support/Patches/Appliance_3.0/am-cap-3.0.0.2-x86-dvd1.iso

 
 
 
 

NOTE: Patches 3.0.0.9  or SP2 (~2.4GB) iare NOT cumulative, 3.0.0.2 is required first. At this time, it is only available on the FTP Server, not on the Download Central site. Be sure to install patch 3.0.0.2 before installing patch 3.0.0.9 or SP2.  RSA usually recommends using SP2 instead of 3.0.0.9 , but SP2 cannot be installed when Replication is in a failed state, so installing 3.0.0.9 is used by parts of this procedure.

ftp://ftp.rsasecurity.com/support/Patches/Appliance_3.0/am-cap-3.0.0.2-x86-dvd1.iso

 

ftp://ftp.rsasecurity.com/support/Patches/Appliance_3.0/am-cap-3.0.0.9-x86-dvd1.iso

ftp://ftp.rsasecurity.com/support/Patches/Appliance_3.0/Appliance-3.0.0.9-readme.txt 

 

In some cases,  a SHA1 or MD5 checksum will be published for the patch .iso files. If one is available, it is recommended to check the SHA1 or MD5 checksum of the download against published values, to verify they were not corrupted during transfer. Please see   A47810 to verify the SHA1SUM or A47982 to verify the MD5 checksums of the files.

Using the .iso files to burn DVDs for 3.0.0.2 or 3.0.0.9 is not recommended, using them to burn a DVD for SP2 is not supported. You can put the files on a UNIX NFS Server, as preparation for the cleanup, or use a USB memory stick. You can also set up the Appliance as it's own NFS Server, this is often easier and more reliable, as some burned DVDs have had problems. Please see  A46950 for details on this, and A47810 to re-verify the SHA1SUM or A47982 to re-verify the MD5 checksum, after copying the .iso to the appliance.


1a. (Radius was set up on the Replica)

- Log in to the Primary's Security Console

- Navigate to RADIUS > RADIUS server 

- Select the stale replica RADIUS server and delete it


1b. Apply the SP2 Factory Reset Patch to the failed appliance through the Operations Console, and then Factory Reset it, to make this an unconfigured SP2 appliance. If this fails, factory-default the failed Replica Appliance, see the owners guide, or A46596 or A44382  . Note: This will bring the Appliance to 3.0.0.0 which cannot be quick-setup after 12/31/2009. You must use a false date before this date (such as 12/01/2009) to be able to do a quick-setup as a standalone Primary. This is required to be able to apply the Factory Reset Patch for SP2, and then factory reset again to make it a unconfigured SP2 appliance.  

2. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup).

3. Download the backup off of the Primary using this procedure: a45460 .

4. If the primary is not already at 3.0.0.2 or higher, Apply and Install the 3.0.0.2 patch you downloaded onto the Primary, be sure to use the readme for your version of the database patch, as there have been some changes.  You can set up the Appliance as it's own NFS Server, please see  A46950 for details on this, and A47810 to verify the SHA1SUM .

5. Apply and Install the 3.0.0.2 patch on all working Replicas, one at a time. Wait 1 hour after finishing installing the 3.0.0.2 patch on the each working Replica, to be sure all Replication updates have finished.

6. Apply and Install the 3.0.0.9  patch you downloaded onto the Primary. NOTE: Patch 3.0.0.9 is NOT cumulative. Be sure to install patch 3.0.0.2 before installing patch 3.0.0.9 . You must have 3.0.0.9 to make sure you have the proper cleanup tools, do not install SP2 on your Primary yet .

7. Apply and Install the 3.0.0.9 patch on all working replicas, do not install SP2 on the working Replicas yet.  

 

8. Open an SSH session to the Primary. Login with the username: emcsrv , and the Operating System Password that was selected during the Appliance setup.

9. Change to the rsaadmin user:

sudo su rsaadmin

10. On the Primary, navigate to /usr/local/RSASecurity/RSAAuthenticationManager/utils and run the following command:

. ./rsaenv   (notice it is dot-space-dot-slash before rsaenv)

./rsautil setup-replication -a list

You will be prompted for the superadmin password to run all rsautil commands. Upon supplying the correct password the command will run.

11. If the failed Replica Installation is in the list, run the following command:

./rsautil setup-replication -a remove-replica -n <name of replica to be removed>

12. On the Primary, run the following command:

./rsautil setup-replication -a remove-unreg-replicas

13. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup).

14. Download the backup off of the Primary (see step 3 for details)

15. On the Primary, change directory to /usr/local/RSASecurity/RSAAuthenticationManager/utils

***NOTE: Steps 16 applies ONLY if there are no working replicas. If you have any working replicas skip step 16.***

16. Issue the following commands:

./rsautil setup-replication -a remove-primary

./rsautil setup-replication -a set-primary

Confirm and answer Y to all questions

***NOTE: Steps 16 applies ONLY if there are no working replicas. If you have any working replicas skip step 16.***

17. On the Primary, change directory to /usr/local/RSASecurity/RSAAuthenticationManager/utils

18. Issue the following command:

./rsautil manage-rep-error -a run-script -o cleanup_propagation.sql

19. Change directory to /usr/local/RSASecurity/RSAAuthenticationManager/server

20. Restart Authentication Manager services TWICE with the following command:

./rsaam restart all
 wait for restart to finish
 
./rsaam restart all

21. Once services have restarted, change directory to /usr/local/RSASecurity/RSAAuthenticationManager/db/admin/<instancename>/bdump

22. If any .trc files were created from the time of the last restart of services are larger than 3MB, then restart services again using the following command:

./rsaam restart all

 23. On the Primary, log into the Security Console and click the Setup->Instances menu. Verify that replication status is "Running"

24. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup).

25. Download the backup off of the Primary

The failed Replica will now be an unconfigured appliance with a SP2 factory image. The Primary and all working Replicas are now at 3.0.0.9 and are ready to be updated to SP2, this will be required before starting the process to add the unconfigured SP2 appliance as a new Replica.

Note:  the following steps will no longer be valid after 12/31/2009, but they are currently being retained for customer reference. After this date, the Quick-Setup process to add a new replica has certain requirements: 
A.  The Primary and all working Replicas must be patched to SP2, before creating a new Replica Package
B. The unconfigured appliance to be added as a new Replica, must have a SP2 factory image  

26. Generate a new replica package and install the replica as per the Owners Guide

27. Install the 3.0.0.2 patch you downloaded onto the reinstalled Replica, be sure to follow the instructions with the patch.

28. Install the 3.0.0.9 patch you downloaded onto the reinstalled Replica. NOTE: Patch 3.0.0.9 is NOT cumulative. Be sure to install patch 3.0.0.2 before installing patch 3.0.0.9

 

 

 

29. On the Primary, run a backup using the backup utility in the Operations Console (Maintenance->Backups->Create Backup).

30. Download the backup off of the Primary (step 3)

NotesIf ALL appliances are at least the SP2 level (including the SP2 factory image) see A51660
Legacy Article IDa45614

Attachments

    Outcomes