An upgrade to RSA Authentication Manager 8.3 on Intel hardware failed and services fail to start

[#1]

Article Number

000036105

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2 SP1, 8.3
Platform: Intel hardware appliance
Platform (Other): Intel only.  VMware, Hyper-V, Amazon Web Services (AWS) and Dell servers are not affected.

Issue

When upgrading from Authentication Manager 8.2 SP1 of any patch level to Authentication Manager 8.3, the upgrade said it was successful, but it actually failed and may not have rolled back.  The reboot that is part of a successful upgrade did not kick off automatically. If you try either to reboot or to restart services manually, the services will not start.

The error shown here is from the /opt/rsa/am/server/logs/update-8.3.0.0.0-build<version>-<timestamp>.log:

[ERROR] Error: Failed to invoke update engine: Failed to apply the update
2018-02-16 12:02:06,481 INFO: === Installing the weblogic CPU Done ===========
174332 2018-02-16 12:02:06,482 INFO: Performing update task InvokeGroovy 
(upgrade dell and intell snmp software )
174333 2018-02-16 12:02:06,483 INFO: Get the platform type::hardware

[exec] rsaadmin's password:
[exec] rsaadmin's password:sudo: /etc/init.d/snmpsa: command not found

UpdateRollback.update_InvokeGroovy_upgrade_dell_and_intell_snmp_software_(UpdateRollback.groovy:458)
at UpdateRollback$_update_closure2.doCall(UpdateRollback.groovy:46)

Cause

The Authentication Manager 8.2 SP1 Intel appliance did not have the snmpsa executable needed to start the Intel Hardware snmpsa service.  The Authentication Manager 8.3 upgrade looks for this executable during the upgrade of the Intel hardware.  As of this writing, there is no root cause as to why the Intel snmpsa executable was missing on the Intel hardware.

This issue is being tracked in AM-31801 (Upgrade to 8.3 failed on snmp, after rollback, services not starting in Intel).

Verifying if the hot fix is needed

To verify if this hotfix is needed, login to the Authentication Manager primary

1. Launch an SSH client, such as PuTTy.
2. Login to the primary Authentication Manager server as rsaadmin and enter the operating system password.

Note that during Quick Setup another user name may have been selected. Use that user name to login.

3. Change to the root user:

login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter operating system password>
Last login: Mon Mar 26 16:43:13 2018 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am82p:~> sudo su -
rsaadmin's password: <enter operating system password>
am82p:~ #

2. Run the following commands. 
   1. First, run dmidecode to confirm the vendor:  

dmidecode | grep Vendor

2. If the vendor returned above is Intel, next check to see if the file called /etc/init.d/snamsa is found:

ls -l /etc/init.d/snmpsa

If /etc/init.d/snmpsa is not found AND this is an Intel hardware appliance, then you need to apply the AM-31801 hot fix.

Resolution

If you have already attempted and failed the Authentication Manager 8.3 update and the 8.3 upgrade did not roll back to Authentication Manager 8.2 SP1, then you will have to factory reset your Intel appliance to Authentication Manager 8.2 SP1 then either restore an Authentication Manager 8.2 SP1 backup to a primary or attach this Intel appliance as a replica.  See article 000034277 - How to factory reset an RSA Authentication Manager 8.2 hardware appliance without a factory reset button from the Operations Console for more information.

To prevent this Authentication Manager 8.3 upgrade failure from happening  - or from happening again - you must update your Authentication Manager 8.2 SP1 Intel appliances with the snmpsa file included in the hot fix AM-31801-HF.zip.

Installation Steps

1. Download the AM-31801-HF.zip, either from this article or download it from the RSA SFTP server.  Simply, click the link to the AM-31801-HF.zip to start the download.  Your site will need access to SFTP for the download to complete.
2. Login to the Intel appliance as the rsaadmin user.

Note that during Quick Setup another user name may have been selected. Use that user name to login.

3. Create a directory to store the hot fix; for example, /home/rsaadmin/intel_hotfix:

mkdir /home/rsaadmin/intel_hotfix

4. Using a file transfer app such as WinSCP or FileZilla, transfer the zip file to the directory created above.
5. Extract the hot fix contents into the directory created above.

cd ​/home/rsaadmin/intel_hotfix
unzip AM-31801-HF.zip

6. Change to the root user:

sudo su -

7. Execute the install script:

./intel_snmp_hotfix.sh

8. When hotfix is done you should see  the following message:

Successfully applied the hotfix.

Note that it is not necessary to restart or reboot the server after applying the fix.

9. Navigate back to /home/rsaadmin and remove the intel_hotfix directory.

cd ​/home/rsaadmin
rm -r intel_hotfix

Notes

This issue affects Intel appliances only. 

VMware, Hyper-V, Amazon Web Services (AWS) and Dell servers are not affected.