RSA Authentication Manager 8.1 primary instance fails to upgrade to 8.2 with error: Replication flush failed

[#1]

Article Number

000034092

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.1.1

Issue

The RSA Authentication Manager 8.1 primary instance fails to upgrade to 8.2 with error Replication flush failed, that appears in the  Advanced Monitor in the Operations C onsole and also in the /opt/rsa/am/server/logs/update-8.2.0.0.0-<build#>-<timestamp>.log, as shown here:

246589 2016-09-26 20:15:05,363 INFO: Processing all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply...(18) 
256591 2016-09-26 20:15:15,365 INFO: Processing all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply...(19) 
266596 2016-09-26 20:15:25,370 FATAL: Replication flush failed. Unable to process all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply. 
266641 2016-09-26 20:15:25,415 FATAL: 
Replication flush failed. Unable to process all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply. 
java.lang.AssertionError: Replication flush failed. Unable to process all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply. 
at com.rsa.plugins.install.CommandLineInstallEngine.fail(CommandLineInstallEngine.groovy:282) 
at com.rsa.plugins.install.CommandLineInstallEngine$fail.call(Unknown Source) 
at com.rsa.plugins.install.GroovyInstallEngine.fail(GroovyInstallEngine.groovy:31) 
at com.rsa.plugins.install.GroovyInstallEngine$_buildScriptEnvironment_closure5.doCall(GroovyInstallEngine.groovy:159) 
at PatchUtils.flushReplication(PatchUtils.groovy:953) 
at PatchUtils$flushReplication.call(Unknown Source) 
at UpdateRollback.preUpdate_ServiceControl_Stop_All_Services(UpdateRollback.groovy:247) 
at UpdateRollback$_preUpdate_closure1.doCall(UpdateRollback.groovy:22) 
at UpdateRollback$_preUpdate_closure1.doCall(UpdateRollback.groovy) 
at Utils.withArtifacts(Utils.groovy:48) 
at Utils$withArtifacts.call(Unknown Source) 
at UpdateRollback.preUpdate(UpdateRollback.groovy:11) 
at com.rsa.plugins.install.GroovyInstallEngine.invokeScript(GroovyInstallEngine.groovy:68) 
at com.rsa.plugins.install.GroovyInstallEngine$_runTask_closure2.doCall(GroovyInstallEngine.groovy:57) 
at com.rsa.plugins.install.GroovyInstallEngine.runTask(GroovyInstallEngine.groovy:56) 
at com.rsa.plugins.install.GroovyInstallEngine$_runTasks_closure3.doCall(GroovyInstallEngine.groovy:106) 
at com.rsa.plugins.install.GroovyInstallEngine.runTasks(GroovyInstallEngine.groovy:105) 
at com.rsa.plugins.install.GroovyInstallEngine$runTasks.call(Unknown Source) 
at com.rsa.plugins.install.CommandLineInstallEngine.main(CommandLineInstallEngine.groovy:40) 
Exception in thread "Main Thread" java.lang.AssertionError: 
Replication flush failed. Unable to process all pending changesets from: /opt/rsa/am/replication/r2p_chgsets_to_apply. 
at com.rsa.plugins.install.CommandLineInstallEngine.fail(CommandLineInstallEngine.groovy:287) 
at com.rsa.plugins.install.CommandLineInstallEngine$fail$0.call(Unknown Source) 
at com.rsa.plugins.install.GroovyInstallEngine.fail(GroovyInstallEngine.groovy:35) 
at com.rsa.plugins.install.GroovyInstallEngine$fail.callCurrent(Unknown Source) 
at com.rsa.plugins.install.GroovyInstallEngine.invokeScript(GroovyInstallEngine.groovy:88) 
at com.rsa.plugins.install.GroovyInstallEngine$_runTask_closure2.doCall(GroovyInstallEngine.groovy:57) 
at com.rsa.plugins.install.GroovyInstallEngine.runTask(GroovyInstallEngine.groovy:56) 
at com.rsa.plugins.install.GroovyInstallEngine$_runTasks_closure3.doCall(GroovyInstallEngine.groovy:106) 
at com.rsa.plugins.install.GroovyInstallEngine.runTasks(GroovyInstallEngine.groovy:105) 
at com.rsa.plugins.install.GroovyInstallEngine$runTasks.call(Unknown Source) 
at com.rsa.plugins.install.CommandLineInstallEngine.main(CommandLineInstallEngine.groovy:40) 
Configuration step UpdateRollback:preUpdate [FAILED] [ERROR] Error: Failed to invoke update engine: Failed to prepare the update. 
java.lang.Exception: Failed to prepare the update. 
at com.rsa.am.updateengine.tasks.BaseUpdateEngineTask.doExecute(BaseUpdateEngineTask.java:56) 
at com.rsa.am.updateengine.tasks.Task.execute(Task.java:136) 
at com.rsa.am.updateengine.components.impl.TaskManagerImpl.doExecuteTask(TaskManagerImpl.java:86) 
at com.rsa.am.updateengine.components.impl.TaskManagerImpl.execute(TaskManagerImpl.java:41) 
at com.rsa.am.updateengine.components.impl.UpdateInstallerImpl.run(UpdateInstallerImpl.java:30) 
[INFO] Invoking config engine goal UpdateRollback:postUpdate 
1 2016-09-26 20:15:26,808 INFO: Arguments: [UpdateRollback.postUpdate]

Cause

• The working primary should show, as in the example below, in /opt/rsa/am/replication/r2p_chgsets_to_apply/.  Note the changeset value then look for that specific value, as shown.

rsaadmin@am81p:~>ll /opt/rsa/am/replication/r2p_chgsets_to_apply/
total 4
drwx------ 3 rsaadmin rsaadmin 4096 Aug 20 09:24 5cb5f3fdd41f3e0a080173ac11da7bb9
rsaadmin@am81p:~>ll /opt/rsa/am/replication/r2p_chgsets_to_apply/5cb5f3fdd41f3e0a080173ac11da7bb9​
total 4

• The working replica should show the changesets referencing the same value in /opt/rsa/am/replication/process/:

rsaadmin@am81r:~>ll /opt/rsa/am/replication/progress
total 4
-rw-r--r-- 1 rsaadmin rsaadmin 10 Jul 8 12:16 5cb5f3fdd41f3e0a080173ac11da7bb9_progress.txt
rsaadmin@am81r:~>cat /opt/rsa/am/replication/progress/5cb5f3fdd41f3e0a080173ac11da7bb9​_progress.txt
DONE,falsersaadmin@am81r:~>

• The failing primary instance shows a pending item and its error in /opt/rsa/am/replication/r2p_chgsets_to_apply/<chgsets>/error/:

rsaadmin@am81p:/opt/rsa/am/replication>cd r2p_chgsets_to_apply
rsaadmin@am81p:/opt/rsa/am/replication>cd r2p_chgsets_to_apply>ls
e1338af513c0100a0801c134008285ca efa34d990a80000a0801733469fa9576
e2ef65c012c0100a0801740805724d35
rsaadmin@am81p:/opt/rsa/am/replication/r2p_chgsets_to_apply> cd efa34d990a80000a0801733469fa9576
rsaadmin@am81p:/opt/rsa/am/replication/r2p_chgsets_to_apply/efa34d990a80000a0801733469fa9576>ls
error r2p_chgset__838176.sql.gz r2p_chgset__838176.sql.gz.sha256

• The cause is unknown as of the writing of this article.

Workaround

As a workaround,

1. Delete the problematic replica instance and apply RSA Authentication Manager 8.2 to the primary instance.
2. In the primary's Operations Console select Deployment Configuration > Instances > Status Report.
3. From the replica context menu, select Delete.  This will enable to apply the RSA Authentication Manager 8.2 upgrade successfully.  Note:  Stopping all the replica services does not correct the issue.  It is required to explicitly delete the replica to clear the relevant tables in the database.
4. Redeploy the problem instance.