WFLYCTL0344 Operation timed out awaiting service container stability error when patching RSA Identity Governance & Lifecycle

[#1]

Article Number

000038024

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle
RSA Version/Condition: 7.1.x
Platform (Other): WildFly 10+

Issue

Patching RSA Identity Governance & Lifecycle fails with:
 

WFLYCTL0344 Operation timed out awaiting service container stability error

The full error can be found in the patch log (/home/oracle/Aveksa_{version_patch}/logs/{version_patch}_{date}-XXXX.log) and will look similar to the following: 
 

Wildfly is running

Repackaging the aveksa.war and aveksa.ear

Archive name is: aveksa_7.1.0_143825_P05-2019-Mar-25-21.06.ear

Creating aveksa.ear file for deployment
      [zip] Building zip: /tmp/customizeACM/aveksa.war
      [zip] Building zip: /tmp/customizeACM/aveksa.ear
     [copy] Copying 1 file to /home/oracle/archive

Undeploying current ear...

Deploying new customized ear...    this may take a while...
Exception encountered while deploying Aveksa.ear
WFLYCTL0344: Operation timed out awaiting service container stability

Aveksa.ear deployment failed after 15 minutes, 26.056 seconds

Updating CURRENTLY_DEPLOYED_ARCHIVE to aveksa_7.1.0_143825_P05-2019-Mar-25-21.06.ear

aveksa_7.1.0_143825_P05-2019-Mar-25-21.06.ear archived

An error occured in the customizeACM command : error code 5

Error deploying the ear file
Step failed! See /home/oracle/Aveksa_7.1.0_P05/logs/7.1.0_P05_2019-04-25-2058.log for more information.

 

Cause

During an RSA Identity Governance & Lifecycle patch installation, some migration of data is typically required for patching. Migration is run as a background process instead of from the user interface because a full migration is not needed. For this reason, the ear deployment may fail to deploy within the default timeout of 900 seconds (15 minutes) causing timeout issues.

This issue has been reported in engineering ticket ACM-96863.
 

Resolution

This issue is resolved in RSA Identity Governance & Lifecycle 7.2.
 

Workaround

Run the migration as a separate step.
 

On a standalone (non-clustered) system:

1. As the oracle user, stop RSA Identity Governance & Lifecycle. 

acm stop

2. As the oracle user, run the migration script.

cd /home/oracle/database
./migrate.sh

NOTE: Due to a known issue reported in RSA Knowledge Base Article 000038148 -- 'java.io.FileNotFoundException: aveksa-version.properties' error running migrate.sh in RSA Identity Governance & Lifecycle this step may fail with two errors. However, the majority of the migration will have been completed which is the goal of running the  migration step manually. In this case, the errors may be ignored and you may proceed to step 3. Subsequently running patch.sh again will finish the migration and in much less time thus avoiding the time out.

3. As the root user, run patch.sh again.

cd {patch-file-directory}
./patch.sh

This will redeploy the patched ear file that failed to deploy.

4. As the oracle user, start RSA Identity Governance & Lifecycle. 

acm restart

On a cluster:

1. As the oracle user, stop RSA Identity Governance & Lifecycle. 

SUSE 11: 
     service aveksa_cluster stop
SUSE 12:
     systemctl stop aveksa_cluster

2. As the avuser user, login to sqlplus and execute the following SQL statements:

update avuser.t_system_settings set value = '1' where parameter = 'Build';
commit;

This will force the migration into an interactive mode which means you will be prompted to migrate the database when accessing the user interface.

3. As the oracle user, rerun the deployment of the newly patched ear file.
4. When the ear deployment completes, it will detect the need for the schema migration and prompt you to migrate the database when you go to the RSA Identity Governance & Lifecycle user interface.
5. Enter AuthorizeMigration at the migration password prompt.
6. After the schema migration finishes, as the oracle user, restart RSA Identity Governance & Lifecycle.

SUSE 11: 
     service aveksa_cluster restart
SUSE 12:
     systemctl restart aveksa_cluster