AFX Server fails to start in RSA Identity Governance & Lifecycle
Originally Published: 2019-11-13
Article Number
Applies To
RSA Version/Condition: 7.1.0, 7.1.1
Issue
Under AFX Server menu the AFX Server shows as State "Not running".
Under AFX Connectors menu the AFX Connectors show as Status "Stopped" and/or as Status "Not Deployed".
After starting AFX using the afx start command the AFX server shows the warning message "Timed out waiting for AFX applications to start".
acm-702:~ # afx start
Successful connection detected to RSA Identity Governance and Lifecycle
Starting AFX Server: Setting MULE_HOME to /home/oracle/AFX/esb...
Checking Aveksa OEM ESB License for AFX...
INFO: Using default configuration
(you can configure options in one of these file: /etc/default/activemq /root/.activemqrc)
INFO: Invoke the following command to create a configuration file
/home/oracle/AFX/activemq/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ]
INFO: Using java '/etc/alternatives/java_sdk_1.7.0/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/home/oracle/AFX/activemq/data/activemq.pid' (pid '32513')
Waiting for ActiveMQ to start....
ActiveMQ has started.
----
MULE_HOME is set to /home/oracle/AFX/esb
Starting Mule Enterprise Edition...
Waiting for AFX applications to start...
...{lines removed}
Waiting for AFX applications to start...
WARNING!! Timed out waiting for AFX applications to start. Please check AFX application log files for detailed status information.
After starting AFX with afx start the afx status command shows that "MMC Console is not running."
acm-702:~ # afx status AFX Server Status:Setting MULE_HOME to /home/oracle/AFX/esb... INFO: Using default configuration (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc) INFO: Invoke the following command to create a configuration file /home/oracle/AFX/activemq/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ] INFO: Using java '/etc/alternatives/java_sdk_1.7.0/bin/java' ActiveMQ is running (pid '32513') ---- MULE_HOME is set to /home/oracle/AFX/esb Mule Enterprise Edition is running (32717). ---- MMC Console is not running. WARNING!!! WARNING!!! WARNING!!! AFX is installed and being administered as privileged user (root). As a security best practice, it is STRONGLY recommended that you install and administer AFX as a less privileged user.
The aveksaServer.log file (/home/oracle/wildfly-8.2.0.Final/standalone/log/aveksaServer.log) shows the following ERROR level log message:
11/13/2019 13:27:02.393 ERROR (default task-32) [com.aveksa.afx.filters.AFXAuthFilter] Failed to authenticate AFX server due to certificate validation error. com.aveksa.server.certificates.CertificateValidationException: No certificate found for client with CN AFX_Server-363975792
The mule_ee.log file (/home/oracle/AFX/esb/logs/mule_ee.log) shows the following modules have failed to start with the these error messages.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: Exception: HTTP response error! Response code=401 ; Reason: RSA Identity Governance and Lifecycle server was unable to authorize initialization request. This usually indicates that the AFX SSL certificate and/or ID currently configured for this installation do not match with records in the RSA Identity Governance and Lifecycle database. You may encounter this problem in the following scenarios:
*****
1.) The AFX SSL certificate was regenerated using the RSA Identity Governance and Lifecycle application but the AFX installation was not updated with keystore containing the new certificate. In this case, please update the AFX installation with latest keystore available for download from RSA Identity Governance and Lifecycle application.
*****
2.) RSA Identity Governance and Lifecycle certificate store has been changed but neither the RSA Identity Governance and Lifecycle server nor AFX installations have been updated with respective keystore containing new certificate and CA entries. In this case, please update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
3.) RSA Identity Governance and Lifecycle database was refreshed / restored using a backup that was generated on another environment (e.g., backup of Production system database was restored on the QA system database). In this case, additional steps are required to get the SSL certificate configuration in the database in sync with what's deployed on the RSA Identity Governance and Lifecycle & AFX server machine(s). Please change the RSA Identity Governance and Lifecycle certificate store and then update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
1.) The AFX SSL certificate was regenerated using the RSA Identity Governance and Lifecycle application but the AFX installation was not updated with keystore containing the new certificate. In this case, please update the AFX installation with latest keystore available for download from RSA Identity Governance and Lifecycle application.
*****
2.) RSA Identity Governance and Lifecycle certificate store has been changed but neither the RSA Identity Governance and Lifecycle server nor AFX installations have been updated with respective keystore containing new certificate and CA entries. In this case, please update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
3.) RSA Identity Governance and Lifecycle database was refreshed / restored using a backup that was generated on another environment (e.g., backup of Production system database was restored on the QA system database). In this case, additional steps are required to get the SSL certificate configuration in the database in sync with what's deployed on the RSA Identity Governance and Lifecycle & AFX server machine(s). Please change the RSA Identity Governance and Lifecycle certificate store and then update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: IllegalArgumentException:
Could not resolve placeholder 'afx.server.activemq.password'
in string value "${afx.server.activemq.password}"
+ Failed to deploy artifact '15_AFX-MAIN', see below +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: IllegalArgumentException:
Could not resolve placeholder 'afx.server.activemq.password'
in string value "${afx.server.activemq.password}"
The esb.AFX-MAIN.log file (/home/oracle/AFX/esb/logs/esb.AFX-MAIN.log shows the following exception.
2019-11-13 13:35:31.288 [ERROR] org.mule.module.launcher.application.DefaultMuleApplication:361 - null java.lang.IllegalArgumentException: Could not resolve placeholder 'afx.server.activemq.password' in string value "${afx.server.activemq.password}"
The esb.AFX-INIT.log file (/home/oracle/AFX/esb/logs/esb.AFX-INIT.log) shows the following exception.
2019-11-13 13:35:29.559 [ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - Server initialization failed! Please correct the issue and restart AFX. java.lang.Exception: HTTP response error! Response code=401 ; Reason: RSA Identity Governance and Lifecycle server was unable to authorize initialization request. This usually indicates that the AFX SSL certificate and/or ID currently configured for this installation do not match with records in the RSA Identity Governance and Lifecycle database. You may encounter this problem in the following scenarios:
*****
1.) The AFX SSL certificate was regenerated using the RSA Identity Governance and Lifecycle application but the AFX installation was not updated with keystore containing the new certificate. In this case, please update the AFX installation with latest keystore available for download from RSA Identity Governance and Lifecycle application.
*****
2.) RSA Identity Governance and Lifecycle certificate store has been changed but neither the RSA Identity Governance and Lifecycle server nor AFX installations have been updated with respective keystore containing new certificate and CA entries. In this case, please update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
3.) RSA Identity Governance and Lifecycle database was refreshed / restored using a backup that was generated on another environment (e.g., backup of Production system database was restored on the QA system database). In this case, additional steps are required to get the SSL certificate configuration in the database in sync with what's deployed on the RSA Identity Governance and Lifecycle & AFX server machine(s). Please change the RSA Identity Governance and Lifecycle certificate store and then update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
*****
1.) The AFX SSL certificate was regenerated using the RSA Identity Governance and Lifecycle application but the AFX installation was not updated with keystore containing the new certificate. In this case, please update the AFX installation with latest keystore available for download from RSA Identity Governance and Lifecycle application.
*****
2.) RSA Identity Governance and Lifecycle certificate store has been changed but neither the RSA Identity Governance and Lifecycle server nor AFX installations have been updated with respective keystore containing new certificate and CA entries. In this case, please update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
3.) RSA Identity Governance and Lifecycle database was refreshed / restored using a backup that was generated on another environment (e.g., backup of Production system database was restored on the QA system database). In this case, additional steps are required to get the SSL certificate configuration in the database in sync with what's deployed on the RSA Identity Governance and Lifecycle & AFX server machine(s). Please change the RSA Identity Governance and Lifecycle certificate store and then update both the RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the RSA Identity Governance and Lifecycle application.
*****
Cause
This can occur in the following situations:
- After an upgrade of the AFX server.
- After restoring a database from another system.
- After restoring an AFX server archive from another system.
- After installing the AFX server on a soft-appliance.
- After clicking on System > Admin > Security > Change Certificate Store and not updating the client.keystore of the AFX server.
- In WebSphere or WebLogic the server might not be configured to use server.keystore for incoming AFX connections.
Resolution
Note: if the Server Certificate Store for Agent SSL Connections has just been changed, you must restart acm for the new certificate to take effect.
From the Settings tab under the AFX Server instance accessed under the AFX Servers menu make the following changes.
- Click the Edit button.
- Update the Default Truststore Password with the default value of "changeit". (This step may not be necessary but ensures that if the previously encrypted value was undecryptable a new value is saved.)
- Click OK to save your changes.
- Click the Change Certificate button to update the AFX instances in database with the current value of the https certificate used for this server.
- Click OK to save your changes.
- Choose one of the following two options. Either generate a new AFXServer.zip archive and deploy it, Or follow the instructions to download and copy over the keystore to the existing AFX instance.
- Generate a new AFXServer.zip archive. This archive will contain an updated copy of the client.keystore file.
- Click "Download Server Archive" button and save the AFXServer.zip file to a location where you can transfer it to your server instance.
- Deploy the AFXServer.zip file according to the instructions the following article
- How to install Access Fulfillment Express (AFX) for use with RSA Identity Governance & Lifecycle
- Copy the new client.keystore directly to your existing AFX instance
- Click "Download Keystore" and save the client.keystore file to a location where you can transfer it to your server instance.
- Copy the client.keystore file to the server and replace the existing file in /home/oracle/AFX/esb/conf/client.keystore
Notes
keytool -list -v -storepass Av3k5a15num83r0n3 -keystore /home/oracle/AFX/esb/conf/client.keystore -alias aveksa_ca
keytool -list -v -storepass Av3k5a15num83r0n3 -keystore /home/oracle/keystore/server.keystore -alias aveksa_ca
