000029249 - AFX Server remains in a Not running State, afx status shows 'timed out waiting for AFX applications to start' and esb.AFX-INIT.log has a 'Keystore was tampered, or password was incorrect' error in RSA Identity Governance & Lifecycle

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support on Jan 7, 2020
Version 12Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000029249
Applies ToRSA Product Set: RSA Identity Governance &  Lifecycle
RSA Version/Condition: 6.9.1, 7.0.x, 7.1.x
 
IssueThe AFX Server in RSA Identity Governance & Lifecycle is in a Not running State in the user interface (AFX > Servers).
User-added image


When logged into the application server as the afx user, the afx status command shows the startup timed out and the AFX Server never fully starts.


$ afx status
● afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sun 2020-01-05 09:24:06 EST; 1h 37min ago
  Process: 30415 ExecStop=/etc/init.d/afx_server stop (code=exited, status=0/SUCCESS)
  Process: 31129 ExecStart=/etc/init.d/afx_server start (code=exited, status=0/SUCCESS)
 Main PID: 31129 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)
   CGroup: /system.slice/afx_server.service

Jan 05 09:23:06 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:16 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:26 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:36 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:46 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:56 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:24:06 acm-711 afx_server[31129]: WARNING!! Timed out waiting for AFX applications to start. 
Please check AFX application log files for detailed status information.
Jan 05 09:24:06 acm-711 afx_server[31129]: done
Jan 05 09:24:06 acm-711 systemd[1]: Started Afx Server.


When starting AFX, the following errors are logged to the AFX log files:

$AFX_HOME/esb/logs/mule_ee.log:




++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: UnrecoverableKeyException: Password verification failed


$AFX_HOME/esb/logs/esb.AFX-INIT.log:



[ERROR] com.aveksa.afx.server.init.ConfigureDefaultSSLContextComponent:107 - Error configuing default SSL context
java.io.IOException: Keystore was tampered with, or password was incorrect

[ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - Server initialization failed! Please correct the issue and restart AFX.
java.io.IOException: Keystore was tampered with, or password was incorrect


 
CauseThis issue occurs because the truststore password that is in use for the JDK truststore (cacerts) does not match the Default Truststore Password defined for the AFX Server.
 
ResolutionThe default JDK truststore (cacerts) password is changeit. If the default password has not been changed, then that is the password that must be used for the Default Truststore Password under AFX > Servers > {AFX Server name} > Edit. If the default JDK truststore password has been changed, then the modified password must be used in the Default Truststore Password definition for the AFX Server.

To resolve this issue, follow the steps below.
  1. Determine what the JDK truststore password is for cacerts. The default truststore password for cacerts is changeit.
    1. Login as the afx user.
    2. Navigate to the appropriate directory with the following command:  


cd $JAVA_HOME/jre/lib/security


  1. List the keystore contents with the following command:  


keytool -list -v -storepass changeit -keystore cacerts


If the cacerts password is changeit, then this command will list the contents of the keystore. If the cacerts password is not changeit, the following error will occur:



keytool -list -v -storepass rowan -keystore cacerts
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.tools.keytool.Main.doCommands(Main.java:839)
        at sun.security.tools.keytool.Main.run(Main.java:368)
        at sun.security.tools.keytool.Main.main(Main.java:361)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
        ... 7 more


If the password is not changeit, someone at your site has changed the password and you will need to determine what that password is.

To confirm that you have the correct password, run the keytool -list command again with the correct password and verify that the truststore contents are listed.


keytool -list -v -storepass {password} -keystore cacerts



  1. Once you confirm the cacerts password, modify the password in the RSA Identity Governance & Lifecycle user interface. Do this even if the password has not been modified, i.e. if the password is changeit.

    1. In the user interface, go to AFX > Servers > {AFX Server name} > Edit. 
    2. In the Default Truststore Password field, type in the cacerts password.
    3. Press OK.
  2. Restart the AFX service as the afx user.


$ afx start


 
NotesIn 7.0.0 and 6.9.1 the AFX logs are located in:
 
$AFX_HOME/mule/logs
 

and the log files are called:

  • $AFX_HOME/mule/logs/mule.AFX-INIT.log
  • $AFX_HOME/mule/logs/mule.AFX-MAIN.log

Attachments

    Outcomes