• Product: RSA Governance & Lifecycle
• Versions: 6.9.1, 7.x, 8.x
• Component: AFX Server, RSA Governance & Lifecycle Certificate Store, JDK Truststore (cacerts)
• Operating System: Linux

afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-01-06 12:30:28 EST; 11s ago
...
Jan 06 12:29:18 <your-afx-server-hostname> afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:28 <your-afx-server-hostname> afx_server[20643]: WARNING!! Timed out waiting for AFX applications to start.
Please check AFX application log files for detailed status information.

Use this article if the AFX Server fails to start and remains in a Not running state in the RSA Identity Governance & Lifecycle UI under AFX > Servers, and the log files show SSL certificate path validation errors during AFX initialization.

Observable symptoms:

• The AFX Server status shows Not running in the UI and does not recover after restart attempts.
• Running afx status as the afx user on the application server shows the startup timed out.

Run the following command as the afx user to confirm:

afx status

Expected output when this issue is present:

afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-01-06 12:30:28 EST; 11s ago
...
Jan 06 12:29:18 <your-afx-server-hostname> afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:28 <your-afx-server-hostname> afx_server[20643]: WARNING!! Timed out waiting for AFX applications to start.
Please check AFX application log files for detailed status information.

Log Evidence Versions 6.9.1 and 7.x

The following key errors appear in the AFX log files when starting AFX on versions 6.9.1 and 7.x.

NOTE: $AFX_HOME refers to the AFX Server installation directory on the Linux host (for example, /home/afx). Full stack traces are provided in the Appendix at the bottom of this article.

In $AFX_HOME/esb/logs/mule_ee.log:

Failed to deploy artifact '10_AFX-INIT'
CertPathBuilderException: Could not build a validated path.
Failed to deploy artifact '15_AFX-MAIN'
IllegalArgumentException: Could not resolve placeholder 'afx.server.activemq.password'
in string value "${afx.server.activemq.password}"

In $AFX_HOME/esb/logs/esb.AFX-INIT.log:

Unable to establish secure (SSL) connection with RSA Identity Governance and Lifecycle server.
SSL certificates for RSA Identity Governance and Lifecycle server and AFX were not issued
by the same RSA Identity Governance and Lifecycle Certificate Authority (CA).

In $AFX_HOME/esb/logs/esb.AFX-MAIN.log:

Could not resolve placeholder 'afx.server.activemq.password'
in string value "${afx.server.activemq.password}"

Log Evidence  Version 8.0.0

The following key errors appear in the AFX log files when starting AFX on version 8.0.0.

In $AFX_HOME/esb/logs/esb.AFX-INIT.log:

HTTP POST on resource 'https://<your-afx-server-hostname>:8444/aveksa/afx/heartbeat'
failed: Received fatal alert: certificate_unknown.
Error type: HTTP:CONNECTIVITY

In $AFX_HOME/esb/logs/esb.AFX-MAIN.log:

HTTP POST on resource 'https://<your-afx-server-hostname>:8444/aveksa/afx/primary'
failed: Received fatal alert: certificate_unknown.
Error type: HTTP:CONNECTIVITY

The default JDK truststore (cacerts) password is changeit. If the default password has not been changed, then that is the password that must be used for the Default Truststore Password under AFX > Servers > {AFX Server name} > Edit.

If the default JDK truststore password has been changed, then the modified password must be used in the Default Truststore Password definition for the AFX Server.

To resolve this issue, follow the steps below:

1. Determine what the JDK truststore password is for cacerts. The default truststore password for cacerts is changeit.
   1. Login as the afx user.
   2. Navigate to the appropriate directory with the following command:  

cd $JAVA_HOME/jre/lib/security

3. List the keystore contents with the following command:  

keytool -list -v -storepass changeit -keystore cacerts

keytool -list -v -storepass rowan -keystore cacerts
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.tools.keytool.Main.doCommands(Main.java:839)
        at sun.security.tools.keytool.Main.run(Main.java:368)
        at sun.security.tools.keytool.Main.main(Main.java:361)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
        ... 7 more

keytool -list -v -storepass {password} -keystore cacerts

2. Once you confirm the cacerts password, modify the password in the RSA Identity Governance & Lifecycle user interface. Do this even if the password has not been modified, i.e. if the password is changeit.
   1. In the user interface, go to AFX > Servers > {AFX Server name} > Edit. 
   2. In the Default Truststore Password field, type in the cacerts password.
   3. Press OK.
3. Restart the AFX service as the afx user.

$ afx start

Verification step

AFX is up and running