000029259 - RSA Identity Governance and Lifecycle Access Fulfillment Express (AFX) Server fails to start with message:  WARNING!! Timed out waiting for AFX applications to start

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 10Show Document
  • View in full screen mode

Article Content

Article Number000029259
Applies ToRSA Product Set: RSA Identity Governance and Lifecycle (RSA G&L)
RSA Version/Condition: 7.0.1, 7.0.0, 6.9.1, 6.9
Platform: Linux
IssueThe AFX server fails to start with the error below:
$ service afx_server start
[... output trimmed ...]
Waiting for AFX applications to start...
Waiting for AFX applications to start...
WARNING!! Timed out waiting for AFX applications to start. Please check AFX application log files for detailed status information.
done

$

The AFX log files located in $AFX_HOME/esb/logs contain the following errors:
  • In the mule_ee.log:
Root Exception stack trace:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:197)
      at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:255)
      at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:319)
    + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)


  • In the mule.AFX-MAIN.log:
Invalid bean definition with name 'jmsConnector' defined in null: Could not resolve placeholder 'afx.server.activemq.password'

  • Running afx status shows ActiveMQ is running, Mule Enterprise Edition is running and MMC Console is NOT running:

$ su oracle
$ afx status
or
$ cd $AFX_HOME/bin
$ ./afx status
or
$ su root
# service afx_server status
 …
MMC Console is not running

CauseThe SunCertPathBuilderException error indicates that the AFX server’s client.keystore needs to be updated and re-deployed to re-establish communication with RSA G&L.
This can occur in the following situations:
  • After an upgrade of the AFX server.
  • After restoring a database from another system.
  • After restoring an AFX server archive from another system.
  • After installing the AFX server on a soft-appliance.
  • After clicking on System > Admin > Security > Change Certificate Store and not updating the client.keystore of the AFX server.
  • In WebSphere or WebLogic the server might not be configured to use server.keystore for incoming AFX connections.
Resolution
The process to resolve this error is to generate a new root (server) certificate and a new client certificate for each AFX server and remote agent, redeploy all certificates,and restart the RSA G&L application, AFX application, and remote agents. Here is how:

  1. Update the root certificate and Certificate Authority (CA).
  2. Update the AFX servers.
  3. Update the Remote Collector Agents (if any).

1.  Update the root certificate and Certificate Authority (CA).


Update the root certificate and CA by updating the server.keystore and restarting the RSA G&L application.

Note: Following these steps will temporarily break the connection for any existing remote agents until all steps are completed.


  1. In the UI, go to Admin > System > Security.
  2. Under Server Certificate Store for Agent SSL Connections, click the Change Certificate Store button. This will generate a new certificate.
User-added image

 

  1. You will get a dialog warning message. Click OK to change the root certificate and CA.
  2. Click the Download button, and save the server.keystore to a location on your computer.
User-added image

  1. Go to the location on the server where your server reads the keystore, back it up, and replace it with the new server.keystore.
  • For 6.8.1 and 6.9.0 and 6.9.1 
The default location of server.keystore on a JBoss appliance is: $HOME/jboss-4.2.2.GA/server/default/conf/keystore.

  • For 7.0.0 and 7.0.1 
The default location of server.keystore on a WildFly appliance is: /home/oracle/keystore.


  1. Go to the keystore directory
$ su oracle
$ cd $HOME/jboss-4.2.2.GA/server/default/conf/keystore or cd /home/oracle/keystore



  1. Backup the existing server.keystore:
$ mv server.keystore server.keystore.bak


  1. Copy the new server.keystore file to this location (this will replace the existing server.keystore with the new server.keystore you just generated.

  1. Restart RSA G&L:
$ su oracle
$ acm restart

2.  Update the AFX servers



Update the AFX server client certificate by updating the client.keystore and restarting AFX and the RSA G&L application.

  1. In the UI, go to AFX > Servers.
  2. For each AFX server, click on the name.
  3. Click the Change Certificate button. This action generates a new client certificate based off the new server certificate just generated in Step 1 and ensures the client certificate stored in the database matches the server certificate stored in the database.

User-added image


  1. You will get a dialog warning message. Click OK to change the client certificate.
  2. Click Download Keystore to download the new client.keystore.
User-added image


 

  1. Go to the location on the server where the client reads the keystore, back it up, and replace with the new client.keystore.
    1. Go to the keystore directory
$ su oracle
$ cd $AFX_HOME/esb/conf

  1. Backup the existing client.keystore:


$ mv client.keystore client.keystore.bak


  1. Copy the new client.keystore file to this location (this will replace the existing client.keystore with the new client.keystore you just generated.)
 

  1. Restart AFX and RSA G&L:
    1. Stop AFX

$ su oracle
$ afx stop
or
$ cd $AFX_HOME/bin
$ ./afx stop
or
$ su root
# service afx_server stop



  1. Restart RSA L&G
$ su oracle
$ acm restart

  1. Restart AFX
$ su oracle
$ afx start
or
$ cd $AFX_HOME/bin
$ ./afx start
or
$ su root
# service afx_server start




3.  Update the Remote Collector Agents (if any)



  1. In the UI, go to Collectors > Agents.
  2. For each remote agent (not the default local AveksaAgent), click on the agent name.
  3. Click the Change Certificate button. This action generates a new client certificate based off the new server certificate just generated in Step 1 and ensures the client certificate stored in the database matches the server certificate stored in the database.
User-added image



  1. You will get a dialog warning message. Click OK to change the client certificate.
  2. Click Download Agent to download a new agent with the new certificate in a zip file called AveksaAgent.zip.

User-added image


  1. Go to the location on the remote server that has the remote agent.
  2. Stop the agent by running agent_stop.sh in the bin directory, as follows:
$ su oracle
$ cd bin
$ ./agent_stop.sh

  1. Backup the agent directory
$ su oracle
$ mv <agent-directory> <agent-directory.bak>

  1. Unzip the agent on the remote server where it runs (replacing the old one.) 
$ unzip AveksaAgent.zip

  1. Start the agent by running agent_start.sh in the bin directory. 



$ cd bin
$ ./agent_start.sh



Attachments

    Outcomes