Running AFX Connectors start to fail, edited connectors remain in a Not Deployed state and 'Failed to start inbound endpoint "endpoint.http.localhost.8089.callback"; Address already in use' error in RSA Governance & Lifecycle
2 years ago
Originally Published: 2016-01-22
Article Number
000049858
Applies To
RSA Product Set: RSA Governance & Lifecycle
RSA Version: 6.x, 7.0.x, 7.1.x, 7.5.x, 8.0
 
Issue
All AFX transactions are failing yet the AFX Server and AFX Connectors show as Running.

Additional symptoms are:

  • The AFX server is Running and the AFX connectors initially show as Running but any change request using AFX eventually fails over to manual fulfillment.
  • A test of any connector times out.
  • Editing any connector and saving the changes (even  if no changes were made), causes the connector to go into a Not Deployed state and remain in the Not Deployed state. 
  • The $AFX_HOME/esb/logs/esb.AFX-MAIN.log has the following exception:
2019-10-17 09:07:04.531 [INFO] org.mule.transport.http.HttpConnector:193 -
Registering listener: AFX_ASYNC_CALLBACK_LISTENER on endpointUri: http://localhost:8089/callback
2019-10-17 09:07:04.533 [INFO] org.mule.transport.service.DefaultTransportServiceDescriptor:193 -
Loading default response transformer: org.mule.transport.http.transformers.MuleMessageToHttpResponse
2019-10-17 09:07:04.545 [INFO] org.mule.lifecycle.AbstractLifecycleManager:193 -
Initialising: 'null'. Object is: HttpMessageReceiver
2019-10-17 09:07:04.547 [INFO] org.mule.transport.http.HttpMessageReceiver:193 -
Connecting clusterizable message receiver
2019-10-17 09:07:04.559 [ERROR] org.mule.module.launcher.application.DefaultMuleApplication:361 -
null
java.net.SocketException: Address already in use (Listen failed)
        at java.net.PlainSocketImpl.socketListen(Native Method)
        at java.net.AbstractPlainSocketImpl.listen(AbstractPlainSocketImpl.java:399)
        at java.net.ServerSocket.bind(ServerSocket.java:376)
 
  • The $AFX_HOME/esb/logs/mule_ee.log has the following errors:  
ERROR 2019-10-17 07:33:41,737 [WrapperListener_start_runner]
org.mule.module.launcher.DefaultArchiveDeployer:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentStartException: SocketException: Address alreadROR 2019-10-17 07:33:41,737 [WrapperListener_start_runner] org.mule.module.launcher.DefaultArchiveDeployer:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentStartException: SocketException: Address already in use (Listen failed)
        at org.mule.module.launcher.application.DefaultMuleApplication.start(DefaultMuleApplication.java:155)
        at org.mule.module.launcher.artifact.ArtifactWrapper$4.execute(ArtifactWrapper.java:98)
        at org.mule.module.launcher.artifact.ArtifactWrapper.executeWithinArtifactClassLoader(ArtifactWrapper.java:129)
        at org.mule.module.launcher.artifact.ArtifactWrapper.start(ArtifactWrapper.java:93)
        at org.mule.module.launcher.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:26)
        at org.mule.module.launcher.DefaultArchiveDeployer.guardedDeploy(DefaultArchiveDeployer.java:310)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployArtifact(DefaultArchiveDeployer.java:330)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedApp(DefaultArchiveDeployer.java:297)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedArtifact(DefaultArchiveDeployer.java:108)
        at org.mule.module.launcher.DeploymentDirectoryWatcher.deployExplodedApps(DeploymentDirectoryWatcher.java:289)
        at org.mule.module.launcher.DeploymentDirectoryWatcher.start(DeploymentDirectoryWatcher.java:146)
        at org.mule.module.launcher.MuleDeploymentService.start(MuleDeploymentService.java:99)
        at org.mule.module.launcher.MuleContainer.start(MuleContainer.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mule.module.reboot.MuleContainerWrapper.start(MuleContainerWrapper.java:52)
        at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4048)
Caused by: org.mule.api.lifecycle.LifecycleException: Failed to start inbound endpoint "endpoint.http.localhost.8089.callback"
 
 
Cause
The exception error indicates that port 8089 is already in use by another application. AFX uses port 8089 for RESTful WebService connectors for receiving asynchronous callback messages. Because the port is in use, the internal Mule HTTP connector for the AFX-MAIN application is getting destroyed and subsequently not available when attempting to connect to the Mule Management Console (MMC). 

Port 8089 is often called by Splunk for splunkd services, a monitoring application. Other applications may also use port 8089.
 
Resolution
Identify what other application or process is using port 8089 and permanently free it up for use with AFX. 

Please refer to the RSA Governance & Lifecycle Installation Guide for your RSA Governance & Lifecycle version to see the ports required by AFX.
Workaround
A temporary workaround is to modify the default port 8089 used by AFX in its configuration file.

WARNING: The proper solution is to permanently free up port 8089 for sole use by AFX. Modifying the AFX configuration file is only a temporary solution as it will be overwritten every time a patch is applied or the system is upgraded.

For versions up to 7.5.x:

Modify the default port used by AFX in the $AFX_HOME/esb/lib/user/afx-config.properties file. Ensure you choose a port that is not used by any other service.
  1. Modify the afx.async.callback.port parameter.
From:
afx.async.callback.port=8089
To:
afx.async.callback.port=<unused port>
  1. Restart AFX:
$ afx restart


For version 8.0:

In version 8.0. the file afx-config.properties exists in TWO paths instead of one as in previous versions. Both Files should be modified and updated with the same new port value for afx.async.callback.port:

  1. Navigate to $AFX_HOME/esb/domains/01-AFX-DOMAIN/afx-config.properties
  2. Modify the afx.async.callback.port parameter. 
    From:
    afx.async.callback.port=8089
    To:
    afx.async.callback.port=<unused port>
  3. Navigate to $AFX_HOME/esb/lib/user/afx-config.properties
  4. Modify the afx.async.callback.port parameter. 
    From:
    afx.async.callback.port=8089
    To:
    afx.async.callback.port=<unused port>
  5. Restart AFX 
    $ afx restart
Notes
  • Prior to version 7.1.x, the AFX server is Running but the connectors initially show in a Not Deployed state. The errors in the log files are the same and the problem cause is the same.
  • Prior to version 7.0.x, the AFX log files are $AFX_HOME/mule/logs/mule.AFX-MAIN.log, $AFX_HOME/mule/mule_ee.log, and $AFX_HOME/esb/lib/user/afx-config.properties.
  • For version 8.0, the parameter should be updated in TWO files that exist in AFX:  $AFX_HOME/esb/domains/01-AFX-DOMAIN/afx-config.properties and $AFX_HOME/esb/lib/user/afx-config.properties

Related Articles

Trending Articles

Don't see what you're looking for?