Running AFX Connectors start to fail, edited connectors remain in a Not Deployed state and 'Failed to start inbound endpoint "endpoint.http.localhost.8089.callback"; Address already in use' error in RSA Identity Governance & Lifecycle

[#1]

Article Number

000032419

Applies To

RSA Product Set: Identity Governance & Lifecycle
RSA Version: 6.x, 7.0.x, 7.1.x
 

Issue

All AFX transactions are failing yet the AFX Server and AFX Connectors show as Running.

Additional symptoms are:

• The AFX server is Running and the AFX connectors initially show as Running but any change request using AFX eventually fails over to manual fulfillment.
• A test of any connector times out.
• Editing any connector and saving the changes (even  if no changes were made), causes the connector to go into a Not Deployed state and remain in the Not Deployed state. 

• The $AFX_HOME/esb/logs/esb.AFX-MAIN.log has the following exception:

2019-10-17 09:07:04.531 [INFO] org.mule.transport.http.HttpConnector:193 -
Registering listener: AFX_ASYNC_CALLBACK_LISTENER on endpointUri: http://localhost:8089/callback
2019-10-17 09:07:04.533 [INFO] org.mule.transport.service.DefaultTransportServiceDescriptor:193 -
Loading default response transformer: org.mule.transport.http.transformers.MuleMessageToHttpResponse
2019-10-17 09:07:04.545 [INFO] org.mule.lifecycle.AbstractLifecycleManager:193 -
Initialising: 'null'. Object is: HttpMessageReceiver
2019-10-17 09:07:04.547 [INFO] org.mule.transport.http.HttpMessageReceiver:193 -
Connecting clusterizable message receiver
2019-10-17 09:07:04.559 [ERROR] org.mule.module.launcher.application.DefaultMuleApplication:361 -
null
java.net.SocketException: Address already in use (Listen failed)
        at java.net.PlainSocketImpl.socketListen(Native Method)
        at java.net.AbstractPlainSocketImpl.listen(AbstractPlainSocketImpl.java:399)
        at java.net.ServerSocket.bind(ServerSocket.java:376)

 

• The $AFX_HOME/esb/logs/mule_ee.log has the following errors:  

ERROR 2019-10-17 07:33:41,737 [WrapperListener_start_runner]
org.mule.module.launcher.DefaultArchiveDeployer:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentStartException: SocketException: Address alreadROR 2019-10-17 07:33:41,737 [WrapperListener_start_runner] org.mule.module.launcher.DefaultArchiveDeployer:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentStartException: SocketException: Address already in use (Listen failed)
        at org.mule.module.launcher.application.DefaultMuleApplication.start(DefaultMuleApplication.java:155)
        at org.mule.module.launcher.artifact.ArtifactWrapper$4.execute(ArtifactWrapper.java:98)
        at org.mule.module.launcher.artifact.ArtifactWrapper.executeWithinArtifactClassLoader(ArtifactWrapper.java:129)
        at org.mule.module.launcher.artifact.ArtifactWrapper.start(ArtifactWrapper.java:93)
        at org.mule.module.launcher.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:26)
        at org.mule.module.launcher.DefaultArchiveDeployer.guardedDeploy(DefaultArchiveDeployer.java:310)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployArtifact(DefaultArchiveDeployer.java:330)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedApp(DefaultArchiveDeployer.java:297)
        at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedArtifact(DefaultArchiveDeployer.java:108)
        at org.mule.module.launcher.DeploymentDirectoryWatcher.deployExplodedApps(DeploymentDirectoryWatcher.java:289)
        at org.mule.module.launcher.DeploymentDirectoryWatcher.start(DeploymentDirectoryWatcher.java:146)
        at org.mule.module.launcher.MuleDeploymentService.start(MuleDeploymentService.java:99)
        at org.mule.module.launcher.MuleContainer.start(MuleContainer.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mule.module.reboot.MuleContainerWrapper.start(MuleContainerWrapper.java:52)
        at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4048)
Caused by: org.mule.api.lifecycle.LifecycleException: Failed to start inbound endpoint "endpoint.http.localhost.8089.callback"

 

Cause

The exception error indicates that port 8089 is already in use by another application. AFX uses port 8089 for RESTful WebService connectors for receiving asynchronous callback messages. Because the port is in use, the internal Mule HTTP connector for the AFX-MAIN application is getting destroyed and subsequently not available when attempting to connect to the Mule Management Console (MMC). 

Port 8089 is often called by Splunk for splunkd services, a monitoring application. Other applications may also use port 8089.
 

Resolution

Identify what other application or process is using port 8089 and permanently free it up for use with AFX. 

Please refer to the RSA Identity Governance & Lifecycle Installation Guide for your RSA Identity Governance & Lifecycle version to see the ports required by AFX.

 

Workaround

A temporary workaround is to modify the default port used by AFX in the $AFX_HOME/esb/lib/user/afx-config.properties file. Ensure you choose a port that is not used by any other service.

1. Modify the afx.async.callback.port parameter.

From:

afx.async.callback.port=8089

To:

afx.async.callback.port=<unused port>

2. Restart AFX:

$ afx restart

 

WARNING: The proper solution is to permanently free up port 8089 for sole use by AFX. Modifying the afx-config.properties file is only a temporary solution as it will be overwritten every time a patch is applied or the system is upgraded.

Notes

NOTE:  Prior to version 7.1.x, the AFX server is Running but the connectors initially show in a Not Deployed state. The errors in the log files are the same and the problem cause is the same. Prior to 7.0.x the AFX log files are $AFX_HOME/mule/logs/mule.AFX-MAIN.log, $AFX_HOME/mule/mule_ee.log, and $AFX_HOME/esb/lib/user/afx-config.properties.