AFX Connectors remain in a Deployed state and Port 8585 already in use error in RSA Identity Governance & Lifecycle

RSA Product Set: Identity Governance & Lifecycle 
RSA Version/Condition: 6.8.x, 6.9.x, 7.0.x, 7.1.x
 

The AFX Server is in a Running state but all previously working RSA Identity Governance & Lifecycle AFX Connectors are stuck in a Deployed state and do not progress to a Running state. For example,
 
The $AFX_HOME/mmc-console/logs/catalina.out has the following errors in the log file:
 

Oct 15, 2019 9:58:10 AM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-8585"]
java.net.BindException: Address already in use (Bind failed) <null>:8585
        at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:413)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:665)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:452)
        at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:840)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:642)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:667)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)
Caused by: java.net.BindException: Address already in use (Bind failed)
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at java.net.ServerSocket.<init>(ServerSocket.java:181)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
        at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:400)
        ... 17 more

Oct 15, 2019 9:58:10 AM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-8585]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-8585]]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:840)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:642)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:667)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:980)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        ... 12 more
Caused by: java.net.BindException: Address already in use (Bind failed) <null>:8585
        at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:413)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:665)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:452)
        at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
        ... 13 more
Caused by: java.net.BindException: Address already in use (Bind failed)
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at java.net.ServerSocket.<init>(ServerSocket.java:181)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
        at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:400)
        ... 17 more

The $AFX_HOME/esb/logs/esb.AFX-MAIN.log has multiple errors such as:

2019-10-15 10:00:20.607 
[WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:95 -
Unable get/setup the server id from the MMC request 
2019-10-15 10:00:20.608
[WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:96 -
Retrying MMC connection and server id setup...

2019-10-15 10:00:25.609 
[WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:99 -
Connection request retry attempt #1 of 12
2019-10-15 10:00:25.612 
[INFO] org.apache.commons.httpclient.HttpMethodDirector:193 -
I/O exception (java.net.ConnectException) caught when processing request:
Connection refused (Connection refused) ...

2019-10-15 10:01:20.724 
[WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:101 -
Unable get/setup the server id from the MMC request
2019-10-15 10:01:20.725 
[ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:102 -
Server Id setup try count exceeded

2019-10-15 10:01:20.725 
[ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:103 -
Please verify that the MMC console is running and Host & Port in the URL are correct.
2019-10-15 10:01:20.726 
[ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:82 - 
Unable to initialize the MMC Request Manager com.aveksa.afx.server.manager.MMCException:
Unable to retrieve and setup the Afx server Id information ...

2019-10-15 10:01:20.916 
[ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:136 -
Unable to get status for all Connectors from MMC
com.aveksa.afx.server.manager.MMCException:
Failed to get flows for server.  No associated server Id.
Check Request Manager Initialization

 

The third party Mule Management Console (MMC) used by AFX has failed to start because the port it uses, 8585, is in use by another process. The key error is:
 

java.net.BindException: Address already in use (Bind failed) <null>:8585

The MMC collects connector information in order for the connectors to run. If the MMC application fails to start, then the connector information is never collected and the connectors cannot move from Deployed to Running.
 

Make sure port 8585 is open for use. 

1. To check what application is using port 8585, as the root user, execute the following:

netstat tulpn | grep 8585

2. Configure the application identified in step 1 to use a different port.

We've seen this situation on Weblogic deployments when the NodeManager is configured to listen on port 8585. Configure NodeManager to list on a different port such as 8586. Please contact your Weblogic Administrator for information on how to modify the NodeManager configuration.