000032419 - After applying a patch or after upgrading RSA Identity Governance & Lifecycle 6.x or 7.0.x, Access Fulfillment Express (AFX) connectors remain in a "Not Deployed" state

Document created by RSA Customer Support Employee on Aug 12, 2016•Last modified by RSA Customer Support on Apr 29, 2019
Version 4Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000032419
Applies ToRSA Product Set: Identity Governance & Lifecycle
RSA Version: 6.x, 7.0.x
 
IssueAfter applying a patch or after upgrading RSA Identity Governance & Lifecycle 6.x or 7.0.x, the Access Fulfillment Express (AFX)  connectors stay in a Not Deployed state.  The AFX server  is in the Running state. Testing of the connectors succeeds.

The following errors can be found in $AFX_HOME/mule/logs/mule.AFX-MAIN.log (In 7.0.1 and 7.0.2 this log file can be found in $AFX_HOME/esb/logs.) The default location for  $AFX_HOME is /home/oracle/AFX.
 
2016-01-21 15:42:12.303 [ERROR] org.mule.exception.DefaultSystemExceptionStrategy:341 - ********************************************************************************
Message : Failed to bind to uri "http://localhost:8089/callback"
Code : MULE_ERROR-0
--------------------------------------------------------------------------------
Exception stack is:
1. Address already in use (java.net.BindException)
java.net.PlainSocketImpl:-2 (null)
2. Failed to bind to uri "http://localhost:8089/callback" (org.mule.transport.ConnectException)
org.mule.transport.tcp.TcpMessageReceiver:80 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:353)
at java.net.ServerSocket.bind(ServerSocket.java:336)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************


You will also see the following error when connectors are failing to move to running state:
2017-09-15 17:44:01.957 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:154 - Unable get/setup the flow list from the MMC request
2017-09-15 17:44:01.958 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:155 - Retrying MMC connection and flow list setup...
2017-09-15 17:44:11.958 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:158 - Connection request retry attempt #1 of 2
2017-09-15 17:44:11.959 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:154 - Unable get/setup the flow list from the MMC request
2017-09-15 17:44:11.959 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:155 - Retrying MMC connection and flow list setup...
2017-09-15 17:44:21.959 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:158 - Connection request retry attempt #2 of 2
2017-09-15 17:44:21.960 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:160 - Unable get/setup the flow list from the MMC request
2017-09-15 17:44:21.960 [ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:161 - Flow list setup try count exceeded
2017-09-15 17:44:21.960 [ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:162 - Please verify that the MMC console is running and Host & Port in the URL are correct.
2017-09-15 17:44:21.961 [ERROR] com.aveksa.afx.server.manager.MMCRequestManagerImpl:118 - Unable to get status for all Connectors from MMC com.aveksa.afx.server.manager.MMCException: Unable to retrieve and setup the flow list for server: local$98b3c841-fbfb-47a3-8738-42084c79b978


at com.aveksa.afx.server.manager.MMCRequestManagerImpl.getFlowList(MMCRequestManagerImpl.java:163)
at com.aveksa.afx.server.manager.MMCRequestManagerImpl.getAllConnectorStatus(MMCRequestManagerImpl.java:101)
at com.aveksa.afx.server.component.PrimaryRequestConstructorComponent.constructRequest(PrimaryRequestConstructorComponent.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source)
at org.mule.model.resolvers.AbstractEntryPointResolver.invokeMethod(AbstractEntryPointResolver.java:151)
at org.mule.impl.model.resolvers.AnnotatedEntryPointResolver.invoke(AnnotatedEntryPointResolver.java:141) ....
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534) Caused by: org.mule.api.lifecycle.LifecycleException: Cannot process event as "httpConnector" is stopped
at org.mule.lifecycle.processor.ProcessIfStartedMessageProcessor.handleUnaccepted(ProcessIfStartedMessageProcessor.java:42)
at org.mule.processor.AbstractFilteringMessageProcessor.process(AbstractFilteringMessageProcessor.java:48) ....
at org.mule.module.client.MuleClient.send(MuleClient.java:602)
at com.aveksa.afx.server.manager.MMCRequestManagerImpl.getFlowList(MMCRequestManagerImpl.java:149)
... 92 more
2017-09-15 17:44:31.956 [WARN] com.aveksa.afx.server.manager.MMCRequestManagerImpl:154 - Unable get/setup the flow list from the MMC request
CauseThe bind exception error indicates that port 8089 is already in use by another application. AFX uses port 8089 for RESTful WebService connectors for receiving asynchronous callback messages.

Because the port is in use, the internal Mule HTTP connector for the AFX-MAIN app is getting destroyed and subsequently not available when attempting to connect to the Mule Management Console (MMC). 

Port 8089 is often called by Splunk for splunkd services, a monitoring application.  Other applications may also use port 8089.



 
ResolutionIdentify what other application or process is using the port and permanently free it up for AFX. 

AFX uses the following ports:
 
PortUsage
61616Apache's ActiveMQ JMS port
8585Default MMC port
8444This port configured for installation on WebSphere and WebLogic. See the Installation and Upgrade on WebSphere Guide and the Installation and Upgrade on WebLogic Guide for our version for information on port configuration when AFX is installed those platforms.  
  
   See links to relevant documentation for your deployment in the Notes field below.
8089Default port for RESTful WebService connectors for receiving asynchronous callback messages. This port will be used by endpoints which process AFX requests asynchronously to send a response back to AFX at a later point in time (once the request has been processed, for example). The response would contain information that AFX can use to determine if the request was successfully processed or if an error was encountered.
WorkaroundA temporary workaround is to modify the default port used by AFX in the afx-config.properties file ($AFX_HOME/mule/lib/user/afx-config.properties).  Ensure you choose a port that is not used by any other services.
  1. Modify the <afx-user>/mule/lib/user/afx-config.properties callback port information from:


afx.async.callback.port=8089


Change to:



afx.async.callback.port=<unused port>


  1. Restart AFX:


$ afx restart

 

WARNING: The proper solution is to permanently free up port 8089 for AFX's sole use. Modifying the afx-config.properties file is only a temporary solution as it will be overwritten every time a patch is applied or the system is upgraded.

NotesFor RSA Identity Governance & Lifecycle 7.1.0 and later, see article 000037372 - Access Fulfillment Express (AFX) connectors remain in a Not Deployed state after applying a patch or after upgrading RSA Identity Governance & Lifecycle 7.1.0 or 7.1.1

Download documentation from:

6.8




6.9





7.0



Attachments

    Outcomes