RSA Via Lifecycle & Governance 6.9.1 Access Fulfillment Express (AFX) connectors stuck in deployed state
2 years ago
Originally Published: 2016-06-29
Article Number
000064065
Applies To
RSA Product Set: Identity Management and Governance
RSA Product/Service Type: Access Fulfill Express
RSA Version/Condition: 6.9.1
Issue
mule_ee.log
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ New app 'AFX-CONN-AD-Connector'     +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO  2016-06-29 14:42:44,135 [WrapperListener_start_runner] org.mule.module.launcher.MuleApplicationClassLoader: [AFX-CONN-AD-Connector] Loading the following jars:
=============================
file:/export/local/home/rsasvc/AFX/mule/apps/AFX-CONN-AD-Connector/lib/jldap-4.3.jar
file:/export/local/home/rsasvc/AFX/mule/apps/AFX-CONN-AD-Connector/lib/afx-transport-ldap.jar
file:/export/local/home/rsasvc/AFX/mule/apps/AFX-CONN-AD-Connector/lib/mule-transport-ldapx-1.0.jar
file:/export/local/home/rsasvc/AFX/mule/apps/AFX-CONN-AD-Connector/lib/afx-connector-activedirectory.jar
=============================

ERROR 2016-06-29 14:42:51,728 [WrapperListener_start_runner] org.mule.module.launcher.DeploymentService:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy app                                     +
+ 'AFX-CONN-AD-Connector', see below  +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: UnrecoverableKeyException: Cannot recover key
        at org.mule.module.launcher.application.DefaultMuleApplication.init(DefaultMuleApplication.java:221)
        at org.mule.module.launcher.application.ApplicationWrapper.init(ApplicationWrapper.java:64)
        at org.mule.module.launcher.DefaultMuleDeployer.deploy(DefaultMuleDeployer.java:46)
        at org.mule.module.launcher.DeploymentService.guardedDeploy(DeploymentService.java:398)
        at org.mule.module.launcher.DeploymentService.start(DeploymentService.java:181)
        at org.mule.module.launcher.MuleContainer.start(MuleContainer.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mule.module.reboot.MuleContainerWrapper.start(MuleContainerWrapper.java:56)
        at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:3925)
Caused by: org.mule.api.config.ConfigurationException: Error creating bean with name 'httpsConnector-endpoint': Invocation of init method failed; nested exception is org.mule.api.lifecycle.InitialisationException: Failed to load Key Manager (org.mule.api.lifecycle.InitialisationException) (org.mule.api.config.ConfigurationException)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:52)
        at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
        at org.mule.context.DefaultMuleContextFactory.createMuleContext(DefaultMuleContextFactory.java:80)
        at org.mule.module.launcher.application.DefaultMuleApplication.init(DefaultMuleApplication.java:209)
        ... 11 more
Caused by: org.mule.api.config.ConfigurationException: Error creating bean with name 'httpsConnector-endpoint': Invocation of init method failed; nested exception is org.mule.api.lifecycle.InitialisationException: Failed to load Key Manager (org.mule.api.lifecycle.InitialisationException)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:52)
        at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
        at org.mule.config.builders.AutoConfigurationBuilder.autoConfigure(AutoConfigurationBuilder.java:101)
        at org.mule.config.builders.AutoConfigurationBuilder.doConfigure(AutoConfigurationBuilder.java:57)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:46)
        ... 14 more
Caused by: org.mule.api.lifecycle.InitialisationException: Error creating bean with name 'httpsConnector-endpoint': Invocation of init method failed; nested exception is org.mule.api.lifecycle.InitialisationException: Failed to load Key Manager
        at org.mule.registry.AbstractRegistry.initialise(AbstractRegistry.java:117)
        at org.mule.config.spring.SpringXmlConfigurationBuilder.createSpringRegistry(SpringXmlConfigurationBuilder.java:116)
        at org.mule.config.spring.SpringXmlConfigurationBuilder.doConfigure(SpringXmlConfigurationBuilder.java:73)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:46)
        ... 18 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpsConnector-endpoint': Invocation of init method failed; nested exception is org.mule.api.lifecycle.InitialisationException: Failed to load Key Manager
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:913)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464)
        at org.mule.config.spring.SpringRegistry.doInitialise(SpringRegistry.java:89)
        at org.mule.registry.AbstractRegistry.initialise(AbstractRegistry.java:109)
        ... 21 more
Caused by: org.mule.api.lifecycle.InitialisationException: Failed to load Key Manager
        at org.mule.transport.http.HttpsConnector.doInitialise(HttpsConnector.java:84)
        at org.mule.transport.AbstractConnector$1.onTransition(AbstractConnector.java:384)
        at org.mule.transport.AbstractConnector$1.onTransition(AbstractConnector.java:359)
        at org.mule.lifecycle.AbstractLifecycleManager.invokePhase(AbstractLifecycleManager.java:141)
        at org.mule.transport.ConnectorLifecycleManager.fireInitialisePhase(ConnectorLifecycleManager.java:48)
        at org.mule.transport.AbstractConnector.initialise(AbstractConnector.java:358)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1581)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1522)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
        ... 32 more
Caused by: org.mule.api.lifecycle.CreateException: Failed to load Key Manager
        at org.mule.api.security.tls.TlsConfiguration.initKeyManagerFactory(TlsConfiguration.java:242)
        at org.mule.api.security.tls.TlsConfiguration.initialise(TlsConfiguration.java:194)
        at org.mule.transport.http.HttpsConnector.doInitialise(HttpsConnector.java:80)
        ... 43 more
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
        at sun.security.provider.KeyProtector.recover(KeyProtector.java:311)
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121)
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38)
        at java.security.KeyStore.getKey(KeyStore.java:763)
        at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:113)
        at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239)
        at org.mule.api.security.tls.TlsConfiguration.initKeyManagerFactory(TlsConfiguration.java:238)
        ... 45 more
  AFX connector log
2016-06-29 14:42:51.713 [ERROR] org.mule.module.launcher.application.DefaultMuleApplication:365 - null
java.security.UnrecoverableKeyException: Cannot recover key
        at sun.security.provider.KeyProtector.recover(KeyProtector.java:311)
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121)
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38)
        at java.security.KeyStore.getKey(KeyStore.java:763)
        at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:113)
        at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239)
        at org.mule.api.security.tls.TlsConfiguration.initKeyManagerFactory(TlsConfiguration.java:238)
        at org.mule.api.security.tls.TlsConfiguration.initialise(TlsConfiguration.java:194)
        at org.mule.transport.http.HttpsConnector.doInitialise(HttpsConnector.java:80)
        at org.mule.transport.AbstractConnector$1.onTransition(AbstractConnector.java:384)
        at org.mule.transport.AbstractConnector$1.onTransition(AbstractConnector.java:359)
        at org.mule.lifecycle.AbstractLifecycleManager.invokePhase(AbstractLifecycleManager.java:141)
        at org.mule.transport.ConnectorLifecycleManager.fireInitialisePhase(ConnectorLifecycleManager.java:48)
        at org.mule.transport.AbstractConnector.initialise(AbstractConnector.java:358)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1581)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1522)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:913)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464)
        at org.mule.config.spring.SpringRegistry.doInitialise(SpringRegistry.java:89)
        at org.mule.registry.AbstractRegistry.initialise(AbstractRegistry.java:109)
        at org.mule.config.spring.SpringXmlConfigurationBuilder.createSpringRegistry(SpringXmlConfigurationBuilder.java:116)
        at org.mule.config.spring.SpringXmlConfigurationBuilder.doConfigure(SpringXmlConfigurationBuilder.java:73)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:46)
        at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
        at org.mule.config.builders.AutoConfigurationBuilder.autoConfigure(AutoConfigurationBuilder.java:101)
        at org.mule.config.builders.AutoConfigurationBuilder.doConfigure(AutoConfigurationBuilder.java:57)
        at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:46)
        at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
        at org.mule.context.DefaultMuleContextFactory.createMuleContext(DefaultMuleContextFactory.java:80)
        at org.mule.module.launcher.application.DefaultMuleApplication.init(DefaultMuleApplication.java:209)
        at org.mule.module.launcher.application.ApplicationWrapper.init(ApplicationWrapper.java:64)
        at org.mule.module.launcher.DefaultMuleDeployer.deploy(DefaultMuleDeployer.java:46)
        at org.mule.module.launcher.DeploymentService.guardedDeploy(DeploymentService.java:398)
        at org.mule.module.launcher.DeploymentService.start(DeploymentService.java:181)
        at org.mule.module.launcher.MuleContainer.start(MuleContainer.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mule.module.reboot.MuleContainerWrapper.start(MuleContainerWrapper.java:56)
        at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:3925)


 
Cause
cacerts contains an entry of type PrivateKeyEntry, where the private key password is different than the keystore password itself
Resolution
cacerts should not contain any entry of type PrivateKeyEntry
 
cd $JAVA_HOME/jre/lib/security
cp cacerts cacerts.bak
keytool -list -keystore cacerts -storepass changeit | grep PrivateKeyEntry
keytool -delete -alias <privatekeyentry_alias> -keystore cacerts -storepass changeit