AFX Server fails to start with java.lang.StackOverflowError errors in the log files after changing the application server hostname in RSA Identity Governance & Lifecycle

[#1]

Article Number

000032612

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle 
RSA Version/Condition: 6.9.1, 7.x
 

Issue

After running modifyhostname.sh on an RSA Identity Governance & Lifecycle hard/soft appliance, the AFX Server fails to start and the following message is displayed on the console:

All remote collector agents and AFX servers will need to have their client certificates
re-created within RSA IGL and then be downloaded and re-deployed.

  The following errors are logged to the AFX log files:
 

In the $AFX_HOME/esb/logs/esb.AFX-INIT.log:

 

2020-08-21 14:29:55.072 [INFO] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:71 - 
Error occurred during initialization java.lang.StackOverflowError
2020-08-21 14:29:55.072 [ERROR] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:196 - 
Error while attempting to execute initialization request!
2020-08-21 14:29:55.073 [ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - 
Server initialization failed! Please correct the issue and restart AFX.
org.mule.api.MessagingException: null (java.lang.StackOverflowError). 
Message payload is of type: PostMethod
	at 
...
Caused by: java.lang.StackOverflowError
	at com.rsa.cryptoj.o.ax.<init>(SourceFile)
	at com.rsa.cryptoj.o.ad.<init>(SourceFile)
	at com.rsa.cryptoj.o.ac.a(SourceFile)
	at com.rsa.cryptoj.o.aw.c(SourceFile)
	at com.rsa.cryptoj.o.av.c(SourceFile)
	at com.rsa.cryptoj.o.c.b(SourceFile)
	at com.rsa.cryptoj.o.c.a(SourceFile)
	at com.rsa.cryptoj.o.ak.c(SourceFile)
	at com.rsa.cryptoj.o.c.b(SourceFile)
	at com.rsa.cryptoj.o.c.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)

 

In the $AFX_HOME/esb/logs/esb.AFX-MAIN.log:

 

2020-08-21 14:29:56.081 [ERROR] org.mule.module.launcher.application.DefaultMuleApplication:361 - 
null java.lang.IllegalArgumentException: 
Could not resolve placeholder 'afx.server.activemq.password' 
in string value "${afx.server.activemq.password}"
...
2020-08-21 14:29:56.083 [INFO] org.mule.module.launcher.application.DefaultMuleApplication:193 - 
App '15_AFX-MAIN' never started, nothing to dispose of

 

In the $AFX_HOME/esb/logs/mule_ee.log:

 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: StackOverflowError: 
	at org.mule.module.launcher.application.DefaultMuleApplication.init(DefaultMuleApplication.java:196)
	at org.mule.module.launcher.artifact.ArtifactWrapper$2.execute(ArtifactWrapper.java:62)
	at org.mule.module.launcher.artifact.ArtifactWrapper.executeWithinArtifactClassLoader(ArtifactWrapper.java:129)
	at org.mule.module.launcher.artifact.ArtifactWrapper.init(ArtifactWrapper.java:57)
	at org.mule.module.launcher.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:25)
	at org.mule.module.launcher.DefaultArchiveDeployer.guardedDeploy(DefaultArchiveDeployer.java:310)
	at org.mule.module.launcher.DefaultArchiveDeployer.deployArtifact(DefaultArchiveDeployer.java:330)
	at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedApp(DefaultArchiveDeployer.java:297)
	at org.mule.module.launcher.DefaultArchiveDeployer.deployExplodedArtifact(DefaultArchiveDeployer.java:108)
	at org.mule.module.launcher.DeploymentDirectoryWatcher.deployExplodedApps(DeploymentDirectoryWatcher.java:289)
	at org.mule.module.launcher.DeploymentDirectoryWatcher.start(DeploymentDirectoryWatcher.java:146)
	at org.mule.module.launcher.MuleDeploymentService.start(MuleDeploymentService.java:99)
	at org.mule.module.launcher.MuleContainer.start(MuleContainer.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mule.module.reboot.MuleContainerWrapper.start(MuleContainerWrapper.java:52)
	at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4048)
Caused by: org.mule.api.config.ConfigurationException: 
Error creating bean with name 'serverInitialization' 
defined in URL [file:/home/oracle/AFX/esb/apps/10_AFX-INIT/mule-config.xml]: 
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: 
Could not instantiate bean class [com.aveksa.afx.server.init.ServerInitializationComponent]: 
Constructor threw exception; nested exception is org.mule.api.lifecycle.InitialisationException: 
Server initialization failed! Please correct the issue and restart AFX. 
(org.mule.api.lifecycle.InitialisationException) (org.mule.api.config.ConfigurationException)
....
Caused by: org.mule.api.MessagingException: null (java.lang.StackOverflowError). 
Message payload is of type: PostMethod
	at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute
(ExceptionToMessagingExceptionExecutionInterceptor.java:32)
	at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute
(MessageProcessorNotificationExecutionInterceptor.java:58)
	at org.mule.execution.MessageProcessorExecutionTemplate.execute
(MessageProcessorExecutionTemplate.java:44)
	at org.mule.processor.chain.SimpleMessageProcessorChain.doProcess
(SimpleMessageProcessorChain.java:43)
	at org.mule.processor.chain.AbstractMessageProcessorChain.process
(AbstractMessageProcessorChain.java:67)
...
Caused by: java.lang.StackOverflowError
	at com.rsa.cryptoj.o.aw.c(SourceFile)
	at com.rsa.cryptoj.o.av.c(SourceFile)
	at com.rsa.cryptoj.o.c.b(SourceFile)
	at com.rsa.cryptoj.o.c.a(SourceFile)
	at com.rsa.cryptoj.o.ak.c(SourceFile)
	at com.rsa.cryptoj.o.c.b(SourceFile)
	at com.rsa.cryptoj.o.c.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)
	at com.rsa.cryptoj.o.a.a(SourceFile)

 

Cause

The console message is explaining that the AFX and remote collection agent client certificates need to be regenerated and redeployed.

When the hostname is changed using the modifyhostname.sh script, the script generates a new server.keystore that uses a newly generated certificate authority (CA) and hostname certificate. When the server.keystore is updated, the AFX and remote collection agent client certificates also need to be updated.
 

Resolution

To resolve this issue:

• Download and redeploy the server certificate.
• Regenerate, download, and redeploy the client certificates for each AFX agent and remote collection agent.

For steps on regenerating and redeploying server and client certificates, see RSA Knowledge Base Article 000038314 -- How to Update the Root (Server) and Client Certificates in RSA Identity Governance & Lifecycle starting at step 3 under Update the server certificate - Click the Download button and save the server.keystore to a location on your computer.

NOTE: The server certificate does not need to be regenerated because the modifyhostname.sh procedure has already done that. However, regenerating the server certificate will cause no harm as long as you do that before all other steps. That is, regenerating the server certificate must be the first step in the process as it is written in the above-referenced RSA Knowledge Base Article.