000036079 - Not able to access GUI due to rabbitmq issue in RSA NetWitness Logs & Packets 10.6.x

Document created by RSA Customer Support Employee on Mar 22, 2018
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000036079
Applies ToRSA Product Set: NetWitness Logs and Packets
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 10.6.3.0
Platform: CentOS
O/S Version: 6
IssueIf rabbitmq process is suddenly stuck or hang, you are not able to login Web UI after restart jetty service.
You may see following errors during restart in jetty log(stderrout.log) under /opt/rsa/jetty9/logs.

2018-02-22 14:24:21,524 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] WARN  com.rsa.smc.sa.live.job.quartz.SubscriptionJob - Resource Subscription with name [ High Risk File ] is Discontinued and hence will not be Synched
2018-02-22 14:24:42,810 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Failed to connect to SMS service, make sure all services are running and restart SA web server.
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
        at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
        at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
        at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.server.Server.start(Server.java:342)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
        at org.eclipse.jetty.server.Server.doStart(Server.java:290)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
        at org.eclipse.jetty.start.Main.start(Main.java:651)
        at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42,812 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Failed to migrate appliances, services and groups
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
        at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
        at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
        at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.server.Server.start(Server.java:342)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
        at org.eclipse.jetty.server.Server.doStart(Server.java:290)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
        at org.eclipse.jetty.start.Main.start(Main.java:651)
        at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42,813 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Migration failed.
2018-02-22 14:24:42,822 [main] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
        at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
        at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
        at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
        at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
        at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
        at org.eclipse.jetty.server.Server.start(Server.java:342)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
        at org.eclipse.jetty.server.Server.doStart(Server.java:290)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
        at org.eclipse.jetty.start.Main.start(Main.java:651)
        at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42.824:INFO:/:main: Closing Spring root WebApplicationContext
2018-02-22 14:24:42,833 [main] INFO  org.quartz.core.QuartzScheduler - Scheduler org.springframework.scheduling.quartz.SchedulerFactoryBean#0_$_NON_CLUSTERED paused.
2018-02-22 14:24:50,228 [main] INFO  com.rsa.netwitness.carlos.jmx.DefaultConnectorServerFactoryBean - Stopping JMX connector server: javax.management.remote.rmi.RMIConnectorServer@28e4680
2018-02-22 14:25:05,210 [Puppet Service Daemon 1936322081] ERROR com.rsa.smc.sa.admin.service.DefaultPuppetService - Failed to discover new hosts
com.rsa.smc.sa.admin.exception.PuppetException: Failed to discover new hosts
        at com.rsa.smc.sa.admin.service.DefaultPuppetService.refreshCertificateList(DefaultPuppetService.java:148)
        at com.rsa.smc.sa.admin.service.DefaultPuppetService$2.run(DefaultPuppetService.java:240)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.rsa.smc.sa.admin.exception.MCOAgentException:
        at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.execute(AbstractMCOAgent.java:72)
        at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.executeAndReturnSingle(AbstractMCOAgent.java:89)
        at com.rsa.smc.sa.admin.service.DefaultPuppetService.refreshCertificateList(DefaultPuppetService.java:132)
        ... 6 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 4 (Exit value: 4)
        at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
        at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
        at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
        ... 1 more
2018-02-22 14:25:42,817 [System Monitoring Service ApplianceConfig 1562356232] ERROR com.rsa.smc.sa.admin.service.DefaultSystemMonitoringService - Failed to start system monitoring task
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
        at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy135.getAllAppliances(Unknown Source)
        at com.rsa.smc.sa.admin.service.DefaultSystemMonitoringService$4.run(DefaultSystemMonitoringService.java:267)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-02-22 14:26:24,097 [Provision Daemon 1684357947] ERROR com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine - com.rsa.smc.sa.admin.exception.MCOAgentException:
2018-02-22 14:26:24,098 [Provision Daemon 1684357947] ERROR com.rsa.smc.sa.admin.management.provision.DefaultApplianceProvision - Failed to add services on c8006394-3995-4e89-b53d-e57674a46625 with error
com.rsa.smc.sa.admin.exception.PuppetException: Failed to add services on c8006394-3995-4e89-b53d-e57674a46625 with error
        at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.provision(ApplianceAddServicesState.java:94)
        at com.rsa.smc.sa.admin.management.provision.DefaultApplianceProvision$3.run(DefaultApplianceProvision.java:478)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.rsa.smc.sa.admin.exception.ProcessEngineException: Failed to get available services for c8006394-3995-4e89-b53d-e57674a46625
        at com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine.getAvailableProcesses(MCOUpstartProcessEngine.java:132)
        at com.rsa.smc.sa.admin.service.DefaultProcessService.getAvailableServices(DefaultProcessService.java:173)
        at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.getEndpointTypes(ApplianceAddServicesState.java:235)
        at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.provision(ApplianceAddServicesState.java:80)
        ... 6 more
Caused by: com.rsa.smc.sa.admin.exception.MCOAgentException:
        at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.execute(AbstractMCOAgent.java:72)
        at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.executeAndReturnSingle(AbstractMCOAgent.java:89)
        at com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine.getAvailableProcesses(MCOUpstartProcessEngine.java:122)
        ... 9 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 4 (Exit value: 4)
        at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
        at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
        at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
        ... 1 more
Causejetty web server port 443 is not listening up due to rabbitmq(beam.smp) process port issue. (CLOSE_WAIT)
 
# service rabbitmq-server status
Status of node sa@localhost ...
Error: unable to connect to node sa@localhost: nodedown

DIAGNOSTICS
===========

attempted to contact: [sa@localhost]

sa@localhost:
  * connected to epmd (port 4369) on localhost
  * node sa@localhost up, 'rabbit' application running

current node details:
- node name: 'rabbitmq-cli-27475@XXXXXXSAHAED'
- home dir: /var/lib/rabbitmq
- cookie hash: AMaoZBL/zIUG+HwcbpJAQw==


# netstat -anop|grep beam
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      6290/beam.smp       off (0.00/0/0)
tcp        0      0 0.0.0.0:25672               0.0.0.0:*                   LISTEN      6290/beam.smp       off (0.00/0/0)
tcp        0      0 0.0.0.0:37876               0.0.0.0:*                   LISTEN      28606/beam.smp      off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:47908        xxx.xxx.xx.147:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:56314        xxx.xx.xx.102:5671          CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:58090        xxx.xxx.xx.138:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:44281        xxx.xxx.xx.144:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:33612        xxx.xxx.xx.36:5671          CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:57301        xxx.xxx.xx.143:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp        0      0 127.0.0.1:5672              127.0.0.1:56702             ESTABLISHED 6290/beam.smp       off (0.00/0/0)
tcp        0      0 127.0.0.1:5672              127.0.0.1:38918             ESTABLISHED 6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:51634        xxx.xxx.xx.135:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:57449        xxx.xxx.xx.136:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp        8      0 127.0.0.1:5672              127.0.0.1:56324             ESTABLISHED 6290/beam.smp       off (0.00/0/0)
tcp        0      0 127.0.0.1:55293             127.0.0.1:4369              ESTABLISHED 28606/beam.smp      off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:50101        xxx.xxx.xx.133:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:39181        xxx.xxx.xx.33:5671          CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:56010        xxx.xxx.xx.146:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:58717        xxx.xxx.xx.146:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:47859        xx.x.xx.101:5671            CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:40090        xxx.xxx.xx.140:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:43488        xxx.xxx.xx.135:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:43597        xxx.xxx.xx.144:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp        8      0 127.0.0.1:5672              127.0.0.1:56630             ESTABLISHED 6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:50561        xxx.xxx.xx.138:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      107      0 xxx.xxx.xx.145:54692        xxx.xxx.xx.141:5671         CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
tcp      160      0 xxx.xxx.xx.145:41234        xxx.xxx.xx.34:5671          CLOSE_WAIT  6290/beam.smp       off (0.00/0/0)
.
.

 
ResolutionOn the command line(SA Server)

To resolve rabbitmq port issue, follow the steps below.

        Pre-steps: stop puppet agent 
        # /etc/init.d/puppet stop

       
        1. stop jetty service
        # stop jettysrv

        2. stop rabbitmq service
        # service rabbitmq-server stop

        3. find beam.smp {PID} and kill
        # netstat -anop|grep beam
        # kill -9 {PID}


        4. restart rsa-sms service
        # service rsa-sms restart 

        5. start rabbitmq service
        # service rabbitmq-server start 

        6. start jetty service
        # start jettysrv

        7. Start the puppet agent
        # /etc/init.d/puppet start

After above steps, port 443 will be listening up and you are able to access Netwitness GUI.

Attachments

    Outcomes