000036424 - RSA NetWitness 11.x Admin Server does not discover new hosts

Document created by RSA Customer Support Employee on Jun 8, 2018
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000036424
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Admin Server
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7
Issue

NetWitness Admin Server does not discover any of the upgraded or newly deployed hosts.
Clicking on the Discover button does not return any host.

Manual attempt to add the hosts by following KB 35662 also fails with the following error.




[root@NWServer~]# orchestration-cli-client --accept-key <UUID>
2018-06-08 01:45:55.019  INFO 12194 --- [           main] Bootstrap                                : Service logs will be written to /var/log/netwitness/orchestration-client
2018-06-08 01:45:55.030  INFO 12194 --- [           main] Bootstrap                                : Service configuration will be read from /etc/netwitness/orchestration-client
2018-06-08 01:45:55.310  INFO 12194 --- [           main] Bootstrap                                : Starting orchestration-client.29a5039f-d998-4d9a-ade9-dc47a4a4b86b (v0.0.0.0)
2018-06-08 01:45:56.202  INFO 12194 --- [           main] Bootstrap                                : Initialized service cryptography with 4 providers (BSAFE=CRYPTOJ 6.2.2 20161215 0745, FIPS-140=true).
2018-06-08 01:45:57.363  INFO 12194 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : Starting OrchestrationApplication on ausasrv with PID 12194 (/usr/bin/orchestration-cli-client.jar started by root in /root)
2018-06-08 01:45:57.364  INFO 12194 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : The following profiles are active: standard
2018-06-08 01:45:57.550  INFO 12194 --- [           main] Bootstrap                                : Service will accept AMQP requests at broker localhost:5672/rsa/system
2018-06-08 01:45:57.558  INFO 12194 --- [           main] Bootstrap                                : Service will use the deployment security-server
2018-06-08 01:45:59.611  INFO 12194 --- [shake Completed] Security                                 : Accepted new connection with CN=5683d138-820a-4f8f-9b73-08996f3de8b6,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US from localhost using TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
2018-06-08 01:46:00.875 ERROR 12194 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : Exception processing request

com.rsa.asoc.launch.api.transport.client.TransportClientException: Connection refused (Connection refused)
        at com.rsa.asoc.launch.api.transport.client.ClientResponseUtils.handleError(ClientResponseUtils.java:99)
        at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSend(AmqpTransportClient.java:131)
        at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:86)
        at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:69)
        at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
        at com.sun.proxy.$Proxy60.getAll(Unknown Source)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationClient.getKeys(OrchestrationClient.java:91)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.run(OrchestrationApplication.java:118)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784)
        at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.main(OrchestrationApplication.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:587)
Caused by: java.net.ConnectException: Connection refused (Connection refused)

[2018-06-08T01:46:01+00:00] <12191> (ERROR) Failed, aborting...


/var/log/netwitness/orchestration-client/orchestration-client.log shows the same error as below.


2018-06-08 01:46:00,875 [                          main] ERROR c.r.n.i.o.c.OrchestrationApplication|Exception processing request
com.rsa.asoc.launch.api.transport.client.TransportClientException: Connection refused (Connection refused)
        at com.rsa.asoc.launch.api.transport.client.ClientResponseUtils.handleError(ClientResponseUtils.java:99)
        at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSend(AmqpTransportClient.java:131)
        at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:86)
        at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:69)
        at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
        at com.sun.proxy.$Proxy60.getAll(Unknown Source)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationClient.getKeys(OrchestrationClient.java:91)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.run(OrchestrationApplication.java:118)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784)
        at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
        at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.main(OrchestrationApplication.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:587)
Caused by: java.net.ConnectException: Connection refused (Connection refused)


/var/log/netwitness/orchestration-server/orchestration-server.log shows an error that contains 'com.rsa.asoc.orchestration.salt.client'.


2018-06-08 01:44:40,422 [                    cfg-mgmt-0] ERROR ConfigurationManagement|Execution of '5b1766b8b0d12e04cd418deb' failed
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:8000/run": Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
        at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118)
        at com.rsa.asoc.orchestration.salt.handler.HostVerifyTaskHandler.execute(HostVerifyTaskHandler.java:52)
        at com.rsa.asoc.orchestration.task.TaskExecutionService.execute(TaskExecutionService.java:164)
        at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$null$3(TaskExecutionService.java:148)
        at java.util.ArrayList.forEach(ArrayList.java:1249)
        at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$submit$4(TaskExecutionService.java:148)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89)
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
        ... 13 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
        ... 26 common frames omitted


salt-api service continues to crash few seconds after starting it. Service status with -l option displays an error on cherrypy.


[root@NWServer ~]# systemctl status salt-api -l
● salt-api.service - The Salt API
   Loaded: loaded (/usr/lib/systemd/system/salt-api.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/salt-api.service.d
           └─salt-api-opts-managed.conf
   Active: inactive (dead)
     Docs: man:salt-api(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html

Jun 08 01:43:56 NWServer salt-api[11205]: [WARNING ] SaltAPI received a SIGTERM. Exiting.
Jun 08 01:43:56 NWServer salt-api[11205]: [INFO    ] The salt-api is shutting down..
Jun 08 01:43:56 NWServer salt-api[11205]: The salt-api is shutdown. SaltAPI received a SIGTERM. Exited.
Jun 08 01:43:56 NWServer systemd[1]: Stopped The Salt API.
Jun 08 01:44:09 NWServer systemd[1]: Starting The Salt API...
Jun 08 01:44:10 NWServer systemd[1]: Started The Salt API.
Jun 08 01:44:10 NWServer salt-api[11695]: [INFO    ] Setting up the Salt API
Jun 08 01:44:10 NWServer salt-api[11695]: [INFO    ] The salt-api is starting up
Jun 08 01:44:10 NWServer salt-api[11695]: [ERROR   ] Not loading 'salt.loaded.int.netapi.rest_cherrypy'. Error loading CherryPy: No module named cherrypy
Jun 08 01:44:10 NWServer salt-api[11695]: [ERROR   ] Did not find any netapi configurations, nothing to start
Hint: Some lines were ellipsized, use -l to show in full.
       




 
CauseThe issue can occur due to corrupted python-libs package.
ResolutionIn order to resolve the issue, please follow the steps below.

1. SSH into the Admin Server.
2. Reinstall python-libs package from the Admin Server.
yum reinstall python-libs

3. Start the salt-api service.
systemctl start salt-api

4. (Optional) If the issue continues, reinstall python-cherrypy as well and start salt-api service.
yum install python-cherrypy

5. Run Discover from NetWitness Server UI under ADMIN-Hosts.
6. If hosts are not discovered, follow KB 35662.

Attachments

    Outcomes