Cause | EXAMPLE 1: If your upgrade doesn't get to the point that chef logs are being generated on the component host, generally, you can find a clue in the Orchestration Server logs. These are located in /var/log/netwitness/orchestration-server/orchestration-server.log on the NetWitness Admin Server. A very common situation includes a message like the following:
018-01-30 19:59:02,629 [ ServiceInstallJobHandler] WARN ConfigurationManagement|Failed to check isJobReady org.springframework.http.converter.HttpMessageNotReadableException: Could not read document: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]-> java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]-> java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]) at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:240) at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:225) at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95) at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:917) at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:901) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:655) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602) at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118) at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.isJobReady(AsynchronousTaskMonitor.java:115) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.pollTasks(AsynchronousTaskMonitor.java:88) at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]-> java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]) at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270) at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1122) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1075) at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:892) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:358) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27) at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:287) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26) at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:101) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:357) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2922) at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:237) ... 23 common frames omitted
This could indicate that there is a problem with the salt service on the host being upgraded that prevents the orchestration server from communicating with it properly. As part of the chef run, we mask the salt-minion service to prevent users from interacting with it until the chef process is done. In the case of a previous failure, we may never get to the step that we unmask the salt service, thus the above error is displayed.
EXAMPLE 2:
Another situation that can occur is a message like the following:
2018-12-17 23:44:10,004 [ cfg-mgmt-0] ERROR TaskManagement|Execution of '5c1834c991c71f054873335b' failed org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:8000/run": Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:636) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:610) at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118) at com.rsa.asoc.orchestration.salt.handler.HostVerifyTaskHandler.execute(HostVerifyTaskHandler.java:52) at com.rsa.asoc.orchestration.task.TaskExecutionService.execute(TaskExecutionService.java:163) at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$null$3(TaskExecutionService.java:147) at java.util.ArrayList.forEach(ArrayList.java:1257) at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$submit$4(TaskExecutionService.java:147) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660) ... 13 common frames omitted
This is also related to another salt issue but you are more likely to see this just on the Admin Server itself rather than another host. |
Resolution | If you find yourself fitting into Example 1, then do the following: Run the following on the target host to unmask and restart the service to allow the upgrade to, properly.
systemctl unmask salt-minion systemctl restart salt-minion
If you are finding yourself fitting into Example 2, then do the above AND also the following on the Admin Server:
systemctl unmask salt-master && systemctl unmask salt-api systemctl restart salt-master && systemctl restart salt-api
Then, you may start your upgrade once more on your target host. Be sure to tail the /var/log/netwitness/config-management/chef-solo.log files to watch the upgrade in action.
tailf /var/log/netwitness/config-management/chef-solo.log
|