| Article Number | 000036947 |
| Applies To | RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Admin Server, Component Host
RSA Version/Condition: 11.X
Platform: CentOS 7 |
| Issue | When reattempting to upgrade a NetWitness 11 host, you may see no new log messages being generated in /var/log/netwitness/config-management/chef-solo.log. Therefore, the upgrade does not seem like it's being started. |
| Cause | If your upgrade doesn't get to the point that chef logs are being generated on the component host, generally you can find a clue in the Orchestration Server logs. These are located in /var/log/netwitness/orchestration-server/orchestration-server.log on the NetWitness Admin Server. A very common situation includes a message like the following:
018-01-30 19:59:02,629 [ ServiceInstallJobHandler] WARN ConfigurationManagement|Failed to check isJobReady
org.springframework.http.converter.HttpMessageNotReadableException: Could not read document: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]);
nested exception is com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:240)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:225)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:917)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:901)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:655)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118)
at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.isJobReady(AsynchronousTaskMonitor.java:115)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.pollTasks(AsynchronousTaskMonitor.java:88)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1122)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1075)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:892)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:358)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:287)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:101)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:357)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2922)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:237)
... 23 common frames omitted
This could indicate that there is a problem with the salt service on the host that prevents the orchestration server from communicating with it properly. As part of the chef run, we mask the salt-minion service to prevent users from interacting with it until the chef process is done. In the case of a previous failure, we never get to the step that we unmask the salt service, thus the above error is displayed.
|
| Resolution | Run the following on the target host to unmask and restart the service to allow the upgrade to, properly.
systemctl unmask salt-minion
systemctl restart salt-minion
Then, you may start your upgrade once more.
Be sure to tail the /var/log/netwitness/config-management/chef-solo.log files to watch the upgrade in action.
tailf /var/log/netwitness/config-management/chef-solo.log
|
| Notes | If you are still having an issue, please submit a case to RSA Technical Support and quote this KB article. It would also help us if you can attach the /var/log/netwitness/orchestration-server/orchestration-server.log file from your NetWitness Admin Server and your /var/log/netwitness/config-management/chef-solo.log from the host that you are attempting to upgrade. |
on Nov 15, 2018