000036947 - RSA NetWitness Host Fails to Upgrade but no new Chef Logs are being generated

Document created by RSA Customer Support Employee on Nov 15, 2018
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000036947
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Admin Server, Component Host
RSA Version/Condition: 11.X
Platform: CentOS 7
IssueWhen reattempting to upgrade a NetWitness 11 host, you may see no new log messages being generated in /var/log/netwitness/config-management/chef-solo.log. Therefore, the upgrade does not seem like it's being started.
CauseIf your upgrade doesn't get to the point that chef logs are being generated on the component host, generally you can find a clue in the Orchestration Server logs. These are located in /var/log/netwitness/orchestration-server/orchestration-server.log on the NetWitness Admin Server. A very common situation includes a message like the following:

018-01-30 19:59:02,629 [      ServiceInstallJobHandler] WARN  ConfigurationManagement|Failed to check isJobReady
org.springframework.http.converter.HttpMessageNotReadableException: Could not read document: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]);
 nested exception is com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:240)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:225)
        at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95)
        at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:917)
        at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:901)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:655)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
        at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118)
        at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.isJobReady(AsynchronousTaskMonitor.java:115)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.pollTasks(AsynchronousTaskMonitor.java:88)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
        at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1122)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1075)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:892)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:358)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:287)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:101)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:357)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2922)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:237)
        ... 23 common frames omitted

This could indicate that there is a problem with the salt service on the host that prevents the orchestration server from communicating with it properly. As part of the chef run, we mask the salt-minion service to prevent users from interacting with it until the chef process is done. In the case of a previous failure, we never get to the step that we unmask the salt service, thus the above error is displayed.

 
ResolutionRun the following on the target host to unmask and restart the service to allow the upgrade to, properly.

systemctl unmask salt-minion
systemctl restart salt-minion

Then, you may start your upgrade once more.
Be sure to tail the /var/log/netwitness/config-management/chef-solo.log files to watch the upgrade in action.

tailf /var/log/netwitness/config-management/chef-solo.log



 
NotesIf you are still having an issue, please submit a case to RSA Technical Support and quote this KB article. It would also help us if you can attach the /var/log/netwitness/orchestration-server/orchestration-server.log file from your NetWitness Admin Server and your /var/log/netwitness/config-management/chef-solo.log from the host that you are attempting to upgrade.

Attachments

    Outcomes