000038097 - RSA NetWitness Platform Service Install fails with error for orchestration-cli-client --update-service-id

Document created by RSA Customer Support Employee on Nov 14, 2019
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038097
Applies To
 
  

RSA Product Set: NetWitness Platform
   RSA Product/Service Type: All NetWitness Platform hosts (appliances)
   RSA Version/Condition: Customer who were on 11.0,11.1,11.2 before and are now on 11.3 or beyond.
   Platform: CentOS 7


  
IssueAfter restoring a backup or attempting to rediscover a node using nw-recovery-tool or nwsetup-tui, a service fails to be installed. When reviewing /var/log/netwitness/config-management/chef-solo.log on a host, the following may be seen.

[2019-10-30T19:15:16+00:00] INFO: Processing file[/etc/netwitness/platform/nodeinfo/concentrator/service-id] action create_if_missing (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/serviceinfo.rb line 70)
[2019-10-30T19:15:16+00:00] INFO: Processing ruby_block[check for asg db migration complete] action run (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/serviceinfo.rb line 77)
[2019-10-30T19:15:16+00:00] INFO: Processing execute[updates migrated concentrator serviceid record] action run (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/serviceinfo.rb line 92)
[2019-10-30T19:15:22+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-10-30T19:15:22+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-10-30T19:15:22+00:00] ERROR: Running exception handlers
[2019-10-30T19:15:22+00:00] ERROR: Exception handlers complete
[2019-10-30T19:15:22+00:00] FATAL: Stacktrace dumped to /var/lib/netwitness/config-management/cache/chef-stacktrace.out
[2019-10-30T19:15:22+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-10-30T19:15:22+00:00] ERROR: nw_base_serviceinfo[nw-concentrator] (nw-concentrator::serviceinfo line 10) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[updates migrated concentrator serviceid record] 
(/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/serviceinfo.rb line 92
) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of orchestration-cli-client --update-service-id --old-id 56ccb89be4b0dff52b175fce --new-id ccbe7e6b-b1b8-49cb-97ea-fc353f5ee850 --broker 8d5ff260-268f-4730-b0d5-3de510056702  ----
STDOUT: 2019-10-30 19:15:17.528  INFO 7850 --- [           main] Bootstrap                                : Service logs will be written to /var/log/netwitness/orchestration-client
2019-10-30 19:15:17.534  INFO 7850 --- [           main] Bootstrap                                : Service configuration will be read from /etc/netwitness/orchestration-client
2019-10-30 19:15:17.620  INFO 7850 --- [           main] Bootstrap                                : Starting orchestration-client.5fad6dbd-eadb-429d-91fc-074479831d07 (v0.0.0.0)
2019-10-30 19:15:18.077  INFO 7850 --- [           main] Bootstrap                                : Initialized service cryptography with 4 providers (BSAFE=CRYPTOJ 6.2.2 20161215 0745, FIPS-140=true).
2019-10-30 19:15:18.856  INFO 7850 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : Starting OrchestrationApplication on concentrator1 with PID 7850 (/usr/bin/orchestration-cli-client.jar started by root in /)
2019-10-30 19:15:18.856  INFO 7850 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : The following profiles are active: standard
2019-10-30 19:15:18.942  INFO 7850 --- [           main] Bootstrap                                : Service will accept AMQP requests at broker localhost:5672/rsa/system
2019-10-30 19:15:18.943  INFO 7850 --- [           main] Bootstrap                                : Service will use the deployment security-server
2019-10-30 19:15:20.528  INFO 7850 --- [shake Completed] Security                                 : Accepted new connection with CN=8d5ff260-268f-4730-b0d5-3de510056702,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US from 
8d5ff260-268f-4730-b0d5-3de510056702 using TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
2019-10-30 19:15:21.670  INFO 7850 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : Started OrchestrationApplication in 4.938 seconds (JVM running for 5.432)
2019-10-30 19:15:22.250 ERROR 7850 --- [           main] c.r.n.i.o.c.OrchestrationApplication     : Exception processing request

java.lang.IllegalArgumentException: 56ccb89be4b0dff52b175fce
STDERR: [2019-10-30T19:15:22+00:00] <7845> (ERROR) Failed, aborting...
---- End output of orchestration-cli-client --update-service-id --old-id 56ccb89be4b0dff52b175fce --new-id ccbe7e6b-b1b8-49cb-97ea-fc353f5ee850 --broker 8d5ff260-268f-4730-b0d5-3de510056702  ----
Ran orchestration-cli-client --update-service-id --old-id 56ccb89be4b0dff52b175fce --new-id ccbe7e6b-b1b8-49cb-97ea-fc353f5ee850 --broker 8d5ff260-268f-4730-b0d5-3de510056702  returned 1
[2019-10-30T19:15:23+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)


The message above may vary based on the service type but it will be a Core service. Archiver, Concentrator, Broker, Decoder, Log Decoder, or a Log Collector.
CauseThere exists a file that was created as part of the 11.X to 11.3 upgrade. It was meant as a temp value for this upgrade path to complete a migration step. This file is never deleted afterwards. Below are some examples of where this file could be. It will vary based on the service installed.

/etc/netwitness/platform/nodeinfo/broker/legacy-id
/etc/netwitness/platform/nodeinfo/decoder/legacy-id
/etc/netwitness/platform/nodeinfo/logdecoder/legacy-id
/etc/netwitness/platform/nodeinfo/logcollector/legacy-id
/etc/netwitness/platform/nodeinfo/concentrator/legacy-id
/etc/netwitness/platform/nodeinfo/workbench/legacy-id
/etc/netwitness/platform/nodeinfo/archiver/legacy-id

When you attempt to reinstall a device afterwards, whether that be restoring a backup, rediscover, or as part of a tech refresh, the service attempts to do that 11.X to 11.3 upgrade step again and complains just like you saw in the above error message.
 
ResolutionOnce you have upgraded to 11.3, this file is no longer needed on the device, you can safely move it out and try the install again. You can use the below to find the file should it exists. Once found, move them out. You may have multiple if you have multiple core services on your device.

[root@saserver ~]# find /etc/netwitness/platform -name legacy-id
/etc/netwitness/platform/nodeinfo/broker/legacy-id
[root@saserver ~]# mv /etc/netwitness/platform/nodeinfo/broker/legacy-id /tmp

After that, you can attempt to do your install again while running a tailf on /var/log/netwitness/config-management/chef-solo.log.
 

Attachments

    Outcomes