000038872 - RSA NetWitness aggregation and collectd fail when not all hosts have been upgraded

Document created by RSA Customer Support Employee on May 27, 2020
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038872
Applies ToRSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Security Analytics Server, concentrator
RSA Version/Condition: 11.3.1.1
 
IssueConcentrator sessions are falling behind, and aggregation fails to start after upgrading only the RSA NetWitness admin server.

Aggregation does not start and collectd keeps failing to send its stats.

/var/log/messages reports:
 


collectd[4335]: NgNativeReader_NwConcentrator-FastUpdate: nwsdk failure: NwOpen returned 0; code 125; error: Connect failed with error "Operation canceled"; thread 4356



On the admin server:
  1. SSH as root
  2. Run:


    orchestration-cli-client --refresh-host -o <salt_node_id>



Where the <salt_node_id> would be found after id: on the concentrator when running:
 

cat /etc/salt/minion


When this fails, you will see in /var/log/netwitness/config-management/chef-solo.log something similar to this:


[2020-05-15T18:23:07+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-05-15T18:23:07+00:00] ERROR: yum_package[rsa-nw-security-cli] (nw-pki::packages line 11) had an error: Chef::Exceptions::Package: Version ["11.3.1.0-1905061831.1.b951110.el7"] of ["rsa-nw-security-cli"] not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)
[2020-05-15T18:23:07+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

CauseUnable to find the proper rsa-nw-security-cli package that the admin server is using to properly connect to the concentrator.
ResolutionUpgrade the concentrator and the rest of the hosts to the upgraded version being used by the admin server.

Attachments

    Outcomes