000011523 - RSA NetWitness NextGen broker  decoder or concentrator service is unresponsive

Document created by RSA Customer Support Employee on Jan 11, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000011523
Applies ToRSA NetWitness NextGen
RSA Netwitness NextGen 9.6
RSA Netwitness NextGen 9.7
RSA NetWitness Investigator
RSA NetWitness Administrator
IssueRSA NetWitness NextGen broker, decoder or concentrator service is unresponsive.

NextGen Broker, Decoder or Concentrator services become unresponsive and cannot be connected to via Administrator or Investigator.

There are no plans for further updates to NextGen version 9.5, so this procedure applies only to versions 9.6 and higher.  For this reason, core dump files for 9.5 will no longer be analyzed, as any problems uncovered in a 9.5 core dump will not apply to 9.6 and higher versions.

ResolutionSteps to Take

1.  DO NOT restart the service yet.

2.  Run the data gathering script on the device

*3.  THIS STEP IS CRITICAL  - stop nwcrashreporter, or the core dump file will get deleted by the nwcrashreporter process
SERIES 3 APPLIANCES: monit stop nwcrashreporter
SERIES 2 APPLIANCES (running Fedora): stop nwcrashreporter

  Via an ssh session, force a core dump of the service by running the appropriate command for the service:
pkill -11 NwDecoder
pkill -11 NwConcentrator
pkill -11 NwBroker
pkill -11 NwLogDecoder

5.  The core dump file will be located in either /var/netwitness/broker/core.X (for Broker), /var/netwitness/decoder/packetdb/core.* (for Decoder), /var/netwitness/concentrator/metadb/core.* (for Concentrator), or /var/netwitness/logdecoder/metadb/core.* (for Panorama), where X is a 'random' number.  It should be the most recent file in the directory, which you can find with 'ls -ltr core* | tail -1'.  Compress the core dump file with 'gzip core.X', where X is a random number .  The file will now be called 'core.X.gz'.

  Move the compressed core dump file to another directory (preferably /root, if there is enough free space) before restarting the service, as the file will soon get deleted.

  Open a case with NetWitness Support and send the core file so it can be analyzed for the root cause of the hang.  Core files will be too large to attach to the case, so please send the file to Support via: https://netwitness.cutesendit.com.  Core files may even be too large to send via cutesendit.  In this case, Support can arrange for an FTP account to be created for the upload.

8.  Restart nwcrashreporter
SERIES 3 APPLIANCES: monit start nwcrashreporter
SERIES 2 APPLIANCES (running Fedora): start nwcrashreporter

  The hung service should have automatically died after the core dump was invoked, and will be restarted automatically by monit (Series 3) or init (Series 2).
Legacy Article IDa58783