|Applies To||RSA NetWitness NextGen|
RSA Netwitness NextGen 9.6
RSA Netwitness NextGen 9.7
RSA NetWitness Investigator
RSA NetWitness Administrator
|Issue||RSA NetWitness NextGen broker, decoder or concentrator service is unresponsive.|
NextGen Broker, Decoder or Concentrator services become unresponsive and cannot be connected to via Administrator or Investigator.
There are no plans for further updates to NextGen version 9.5, so this procedure applies only to versions 9.6 and higher. For this reason, core dump files for 9.5 will no longer be analyzed, as any problems uncovered in a 9.5 core dump will not apply to 9.6 and higher versions.
|Resolution||Steps to Take: |
1. DO NOT restart the service yet.
2. Run the data gathering script on the device
*3. THIS STEP IS CRITICAL - stop nwcrashreporter, or the core dump file will get deleted by the nwcrashreporter process
SERIES 3 APPLIANCES: monit stop nwcrashreporter
SERIES 2 APPLIANCES (running Fedora): stop nwcrashreporter
4. Via an ssh session, force a core dump of the service by running the appropriate command for the service:
pkill -11 NwDecoder
pkill -11 NwConcentrator
pkill -11 NwBroker
pkill -11 NwLogDecoder
5. The core dump file will be located in either /var/netwitness/broker/core.X (for Broker), /var/netwitness/decoder/packetdb/core.* (for Decoder), /var/netwitness/concentrator/metadb/core.* (for Concentrator), or /var/netwitness/logdecoder/metadb/core.* (for Panorama), where X is a 'random' number. It should be the most recent file in the directory, which you can find with 'ls -ltr core* | tail -1'. Compress the core dump file with 'gzip core.X', where X is a random number . The file will now be called 'core.X.gz'.
6. Move the compressed core dump file to another directory (preferably /root, if there is enough free space) before restarting the service, as the file will soon get deleted.
7. Open a case with NetWitness Support and send the core file so it can be analyzed for the root cause of the hang. Core files will be too large to attach to the case, so please send the file to Support via: https://netwitness.cutesendit.com. Core files may even be too large to send via cutesendit. In this case, Support can arrange for an FTP account to be created for the upload.
8. Restart nwcrashreporter
SERIES 3 APPLIANCES: monit start nwcrashreporter
SERIES 2 APPLIANCES (running Fedora): start nwcrashreporter
9. The hung service should have automatically died after the core dump was invoked, and will be restarted automatically by monit (Series 3) or init (Series 2).
|Legacy Article ID||a58783|