000038010 - Unable to start concentrator service throwing core files in the RSA NetWitness Platform

Document created by RSA Customer Support Employee on Oct 11, 2019
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000038010
Applies ToRSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.6.x
Platform: CentOS
O/S Version: 6
IssueConcentrator service is unable to start throwing core files under /var/netwitness/concentrator/metadb.

Tried to move the core files from the metadb folder and start concentrator service but it doesn't work with new core files. 

From /var/log/messages, you may observe the following messages "nwconcentrator process ended" after the index warning, [Index] [warning] Key ipv6.dst is missing from the most recent slice of 4800 …

NwConcentrator[20178]: [Index] [info] Indexes are being initialized
NwConcentrator[20178]: [Index] [info] Checking integrity of slice 4800
NwConcentrator[20178]: [Network] [info] Accepting connection from trusted peer with subject name CN = 2851ae89-7577-41fe-bfba-1cd7346c2cff
NwConcentrator[20178]: [Engine] [audit] User admin (session 421, has logged in
NwConcentrator[20178]: [Index] [warning] Key ipv6.dst is missing from the most recent slice of 4800, but was found in slice 4799.  This is most likely caused by an index configuration change.
init: nwconcentrator main process (20178) killed by ABRT signal
init: nwconcentrator main process ended, respawning
NwConcentrator[7618]: [meta] [warning] There are core files taking up 41.46 GB on the partition /var/netwitness/concentrator/metadb. Please open a support ticket to troubleshoot.

CauseFor some reason, index slice was broken or corrupted due to unexpected service stop or Filesystem issue.
WorkaroundYou can try those steps as a workaround.
  1. Login to the Concentrator device as a root.
  2. Find index slice folder(/var/netwitness/concentrator/index) which is broken. (In this case, /var/netwitness/concentrator/index/managed-values-4800)
  3. Make a copy of this folder as a backup and remove this folder.
  4. Start concentrator service again as concentrator is able to initialize the index slices even with missing keys.

After a period of time for index initializing, you are able to start concentrator service without any issue.

Note: If this workaround doesn't work, please contact customer support with core files and stack trace output.