000032658 - RSA NetWitness Logs & Network Log Decoder Hybrid continuously crashing and creating core dump files

Document created by RSA Customer Support Employee on Jun 30, 2016Last modified by RSA Customer Support on Apr 18, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000032658
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Logs & Network UI
RSA Version/Condition: 10.6.x
Platform: CentOS
O/S Version: 6
Product Name: SA-HYBRID-L
Product Description: SecAnlytcs Hybrd Dplymnt Logs
IssueThe NetWitness Log Decoder service will not remain started, it keeps looping to start and then crashes and creates core dump files.
This results in filling up to 100% of the filesystem /var/netwitness/logdecoder.  Even if you try removing the core files and restarting, the log decoder service will not start up, it just keeps core dumping.

Note the error below repeating in /var/log/messages file while the Log Decoder service tries to start up, until core dump files fill up the filesystem:

Mar 10 11:31:18 axinsadec3 nw[8057]: [Engine] [warning] Module logdecoder failed to load: Invalid deserialized string length 1868955648. Maximum size exceeded.
Mar 10 11:31:18 axinsadec3 nw[8057]: [Engine] [warning] Module logdecoder failed to load: Diagnostic information: Throw in function static void nw::serialization::Serializer<A, std::basic_string<char> >::load(A&, std::string&, unsigned int) [with A = nw::InputArchive; std::string = std::basic_string<char>]Dynamic exception type: N5boost16exception_detail10clone_implIN2nw18SerializationErrorEEEstd::exception::what: Invalid deserialized string length 1868955648. Maximum size exceeded.[PN5boost16errinfo_at_line_E] = 507
Mar 10 11:31:18 axinsadec3 nw[8057]: [stats] [info] Found 7 files (399.61 MB) when loading /var/netwitness/logdecoder/statdb of max size 1 GB
Mar 10 11:31:18 axinsadec3 nw[8057]: [ObjectStoreIndex] [warning] Invalid index /var/netwitness/logdecoder/statdb/stats-000000028.statsdbindex. Last object position 16740126 exceeds store size 16740126. Regenerating index...
CauseIn this example, the Log Decoder crashes were due to a statdb file corruption. It keeps looping over and over whilst trying to start the nwlogdecoder service.
ResolutionEnsure the latest NetWitness patch release has been installed.

If NetWitness is running with the latest version, then collect a sample core file from when the issue started to occur, and contact RSA Support to open a case with RSA Engineering to investigate the cause, then proceed to the workaround.
WorkaroundAs a workaround, delete the core files.
If the core file creation is due to a corrupt statdb file, then move or rename the statdb file mentioned in the error message, and then restart the nwlogdecoder service.

start nwlogdecoder

Backup at least one recent core file in case further an investigation of the issue is needed with RSA Engineering.