000032658 - RSA Security Analytics Log Decoder Hybrid continuously crashing and creating core dump files

Document created by RSA Customer Support Employee on Jun 30, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000032658
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics UI
RSA Version/Condition:
Platform: CentOS
Platform (Other): null
O/S Version: 6
Product Name: SA-HYBRID-L
Product Description: SecAnlytcs Hybrd Dplymnt Logs
IssueThe log decoder service will not remain started up, it keeps looping to start and then crashes and creates core dump files. This results in filling up to 100% of the filesystem /var/netwitness/logdecoder.  Even if you try removing the core files and restarting, the log decoder service will not start up, it just keeps core dumping.

Note the error below repeating in /var/log/messages while the log decoder service tries to start up, until core dump files fill up the filesystem:

Feb 29 22:21:57 SAlogs nw[9766]: [Engine] [warning] Module logdecoder failed to load: Diagnostic information: Throw in function static void nw::serialization::Serializer<A, std::basic_string<char> >::load(A&, std::string&, unsigned int) [with A = nw::InputArchive; std::string = std::basic_string<char>]Dynamic exception type: N5boost16exception_detail10clone_implIN2nw18SerializationErrorEEEstd::exception::what: Invalid deserialized string length 1986358889. Maximum size exceeded.[PN5boost16errinfo_at_line_E] = 507

CauseLog Decoder crashes due to statdb corruption. It keeps looping over and over when you start the nwlogdecoder service.
ResolutionIssue currently being investigated by engineering.
WorkaroundAs a workaround, delete the core files, move or rename the /var/netwitness/statsdb file and then restart the nwlogdecoder service.
You may want to backup at least one recent core file just in case we need to further investigate the issue with engineering.