Our LogDecoder refuse to work today. When we "Start Capture" an Initialization Error occurs.
Failed to start capture: Failed to process message start for /decoder com.rsa.netwitness.carlos.transport.TransportException: terminated
The logs you sent indicated that there are CORE files located in /var/netwitness/logdecoder/metadb. This indicates that the nwlogdecoder service is crashing and generating core dumps and filling up that filesystem.
While performing a data reset did wipe all the data on the appliance, it won't have deleted the core files in that folder.
You will want to open a support case to see if the service is crashing due to a known issue and whether there is a hot fix already released to address the cause of the core dumps.
Is it having database issues?
in 10.4 you are now able to access the services before they are fully started. I see that it has only been up for one minute so it might still be opening the database files depending on the size of your data.
The logs generated by Log Decoder need to be checked to see if there are any warnings or errors.
grep "warning\|failure" /var/log/messages
tail -f /var/log/messages "warning\|failure"
You can do the same thing using SA-UI as well by going to Log Decoder > Logs > Historical and then filtering the logs once by "WARN" and another time by "ERROR".
Usually, when an "Initialization Error" happens in Log Decoder, the reason behind it can be explained by seeing the warning and failure logs that were generated by Log Decoder while the Log Decoder was starting.
There were really appropriate entries in the syslog. So I conducted a data reset, now it works.
Here are the appropriated syslog entries.
Jan 16 16:30:11 logdecoder nw: [meta] [warning] There are core files taking up 2.18 GB on the partition /var/netwitness/logdecoder/metadb. Please open a support ticket to troubleshoot.
Jan 16 16:30:11 logdecoder nw: [Database] [failure] One of the databases (session or meta) is missing data. To correct this error, first make sure the databases are configured properly and the drives are correctly mounted. If this is the case and there's very little data in one of the databases, perform a data reset to correct.
Jan 16 16:30:11 logdecoder nw: [Engine] [warning] Module logdecoder failed to load: One of the databases (session or meta) is missing data. To correct this error, first make sure the databases are configured properly and the drives are correctly mounted. If this is the case and there's very little data in one of the databases, perform a data reset to correct.
Jan 16 16:30:11 logdecoder nw: [Engine] [warning] Module logdecoder failed to load: Diagnostic information: Throw in function bool nw::AssemblerDatabase::trimCorruption(bool)Dynamic exception type: N5boost16exception_detail10clone_implIN2nw9ExceptionEEEstd::exception::what: One of the databases (session or meta) is missing data. To correct this error, first make sure the databases are configured properly and the drives are correctly mounted. If this is the case and there's very little data in one of the databases, perform a data reset to correct.[PN5boost16errinfo_at_line_E] = 1366
Jan 16 16:30:11 logdecoder nw: [ObjectStore] [warning] The file '/var/netwitness/logdecoder/statdb/stats-000000096.statsdb' was not properly closed.
Did you see red color "Initialization error" ? I think missing database path caused the issue. How you reset your data? Did you reset config as well?
As you can see in my post from 16.01.2015 7:39, there was the red "Initialization error".
I've done the reset for all databases
reset data=1 index=1 log=1 stats=1 config=1
After that I stopped the NWLogDecoder service via shell command and cleaned the directories:
So that all core.* files are purged.
After all I've done a restart of the NWLogDecoder service.
Do you have a blurb "How to repair a corrupt NetWitness Core Database"?
You can use the NwConsole tool to check databases. There is a dbcheck function that will check various databases.
For more details see KB article 26605 at https://rsaportal.force.com/customer/articles/How_To/a59828-How-to-manually-run-dbcheck-on-an-RSA-NetWitness-appliance?popup=false&navBack=H4sIAAAAAAAAAIuuVipVslLSTy4tLsnPTS3Sjy_N1M_Oyy_PSU1JT9UHcrxhHA-gvH1xamJRcoatkZmZgamSjlIxUC-KAqBYNlCsIDE9NSSzJCdVqTYWACkGVlljAAAA
Thanks a lot. I will try it.
"NextGen 9.5" was not realy my focus at SCOL search ;-)
It is possible that the KB article needs an update. But this tool is available on Security Analytics core appliances as well as NextGen core appliances.
Hi guys, I had thasame problem and I fixed it the same way, running a data reset, but my folder /var/netwitness/logdecoder/metadb is increasing very fast, about 25GB a day.
I am collecting data from 80 Windows Server via winrm.
Is normal this folder to increase this way?
That seems to be normal at first glance. Depending on the load on the Windows machines.
What about the size of /var/netwitness/logdecoder/packetdb - nearly the same?
How many entries in table-map.xml and table-map-custom.xml with flags="None".
Have you used the Windows Event Source "Channel selection feature" for filtering unnecessary events like System^(101|201), Security(4672), Application^(211|300) or do you system collect all the windows channels?
Think about rising of /Database/Config -> meta.compression.level ...
Thanks for your reply Davme.
The size of /var/netwitness/logdecoder/packetdb
is not as big as /var/netwitness/logdecoder/metadb, I will check which is the size exactly.
I am not using "Channel selection feature". Is there a recommendation of Event IDs to exclude?
After run a data reset I still can see the logs in Investigation tab, so what kind of data is deleted using this command? Would be the raw log?
Can you really see the raw logs in investigation after a decoder data reset,or did you only see the indexed metadata at the concentrator/broker level? Security Analytics is a distributed system, you have to clear data in several places decoder/concentrator/broker.
There is no recommendation of Event IDs to exclude, it depends on your experience. Useful Links for it:
Yes, I see just indexed metada at Concentrator.
My question is, in LogDecoder we have huge logs because LogDecoder stores the raw logs, right?
Is possible to schedule a data reset to run once a week for example?
Most of our requests are to store more data rather than purging existing data more quickly. I am not sure why you are interested in manually purging the raw logs from the system.
Your Log Decoder should be rolling out all data (session, meta, and packet) when the allocated filesystems reach 95% capacity. If it is not, then something may not be configured properly in the appliance.
There are several things you can do to purge data earlier than the default retention period but I'd like to make sure I understand "what" and "why" before I start making recommendations. But it is possible to script a "time-roll" or a "size-roll" or even data reset and execute that via a cron job. These are just kind of unusual requests.
A sample "size-roll" operation is shown in KB article 17193 found at https://rsaportal.force.com/customer/articles/Break_Fix/a64807-Log-Decoder-partitions-in-RSA-Security-Analytics-hybrid-o…
Not sure if it would help, but it is also possible to enable metadb compression on Log Decoders and Concentrators. See KB 27795 at https://rsaportal.force.com/customer/articles/How_To/a65372-Q-A-on-Enabling-Meta-Compression-in-RSA-Security-Analytics-1…
If it works I wont need to use a data reset.
If you run the latest technical support script (nwtech.sh) and submit the results for evaluation we can let you know if we find any configuration problems that might prevent data from automatically rolling out.
The latest script may be found at https://rsaportal.force.com/customer/articles/How_To/a59741-RSA-NetWitness-and-Security-Analytics-Tech-Support-Data-Gath….
Hope that helps.
I am going to wait a week and check if it the logs are being automatically rolling out, if not I will sent you this technical support script.
Retrieving data ...