|Applies To||RSA Product Set: NetWitness Platform|
RSA Product/Service Type: Packet Decoder, Health & Wellness
RSA Version/Condition: 10.6.x, 10.5.x, 11.0,11.1,11.2
O/S Version: EL6, EL7
|Cause||Due to a session that was stuck in Parse thread on Decoder, all available memory pages for Decoder was allocated to Assembler and Parse Thread.|
There is no available memory for Capture thread so that newly ingested packets cannot be captured.
|Resolution||To resolve the issue you must unload or remove the problematic parser that is causing the Decoder to become stuck and then restart the Decoder service.|
How to figure out a session is stuck in Parse Thread
To resolve this issue, you must review statdb or statHist for the stat values listed below. Alsoyou need to understand the pattern under both Normal and Issue status. It can be varied in each customer's environment, incoming traffic amount and loaded parsers. Please take some time to review statdb in Normal status and compare the stats under the issue.
This behavior can be checked with statdb and statHist output.
How to figure out the problematic parser to lead the stuck
- Need to have a core dump of Decoder when it happens. Core Dump Analysis by CE team will tell the problematic parser. However, the dump size is two digit GB, it might be pretty hard to transfer the file.
- If Core Dump is not allowed, list up the recently changed/added parsers. And remove one by one and let Decoder run several days to figure out the bad parser.
- This can rarely happen with RSA Live Parsers. However, mostly it happens with the customer's custom parsers.
- CE team has gencore.sh to collect the dump. Refer the reference link in below.
How to collect/investigate statdb
- Request the customer to transfer Decoder statdb files under /var/netwitness/decoder/statdb on Decoder.
- No need to stop Decoder service to get a closed db file as the opened db file can be queried without any problem.
- Once the statdb file is ready, copy them onto /var/netwitness/decoder/statdb on the Lab Decoder and restart Decoder service.
- Can run statHist or access http://decoderIP:50104/sdk/app/stats, http://decoderIP:50104/sdk/app/packetdrops.
How to collect statHist
- Refer Decoder explore page /sys/statHist function. Can access with NWConsole or Decoder REST web page (http://decoderIP:50104/sys).