Decoder Packet Drops and Throttling
I've been getting a lot of questions lately about how to prevent Decoder packet drops when a Spectrum is added to the environment. If the Decoder does not drop when Spectrum is not running, but starts dropping when Spectrum (or even Visualize) requests content from that Decoder, then there is a setting that will help.
This setting is normally off (zero). But if Spectrum is causing drops, it should be enabled (a value of 1). The change takes effect on the next Spectrum request. If a value of 1 does not prevent drops, try increasing the value until it does work, up to the maximum of 100.
Will this setting prevent drops? There's no guarantee it can prevent all drops. If the Decoder packet writes are already pushing the hardware (disk spindles) to the limit, this setting will probably not prevent all packet drops. But it should help. Spectrum naturally wants to read packets from the disk, which will cause the platter heads to seek to other locations on disk. This naturally will cause the next packet write to take longer as the disk head will seek back to the place it originally was to write the next packet. Normally, this kind of churn does not cause writes to back up because of buffering, but if the write speeds are so fast that it doesn't leave a lot of room for the additional seek time, the write thread will start to back up and eventually it will cause packets to drop in the capture thread. Even if it doesn't prevent all drops, it will help.
Will enabling this slow Spectrum? To some degree yes. Throttling does not take place unless it detects the packet write thread is backing up. However, when throttling happens, it slows down packet reads and it will take longer to respond to Spectrum's content requests.
Is there a downside to just setting it all the way to 100? Yes, do not enable it if you are not having issues. Any non-zero value will cause all packet requests to enter a queue to be serviced one at a time, *even* if the write thread is not backed up. So just enabling it will slow down concurrent content requests regardless if packet writes are backing up. The difference between a value of 1 and a value of 100 is higher numbers cause slower packet reads. If a value of 1 suffices to prevent drops, then Spectrum will run somewhat faster than if it is set to 100, assuming throttling is happening.
Does it affect query speeds? No. Queries are normally handled by a Concentrator, not a Decoder. Regardless, queries do not touch the packet db and are not affected by packet throttling.
What SDK calls are affected? There are only 3 SDK calls that are effected by throttling: packets, content and search. Search is only affected if you want to do a regex on a session's packets.
Hope this helps. Feel free to ask any questions or ask for clarification on anything I've written by responding to this post.
- Community Thread
- Forum Thread
- RSA NetWitness
- RSA NetWitness Platform
Great info! As a best practice, you can properly tune your Spectrum too so it doesn't hammer the decoders extracting all the packets. One way you can tune is to simply let Spectrum run for a few hours, then disconnect from the stack while analysis is performed and internal servers are filtered and whitelisted. Then when the Spectrum is readded, the impact will be much less.