10.6 Virtual Decoder dropping more packets than 10.3 Physical Decoder
I've only had our 10.6 Virtual Decoder online for about 24 hours, and it has dropped more packets than what our 10.3 physical decoder over it's entire life span.
I'm a little concerned, so I was wondering if there are anyways to determine if this is something normal with 10.6 (ie. filtering etc) or should I be investigating more to find a cause to the problem.
Some stats for reference.
|10.6 Virtual Decoder||10.3 Physical Decoder|
|Max Capture Rate||846 MbPS||987 MbPS|
|1.6 Billion Packets||8.1 Billion Packets|
|172.5 Million Packets (0% loss)|
10,235 Packets (0% loss)
1.6 Billion Packets
498.6 Billion Packets
What hardware are the hardware specs for the 10.3 physical Decoder?
- Chassis Type (Dell SuperMicro)?
- Storage type (DAC - 3Gbps or 6Gbps) (VNX?)
What are your specs for the Virtual Decoder?
What type of storage is being used for the following filesystems?:
And what is the rated speed of the I/O channel to the storage?
What content is deployed to the virtual decoder?
- Parsers (built-in/flex/lua)
- Application Rules (any regex rules? are you running old IR rules?)
- Correlation Rules (Don't belong on a decoder, remove them)
What are the settings for the following parameters on the Virtual Decoder
I'll try to answer your questions as best as I can.
The physical specs on the decoder, all I know is that it's series 3 hardware with 16GB RAM
Virtual has 8 vCPU and 40GB RAM
Storage for /var/netwitness/decoder/index are VMDK's on local SSD storage
Everything else the VMDKs are on Fibre Channel NetApp storage at 8Gb/s
Parsers are all flex, and we've deployed a lot of them (if there is an easy way to get a text based list I can provide that)
Feeds we have the nw feeds, alert feeds and spectrum feeds
app rules are all default
no correlation rules
assembler.session.size <can't find it>
assembler.size.max 32 MB
from the Explore view on the virtual decoder
right click on the "decoder" folder and select properties
in the Properties frame, select "reconfig" from the drop down and hit "Send"
copy the Response Output window and post back here.
ok now do the same thing, but in the parameters box, enter "update=1" (no quotes)
that will "Set" the parameters to a best guess starting point for the resources available to the decoder, right now several of them are too low, and since the virtualized decoder does run slightly slower than the hardware based one, you are dropping more packets.
after you run that command, restart the decoder service for them to take affect.
Next thing to do is dump the flex parsers and use the lua equivalents, and make sure you are not running any deprecated feeds (you probably are if you still have flex parsers deployed)
It sounds like you really need to have RSA do a Tuning and Optimization engagement as part of your virtualization project.
On your stats page for the decoder, configure these timeline charts, when things are working correctly, the top 2 charts should never reach zero, and the bottom two should only have occasional rises
Packet Capture Queue (pool.packet.pages) = packet Capture buffer, if it reaches 0 you are dropping packets.
Assembler Packet Pages (assembler.packet.pages) = Session Assembly buffer
Packet Assembler Queue (pool.packet.assembler) - active sessions being assembled
Packet Write Queue (pool.packet.write) - Raw Packet Write buffer for moving to disk.
If the write queue gets backed up, either the IOPS performance of the storage is slow, or too many reads are occurring and causing read/write contention, check System sessions page for active queries for pulling data from decoder
If the assembler queues get backed up, may be too many hanging sessions (not properly ended) being captured and waiting on 60 second session timeout or the parsing of the sessions is taking too long (too many parsers/feeds/rules deployed for the capacity of the device
Thanks John, that's good information with the stats, I'll certainly configure those.
I'll make those changes you suggest when I'm back online.
The is the first Virtual appliance I've migrated. I'm pretty sure I did the reconfig update=1 before but I'll do it again just to be sure.
I performed the update=1 command and the numbers didn't change. They are the same as before.
Sorry, I misspoke, we only have lua parsers and up to date feeds installed (I only installed it about a month ago)