10.6 Virtual Decoder dropping more packets than 10.3 Physical Decoder

[JeremyKerwin]

I've only had our 10.6 Virtual Decoder online for about 24 hours, and it has dropped more packets than what our 10.3 physical decoder over it's entire life span.

I'm a little concerned, so I was wondering if there are anyways to determine if this is something normal with 10.6 (ie. filtering etc) or should I be investigating more to find a cause to the problem.

Some stats for reference.

	10.6 Virtual Decoder	10.3 Physical Decoder
Max Capture Rate	846 MbPS	987 MbPS
Total Captured	1.6 Billion Packets	8.1 Billion Packets
Total Dropped	172.5 Million Packets (0% loss)	10,235 Packets (0% loss)
Total Packets	1.6 Billion Packets	498.6 Billion Packets

Thanks.

[JohnSnider]

What hardware are the hardware specs for the 10.3 physical Decoder?

• Memory
• CPU's
• Chassis Type (Dell SuperMicro)?
• Storage type (DAC - 3Gbps or 6Gbps) (VNX?)

What are your specs for the Virtual Decoder?

• Memory
• CPU's

What type of storage is being used for the following filesystems?:

• /var/netwitness/decoder/index
• /var/netwitness/decoder/metadb
• /var/netwitness/decoder/sessiondb
• /var/netwitness/decoder/packetdb

And what is the rated speed of the I/O channel to the storage?

What content is deployed to the virtual decoder?

• Parsers (built-in/flex/lua)
• Feeds
• Application Rules (any regex rules? are you running old IR rules?)
• Correlation Rules (Don't belong on a decoder, remove them)

What are the settings for the following parameters on the Virtual Decoder

• /decoder/config
  • assembler.session.pool
  • assembler.session.size
  • capture.buffer.size
  • pool.packet.pages
  • pool.session.pages

[JeremyKerwin]

Hi John,

I'll try to answer your questions as best as I can.

The physical specs on the decoder, all I know is that it's series 3 hardware with 16GB RAM

Virtual has 8 vCPU and 40GB RAM

Storage for /var/netwitness/decoder/index are VMDK's on local SSD storage

Everything else the VMDKs are on Fibre Channel NetApp storage at 8Gb/s

Parsers are all flex, and we've deployed a lot of them (if there is an easy way to get a text based list I can provide that)

Feeds we have the nw feeds, alert feeds and spectrum feeds

app rules are all default

no correlation rules

assembler.session.pool 350000

assembler.session.size <can't find it>

assembler.size.max 32 MB

assembler.size.min 0

pool.packet.pages 30000

pool.session.pages 10000

[JohnSnider]

from the Explore view on the virtual decoder

right click on the "decoder" folder and select properties

in the Properties frame, select "reconfig" from the drop down and hit "Send"

copy the Response Output window and post back here.

[JeremyKerwin]

/decoder/config/pool.session.page.size:8 KB
/decoder/config/pool.packet.pages:50000
/decoder/config/parse.threads:0
/decoder/config/assembler.session.pool:250000
/decoder/config/numa.bindings:
/decoder/config/capture.buffer.size:64 MB
/decoder/config/pool.packet.page.size:32 KB
/decoder/config/pool.session.pages:25000

[JohnSnider]

ok now do the same thing, but in the parameters box, enter "update=1" (no quotes)

that will "Set" the parameters to a best guess starting point for the resources available to the decoder, right now several of them are too low, and since the virtualized decoder does run slightly slower than the hardware based one, you are dropping more packets.

after you run that command, restart the decoder service for them to take affect.

Next thing to do is dump the flex parsers and use the lua equivalents, and make sure you are not running any deprecated feeds (you probably are if you still have flex parsers deployed)

It sounds like you really need to have RSA do a Tuning and Optimization engagement as part of your virtualization project.

[JohnSnider]

On your stats page for the decoder, configure these timeline charts, when things are working correctly, the top 2 charts should never reach zero, and the bottom two should only have occasional rises

Packet Capture Queue (pool.packet.pages) = packet Capture buffer, if it reaches 0 you are dropping packets.

Assembler Packet Pages (assembler.packet.pages) = Session Assembly buffer

Packet Assembler Queue (pool.packet.assembler) - active sessions being assembled

Packet Write Queue (pool.packet.write) - Raw Packet Write buffer for moving to disk.

If the write queue gets backed up, either the IOPS performance of the storage is slow, or too many reads are occurring and causing read/write contention, check System sessions page for active queries for pulling data from decoder

If the assembler queues get backed up, may be too many hanging sessions (not properly ended) being captured and waiting on 60 second session timeout or the parsing of the sessions is taking too long (too many parsers/feeds/rules deployed for the capacity of the device

[JeremyKerwin]

Thanks John, that's good information with the stats, I'll certainly configure those.

I'll make those changes you suggest when I'm back online.

The is the first Virtual appliance I've migrated. I'm pretty sure I did the reconfig update=1 before but I'll do it again just to be sure.

[JeremyKerwin]

Hi John,

I performed the update=1 command and the numbers didn't change. They are the same as before.

Sorry, I misspoke, we only have lua parsers and up to date feeds installed (I only installed it about a month ago)

[JeremyKerwin]

I configured those graphs and it appears that it looks all ok. based on your screen shot above.