000017511 - Fragmented packets/frames are being merged prior to capture in RSA NetWitness Decoder and Hybrid appliances

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000017511
Applies ToRSA Product Set: Security Analytics
RSA Product Set:  NetWitness NextGen
RSA Version/Condition: 9.8, 10.1, 10.2, 10.3.x, 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: 5, 6
 
IssueFragmented packets/frames are being merged prior to being captured by the decoder process in RSA SA/NetWitness Decoder and Hybrid appliances.
Incomplete sessions when attempting to perform session reconstruction. An example of this are incomplete files when attempting to view files found within sessions.
Running tcpdump on this decoder reveals frame/packet sizes which are greater then the MTU of network interface (as seen using ifconfig) suggesting that the network interface is merging packets prior to passing these packets to the Linux kernel.
So when using tcpdump output or running script in KB article #000026720 to calculate snaplen, much larger then expected values will be seen.
 
CauseThe Generic Receive Offload (GRO) feature is enabled in the network interface card (NIC) driver which can cause incomplete sessions and delays in decoder session processing as the NIC is holding packets rather then immediately forwarding them onto network stack. Note: This feature is enabled by default in many stock Linux NIC drivers e.g. tg3 drivers
 
ResolutionDisable Generic Receive Offload (GRO) in your NIC driver.  GRO allows the NIC driver to combine received packets into a single large packet so the kernel requires less CPU to process packets. Unfortunately this negatively impacts session assembly with the Decoder service.  Follow these steps to confirm and resolve the problem.
Note: It is recommended this feature be turned off in all appliance network interfaces.
1. Use 'ifconfig -a' to identify the network interfaces present on the appliance e.g. em1, em2, em3 & em4 (or eth0, eth1)
2.  For each interface, run 'ethtool -k <interface>', where <interface> is the network interface e.g. em3.  You will see output similar to this:
e.g.
# ethtool -k em3
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: on


3.  If you see "generic-receive-offload: on" in the above output, then this is the feature we would like to disable.
Run the following commands for each interface, taking care to update the network interface reference in each command:
ethtool -K <interface> gro off
echo "/sbin/ethtool -K <interface> gro off"  >> /etc/rc.local

Note: Rather than run steps 2 and 3 for each interface identified in step 1, you can simply run the following 2 commands to disable this feature on interfaces starting with e (e.g. eth0/em1) or p (e.g. p1p1):
find /sys/class/net \( -name 'e*' -o -name 'p*' \) -exec basename {} \; | xargs -I % ethtool -K % gro off
echo "find /sys/class/net \( -name 'e*' -o -name 'p*' \) -exec basename {} \; | xargs -I % ethtool -K % gro off" >> /etc/rc.local

Note: /etc/rc.local is a symbolic link to /etc/rc.d/rc.local
SA 10.6.X and Higher
Starting from 10.6, /etc/rc.d/rc.local is managed through /etc/puppet/modules/base/files/rc.local on the SA server
So instead of updating /etc/rc.local (2nd command above), we need to run the following command on the SA server to add to the puppet recipe
sed -ri '/.*ethtool.*/d' /etc/puppet/modules/base/files/rc.local;sed -ri "s/(touch \/var\/lock\/subsys\/local)/\1\n\/bin\/find \/sys\/class\/net \\\( -name \'e*\' -o -name \'p*\' \\\) -exec \/bin\/basename {} \\\; | xargs -I % \/sbin\/ethtool -K % gro off/" /etc/puppet/modules/base/files/rc.local

4.  Confirm that GRO is now off by re-running 'ethtool -k <interface>':
e.g.
# ethtool -k em1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

5.  For decoders, run the procedure in KB article #000026720 to ensure that the service's snaplen setting is now configured correctly.
6.  If the problem persists, contact RSA Support and quote this article ID for further assistance.
 
Notesethtool syntax:
-k (display offload parameter)
-K (change offload parameters)
After 10.6.2, need to ensure this addition to the puppet recipe occurs BEFORE the following 2 lines:
printf "\n\n`date`  rc.local  SA server node\n\n" >> /var/log/messages
source /etc/init.d/sa_addrd


Otherwise the disabling of this feature does not occur correctly.
Legacy Article IDa58730

Attachments

    Outcomes