on Jun 14, 2016Article Content
| Article Number | 000017511 |
| Applies To | RSA Product Set: Security Analytics RSA Product Set: NetWitness NextGen RSA Version/Condition: 9.8, 10.1, 10.2, 10.3.x, 10.4.x, 10.5.x, 10.6.x Platform: CentOS O/S Version: 5, 6 |
| Issue | Fragmented packets/frames are being merged prior to being captured by the decoder process in RSA SA/NetWitness Decoder and Hybrid appliances. Incomplete sessions when attempting to perform session reconstruction. An example of this are incomplete files when attempting to view files found within sessions. Running tcpdump on this decoder reveals frame/packet sizes which are greater then the MTU of network interface (as seen using ifconfig) suggesting that the network interface is merging packets prior to passing these packets to the Linux kernel. So when using tcpdump output or running script in KB article #000026720 to calculate snaplen, much larger then expected values will be seen. |
| Cause | The Generic Receive Offload (GRO) feature is enabled in the network interface card (NIC) driver which can cause incomplete sessions and delays in decoder session processing as the NIC is holding packets rather then immediately forwarding them onto network stack. Note: This feature is enabled by default in many stock Linux NIC drivers e.g. tg3 drivers |
| Resolution | Disable Generic Receive Offload (GRO) in your NIC driver. GRO allows the NIC driver to combine received packets into a single large packet so the kernel requires less CPU to process packets. Unfortunately this negatively impacts session assembly with the Decoder service. Follow these steps to confirm and resolve the problem. Note: It is recommended this feature be turned off in all appliance network interfaces. 1. Use 'ifconfig -a' to identify the network interfaces present on the appliance e.g. em1, em2, em3 & em4 (or eth0, eth1) 2. For each interface, run 'ethtool -k <interface>', where <interface> is the network interface e.g. em3. You will see output similar to this: e.g. # ethtool -k em3 3. If you see "generic-receive-offload: on" in the above output, then this is the feature we would like to disable. Run the following commands for each interface, taking care to update the network interface reference in each command: ethtool -K <interface> gro off Note: Rather than run steps 2 and 3 for each interface identified in step 1, you can simply run the following 2 commands to disable this feature on interfaces starting with e (e.g. eth0/em1) or p (e.g. p1p1): find /sys/class/net \( -name 'e*' -o -name 'p*' \) -exec basename {} \; | xargs -I % ethtool -K % gro offNote: /etc/rc.local is a symbolic link to /etc/rc.d/rc.local SA 10.6.X and Higher Starting from 10.6, /etc/rc.d/rc.local is managed through /etc/puppet/modules/base/files/rc.local on the SA server So instead of updating /etc/rc.local (2nd command above), we need to run the following command on the SA server to add to the puppet recipe sed -ri '/.*ethtool.*/d' /etc/puppet/modules/base/files/rc.local;sed -ri "s/(touch \/var\/lock\/subsys\/local)/\1\n\/bin\/find \/sys\/class\/net \\\( -name \'e*\' -o -name \'p*\' \\\) -exec \/bin\/basename {} \\\; | xargs -I % \/sbin\/ethtool -K % gro off/" /etc/puppet/modules/base/files/rc.local4. Confirm that GRO is now off by re-running 'ethtool -k <interface>': e.g. # ethtool -k em1 5. For decoders, run the procedure in KB article #000026720 to ensure that the service's snaplen setting is now configured correctly. 6. If the problem persists, contact RSA Support and quote this article ID for further assistance. |
| Notes | ethtool syntax: -k (display offload parameter) -K (change offload parameters) After 10.6.2, need to ensure this addition to the puppet recipe occurs BEFORE the following 2 lines: printf "\n\n`date` rc.local SA server node\n\n" >> /var/log/messages Otherwise the disabling of this feature does not occur correctly. |
| Legacy Article ID | a58730 |