| Article Number | 000017511 |
| Applies To | RSA Product Set: Security Analytics
RSA Product Set: NetWitness NextGen
RSA Version/Condition: 9.8, 10.1, 10.2, 10.3.x, 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: 5, 6
|
| Issue | Fragmented packets/frames are being merged prior to being captured by the decoder process in RSA SA/NetWitness Decoder and Hybrid appliances.
Incomplete sessions when attempting to perform session reconstruction. An example of this are incomplete files when attempting to view files found within sessions.
Running tcpdump on this decoder reveals frame/packet sizes which are greater than the MTU of network interface (as seen using ifconfig) suggesting that the network interface is merging packets prior to passing these packets to the Linux kernel.
So when using tcpdump output or running script in KB article #000026720 to calculate snaplen, much larger than expected values will be seen.
|
| Cause | The Generic Receive Offload (GRO) feature is enabled in the network interface card (NIC) driver which can cause incomplete sessions and delays in decoder session processing as the NIC is holding packets rather than immediately forwarding them onto network stack. Note: This feature is enabled by default in many stock Linux NIC drivers e.g. tg3 drivers
|
| Resolution | Disable Generic Receive Offload (GRO) in your NIC driver. GRO allows the NIC driver to combine received packets into a single large packet so the kernel requires less CPU to process packets. Unfortunately, this negatively impacts session assembly with the Decoder service. Follow these steps to confirm and resolve the problem.
Note: It is recommended this feature be turned off in all appliance network interfaces.
- Use 'ifconfig -a' to identify the network interfaces present on the appliance e.g. em1, em2, em3 & em4 (or eth0, eth1)
- For each interface, run 'ethtool -k <interface>', where <interface> is the network interface e.g. em3. You will see output similar to this:
e.g.
# ethtool -k em3
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: on
- If you see "generic-receive-offload: on" in the above output, then this is the feature we would like to disable.
Run the following commands for each interface, taking care to update the network interface reference in each command:
ethtool -K <interface> gro off
echo "/sbin/ethtool -K <interface> gro off" >> /etc/rc.local
Note: Rather than run steps 2 and 3 for each interface identified in step 1, you can simply run the following 2 commands to disable this feature on interfaces starting with e (e.g. eth0/em1) or p (e.g. p1p1):
find /sys/class/net \( -name 'e*' -o -name 'p*' \) -exec basename {} \; | xargs -I % ethtool -K % gro off
echo "find /sys/class/net \( -name 'e*' -o -name 'p*' \) -exec basename {} \; | xargs -I % ethtool -K % gro off" >> /etc/rc.local
Note: /etc/rc.local is a symbolic link to /etc/rc.d/rc.local
SA 10.6.X and Higher
Starting from 10.6, /etc/rc.d/rc.local is managed through /etc/puppet/modules/base/files/rc.local on the SA server
So instead of updating /etc/rc.local (2nd command above), we need to run the following command on the SA server to add to the puppet recipe
sed -ri '/.*ethtool.*/d' /etc/puppet/modules/base/files/rc.local;sed -ri "s/(touch \/var\/lock\/subsys\/local)/\1\n\/bin\/find \/sys\/class\/net \\\( -name \'e*\' -o -name \'p*\' \\\) -exec \/bin\/basename {} \\\; | xargs -I % \/sbin\/ethtool -K % gro off/" /etc/puppet/modules/base/files/rc.local
- Confirm that GRO is now off by re-running 'ethtool -k <interface>':
e.g.
# ethtool -k em1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off
- For decoders, run the procedure in KB article #000026720 to ensure that the service's snaplen setting is now configured correctly.
- If the problem persists, contact RSA Support and quote this article ID for further assistance.
|
| Notes | ethtool syntax:
-k (display offload parameter)
-K (change offload parameters)
After 10.6.2, need to ensure this addition to the puppet recipe occurs BEFORE the following 2 lines:
printf "\n\n`date` rc.local SA server node\n\n" >> /var/log/messages
source /etc/init.d/sa_addrd
Otherwise the disabling of this feature does not occur correctly. |
| Legacy Article ID | a58730 |
on Jun 14, 2016