(Optional) Data Plane Development Kit Packet Capture

The NetWitness Platform Decoder supports capturing packets through the Data Plane Development Kit, also known as DPDK. DPDK is a set of libraries and drivers that support applications that work on directly on packets. DPDK's web site, www.dpdk.org, provides more information.

Advantages

The Decoder can use the DPDK to ingest packets from Ethernet interfaces. It is an alternative to existing packet capture options built into the Decoder. DPDK offers these advantages:

  • It is entirely open source, with no binary-only driver components, unlike PF_RING.
  • It is directly supported by multiple vendors, including our main vendor, Intel.
  • Since the kernel-level interface is in the mainline Linux kernel, there are no separate driver packages to compile and distribute.
  • DPDK devices are completely separate from the Linux network stack, which avoids unintentional interaction between Linux networking and packet capture.
  • DPDK is capable of capturing at greater than 10G, unlike the packet_mmap.

How it Works

DPDK completely detaches the Ethernet interface from the Linux kernel. It does this by doing the following:

  • It uses a Linux User I/O driver mechanism to expose the raw PCI-Express device to normal Linux programs. This means that it can take the raw memory space mapped to the PCI device and expose it, so that a normal Linux program can directly manipulate the hardware. There are a couple of drivers built into the Linux mainline kernel that can accomplish this. Decoder is designed to use vfio_pci_generic.
  • Note: Only one driver can be attached to an Ethernet interface at a time. When vfio_pci_generic is attached to an interface, the Linux network driver for that interface is detached.

  • The DPDK provides its own user-space implementation of the hardware drivers for a wide variety of Ethernet cards. When a DPDK application like Decoder is loaded, it can attach directly to the PCI device and manipulate it using DPDK-specific library functions. When a device is attached to DPDK, the Decoder can see it and automatically sets the device up with the DPDK Capture Device option.

Migrate PF_RING Devices to DPDK

For configurations using multiple adapter packet capture (available in 11.5 and later), you must use the manual procedure outlined in Manually Move a Capture Interface to DPDK . For example, the migration (dpdk migrate) does not support a Decoder when capture is configured with: capture.interface=PFRINGZC,em3; PFRINGZC,em4;packet_mmap,em2.

For configurations prior to 11.5, the procedure that uses the devices parameter to capture from multiple PF_RING-supported interfaces (for example, ixgbe and i40e) will be converted to using DPDK, along with the multiple adapter packet capture, so that one capture thread is used per interface.

For example, the migration will work if configured as follows:

  • For multi-interface capture: capture.interface=PFRINGZC,em3 along with capture.device.params=device=zc:em3,zc:em4
  • For single interface capture: capture.interface=PFRINGZC,em3

Note: dpdk migrate only functions with PFRINGZC, not mmap interfaces.

If you are using PF_RING to capture packets on your Decoder, you can use the Decoder's dpdk migrate command to migrate your existing PF_RING capture devices to DPDK. This command uses your currently-selected capture device configuration to determine which network interfaces are moved to DPDK control.

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services and select a Decoder.
  2. Click netwitness_ic-actns.png > View > Explore, right-click decoder and select Properties.
  3. From the command drop-down list, select the dpdk command, type the parameter migrate, and click Send.
  4. In the command output window, the changes that will be made on the Decoder to perform the migration are displayed. If everything looks correct for the migration, add the parameter commit=1 to the command after migrate to actually make the changes on the Decoder.

    netwitness_116-dpdk.png
  5. At the reboot prompt, reboot the host to make the driver loading changes.
  6. (Optional) Run the /decoder/devices/<devicename>/msg=prune command to remove configuration options for devices and interfaces that no longer exist after the migration.

    Note: The prune command removes only the undetected interface folders. Also, it ignores the interfaces, stats, nwimport, and flow_events folders available in /decoder/devices/.

Manually Move a Capture Interface to DPDK

  1. Run the new host service command /appliance/nicToDp, that converts a named network interface and sends it to DPDK.
    For example, if you provide nicToDp an interface name like p1p1 it detaches it and sets it up for DPDK to use. It does the following:
    1. Identifies the PCI address of the named interface.
    2. Note: From this point on, the Ethernet interface is identified by its PCI address alone, which is a physical characteristic based on where the Ethernet interface is installed in the computer.

    3. Creates configuration files that describe which PCI addresses DPDK is allowed to use.
    4. Sets up a systemd unit that binds the configured PCI addresses to DPDK instead of to the default Linux drivers on the next system reboot.
  2. Reboot the Decoder host. The reboot accomplishes the following:
    1. Ensures that live capture on the interfaces stops before the network driver is unloaded.
    2. Binds the vfio_pci_generic driver to the configured interfaces.
    3. Pre-allocates some Huge Page memory for each interface, as required by the DPDK libraries.
  3. After the reboot, the Decoder does not start capturing, since the network interface it had been using to capture has been moved to DPDK. Change the Decoder's capture configuration to select the DPDK device that is now available. For information about how to change the capture configuration, see Configure Capture Settings.
  4. Restart the service.

Utilizing Receive Side Scaling with DPDK

Receive Side Scaling (RSS) efficiently distributes the network receive processing across multiple CPUs in a multi-processor system. RSS ensures that the processing associated with a given connection stays on the assigned CPU.

IMPORTANT: Receive Side Scaling is supported on DPDK devices using the ixgbe or i40e device drivers only.

Advantages

When RSS is enabled, the DPDK capture devices split the incoming packet stream into multiple queues providing following benefits:

  • Each queue is processed with a dedicated capture worker thread. As there are more threads processing the capture work, it allows more CPU time available for Berkeley Packet Filters (BPF) rule and network rule evaluation.
  • Each capture worker thread can feed multiple assemble worker threads simultaneously which allows more CPU time available for assembly.
  • Each CPU thread processes only a fraction of the packet stream. However, the total number of packets processed from all queues working in parallel increases the overall output (packets processed per second) of Decoder.

Configuring Receive Side Scaling

To enable RSS in a DPDK controlled NIC, set the /decoder/devices/DPDK/(PCI_ID)/config/rss value to a value greater than 1. The typical values of RSS must be a small power of 2, for example, 4 or 8. This value determines the number of CPU threads to be allotted for packet processing during the next capture. Similarly, if assembler threading is enabled, a matching number of assembler threads will be created to process the packets on each incoming queue.

For example, the following sample configuration will utilize a total of eight threads (four capture threads + four assembler threads) for multi-adapter packet capture with DPDK and multi-assembler threading turned on. This is the maximum recommended value for an eight core Decoder CPU.

assembler.threading.enabled set to on or true

/decoder/devices/DPDK/0000:01:00.1/config/rss=2

/decoder/devices/DPDK/0000:01:00.0/config/rss=2

To minimize packet drops, it is recommended that you keep the total number of capture and assembler threads less than the number of cores per CPU.

How it Works

When RSS is enabled, Decoder utilizes the RSS hardware acceleration on the capture NIC to apply a symmetric RSS hash function. The hash function sends packets from both directions of a given stream to the same queue. It enables the packets received on that queue to be processed by a dedicated assembler thread. This eliminates the need for Decoder to merge streams from different RSS queues and reassemble the streams. Hence, it is recommended that you enable multi-threaded assembly when using RSS. To enable, set /decoder/config/assembler.threading.enabled to true or on.