Storage: Storage Requirements

Document created by RSA Information Design and Development on May 29, 2019Last modified by RSA Information Design and Development on Sep 25, 2019
Version 7Show Document
  • View in full screen mode
 

This section contains generic RSA NetWitness Platform storage requirements. Contact your NetWitness Platform representative for a detailed analysis of the current and future storage requirements for your NetWitness Platform deployment.

General Specifications

General specifications for core NetWitness Platform Hosts are:

  • IO size 490/Dec
  • Response/Latency < 20ms
  • Decoder 10/90 read/write (low random I/O)
  • Concentrator 50/50 read/write (high random I/O)
                   
RAID GroupSuitable Volumes
NL-SAS or 10K SAS

All Packet Decoder volumes

All Log Decoder volumes

All Archiver volumes

Concentrator meta volume

SSDConcentrator index volume

Performance Recommendations

RSA recommends that Packet and Log Decoders receive two LUNs or Block Devices, one for Packet data, the other for all other databases. This allows you to segregate the high-bandwidth Packet Database from the other databases so they do not compete for I/O bandwidth with other activity.

Concentrators require a separate SSD-based index volume for best performance. You must house this index volume on a different RAID group than the Concentrator Meta database volume, which you can stored on NL-SAS. Archivers can use a single large NL-SAS storage volume per appliance.

Recommended Storage Allocation Per NetWitness Host

                                 
Host
Type
First LUN/Block Device Second LUN/Block Device
DecoderMeta/Session Volume (smaller volume)Packet Volume (large volume)
Log DecoderMeta/Session Volume (medium-sized volume)Packet Volume (medium-sized volume)
ConcentratorMeta Volume (large volume)Index Volume (smaller SSD volume)
ArchiverData Archive Volume (large volume)Not used

Input/Output Operations Per Second

The following table lists the IOPS requirements for the Decoder and Concentrator hosts.

                            
LogsLog Decoder Concentrator
10K EPS4008,000
20K EPS55010,300

25K EPS

1,200

10,800

 

                                      
PacketsNetwork DecoderConcentrator
1Gbps6006,050
2 Gbps9508,300

4 Gbps

1,650

12,800

6 Gbps2,40017,300

8 Gbps

3,200

21,800

General Description of How NetWitness Platform Hosts Store Data

In most deployments, NetWitness Platform Decoders, Log Decoders, Concentrators, Archivers, and Hybrid hosts require external storage to house their data. Each host uses the external storage in different ways and with different expectations on throughput and performance of the external storage. Some hosts have a higher occurrence of sequential writes and some hosts have a higher occurrence of random reads and writes.

Decoder Hosts

Log Decoders and Network Decoders capture data and parse meta. The difference between these two hosts is in the type of data they capture:

  • Log Decoder captures logs.
  • Network Decoder captures packets.

Both Log Decoders and Network Decoders parse out meta data from the raw captured traffic. The meta data is then aggregated to a Concentrator for indexing. The host requires storage to house the raw payload data (raw packets or raw logs) and a cache for the meta extracted during data capture for Concentrator aggregation.

Your retention requirements is a key factor in determining the amount of storage you need for the raw packets or raw logs. In most deployments, you add storage over time based on increased retention requirements and increased capture rates. The storage for the raw data must support a high amount of sequential writes with random reads. Especially in the case of higher speed Network Decoder environments, it is recommended to have a minimum of two partitions exposed to the host to support the throttling between partitions for reads and writes.

The meta cache on a Decoder is generally fixed in size but you can expand it to support additional cache the possible loss of connectivity between the Decoder and a corresponding Concentrator. The meta cache must support a random IOPS rate for sustained writes from the Decoder of meta extracted and the corresponding reads from the Concentrator as meta is aggregated to a Concentrator.

Concentrator Host

A Concentrator aggregates and indexes the meta data from a Decoder. Both the meta and index storage needs are scaled based on a your NetWitness Platform deployment retention requirements. Similar to raw data stored on the Decoders, you may need to increase the storage for both meta data and index data over time to meet your retention requirements.

The meta storage houses all meta data extracted from either a Network Decoder or Log Decoder. Although the ratio of how much meta is extracted may change, the expectations for performance against meta storage is the same for both packet capture and log capture environments. The meta storage must support a sustained amount of sequential writes with random reads of meta data.

The index storage houses the live index generated from the meta data aggregated to a Concentrator. The size of the index is directly related to the size of the meta store. In addition to supporting IOPS for sustained writes, the index also needs to support a much higher rate IOPS for reads than meta based on interactive queries run through analyst interaction and reports and alerts.

Archiver Host

The Archiver host requires a single partition for both meta and raw log storage. The storage pool deals primarily with sequential writes for long term data written from a Log Decoder or Network Decoder and random reads for reports and analysis.

Hybrid Hosts

A Hybrid hosts two or more services on a single host. For example, a:

  • Network Hybrid hosts both the Decoder and Concentrator services handling packets exclusively. It captures packet data and indexes this data to the Concentrator service. Expectations for storage performance match what is outlined for a dedicated Network Decoder host and dedicated Concentrator host.
  • Log Hybrid hosts both the Log Decoder and Concentrator services handling logs exclusively. It captures log data and indexes the data to a Concentrator service. Expectations for performance match what is outlined for a dedicated Log Decoder and dedicated Concentrator.

Options for SAN Configurations

If you want to use a Storage Area Network (SAN) , use the same basic drive groups and partition organization that you use for the other RSA storage devices. Depending on the SAN configuration and overhead, SAN configurations may require more enclosures and drives to operate with the same performance as on PowerVault or DAC. When deciding whether to use SAN, PowerVault, or DAC, any additional overhead on the SAN will be important to determine the minimum storage required.

Previous Topic:Storage Overview
You are here
Table of Contents > Storage Requirements

Attachments

    Outcomes