Implementing the RSA NetWitness Platform with Dell EMC Unity Storage

Document created by Kevin Arunski Employee on Mar 21, 2018Last modified by RSA Link Team on Jun 1, 2018
Version 6Show Document
  • View in full screen mode

Who should read this document

System administrators installing RSA NetWitness appliances that are attached to Dell EMC Unity storage.  

 

Prerequisites

  • Dell EMC Unity storage is online and accessible
  • Dell EMC PowerPath licenses are available
  • An RSA NetWitness appliance is available that can be attached to Dell EMC Unity storage:
    • Packet Decoder
    • Log Decoder
    • Concentrator
    • Archiver

 

Overview

It is important to follow the steps in the order listed below. 

  1. Install the appliance software using the OS installation image
  2. Allocate storage through the Dell EMC Unity UI
  3. Install PowerPath on the appliance
  4. Instantiate the appliance with the RSA NetWitness Platform software
  5. Run the RSA NetWitness Unity configuration program

 

Install the Appliance

If your appliance does not already have the RSA NetWitness Platform 11.1 or 10.6.5 base OS image on it yet, you may have to install it using the RSA-provided OS image.  Further details on installing the RSA NetWitness Platform install image are located in the following documents:

 

For the purpose of a Unity storage install, we will perform storage setup BEFORE setting up the appliance as an RSA NetWitness node.

Set up the Unity storage before running nwsetup-tui

Allocate Storage in Unity

You will need to work with your Dell EMC Storage Engineer to allocate storage within your Unity environment for the RSA NetWitness Platform.

 

RAID Groups needed for the RSA NetWitness Platform

Your Unity Storage will contain some number of NL-SAS drives and SSD drives, which may vary depending on the exact configuration purchased.   For the purposes of the RSA NetWitness Platform, we recommend organizing RAID groups that correspond to each drive type.  

 

RAID Group TypeSuitable For
NL-SAS

All Packet Decoder volumes

All Log Decoder volumes
All Archiver volumes

Concentrator meta volume

SSDConcentrator index volume

Performance Recommendations

RSA recommends that Packet and Log Decoders receive two LUNs, one for Packet data, the other for all other databases.  This allows you to segregate the high-bandwidth Packet Database from the other databases so they do not compete for I/O bandwidth with other activity.

Concentrators require a separate SSD-based index volume for best performance.   This will necessarily be housed on a different RAID group than the Concentrator Meta database volume, which can be stored on NL-SAS.  Archivers can utilize a single large NL-SAS storage volume per appliance.

 

Appliance TypeFirst LUNSecond LUN
DecoderMeta/Session Volume (smaller NL-SAS volume)Packet Volume (large NL-SAS)
Log DecoderMeta/Session Volume (medium-sized NL-SAS)Packet Volume (medium-sized NL-SAS)
ConcentratorMeta Volume (large NL-SAS)Index Volume (SSD)
ArchiverData Archive Volume (large-NL-SAS)Not used

 

Every RSA NetWitness appliance that will be using the Unity storage needs to be added as a host within the Unity interface. After hosts and LUNs are created, you must assign the LUNs to the hosts.  Assigning the LUNs to the hosts makes the storage visible to the host.  At this point the host will be able to locate the storage through the host-based Dell EMC PowerPath software.

 

Install Dell EMC PowerPath on the Appliance

Dell EMC PowerPath must be installed on the appliance.  Work with your Dell EMC Storage Engineer to receive your licenses and install the software. 

 

Verify that the PowerPath license is installed using the emcpreg command:

[root@NWAPPLIANCE24932 ~]# emcpreg -list
Key BQPO-DB4M-VFC2-Q24R-ML9Z-EQTU
  Product: PowerPath
  Capabilities: All

A Reboot is recommended after installing PowerPath

After the PowerPath install is complete, you may verify that the LUNs are successfully attached to the appliance using the command powermt display dev=all.   An example powermt output is shown here:

[root@NWAPPLIANCE24932 ~]# powermt display dev=all
Pseudo name=emcpowera
Unity ID=APM00174407815 [Host_21]
Logical device ID=600601609D9046006996745A46B60AB6 [DecoderSmall01]
state=alive; policy=CLAROpt; queued-IOs=0
Owner: default=SP A, current=SP A   Array failover mode: 4
==============================================================================
--------------- Host ---------------   - Stor -  -- I/O Path --   -- Stats ---
###  HW Path               I/O Paths    Interf.  Mode     State   Q-IOs Errors
==============================================================================
  13 lpfc                   sde        SP A6     active   alive      0      0
  12 lpfc                   sdc        SP B6     active   alive      0      0


Pseudo name=emcpowerb
Unity ID=APM00174407815 [Host_21]
Logical device ID=600601609D904600BD96745A8040063A [DecoderLarge01]
state=alive; policy=CLAROpt; queued-IOs=0
Owner: default=SP B, current=SP B   Array failover mode: 4
==============================================================================
--------------- Host ---------------   - Stor -  -- I/O Path --   -- Stats ---
###  HW Path               I/O Paths    Interf.  Mode     State   Q-IOs Errors
==============================================================================
  13 lpfc                   sdf        SP A6     active   alive      0      0
  12 lpfc                   sdd        SP B6     active   alive      0      0

 

Set up the Appliance as an RSA NetWitness Node

Proceed with installation of the RSA NetWitness Platform software by running nwsetup-tui.  Further details can be found in the RSA NetWitness Logs & Packets 11.0  Physical Host Installation Guide

 

Run the RSA NetWitness Unity Configuration Program

Run NwUnityConfig.py to allocate the storage volumes presented to your appliance within the RSA NetWitness Platform software.  

 

This utility performs these tasks automatically:

  1. Identifies PowerPath-controlled LUNs presented to the appliance.
  2. Builds Linux volumes and filesystems
  3. Creates mount points for the volumes
  4. Updates the RSA NetWitness service configurations to utilize the provided volumes

 

Invoking NwUnityConfig

/opt/rsa/saTools/NwUnityConfig.py

 

Below is an example of successful output from NwUnityConfig successfully configuring Log Decoder storage:

[root@NWAPPLIANCE24932 ~]# /opt/rsa/saTools/NwUnityConfig.py
Creating new volume group logdecodersmall on /dev/emcpowera
  Volume group "logdecodersmall" successfully created
Creating new volume group logdecoder on /dev/emcpowerb
  Volume group "logdecoder" successfully created
Success!: Added all available storage found. Successfully configured the logdecoder with the appropriate disk arrays. You will need to restart the logdecoder service for the database configuration to be loaded.

 

Validate that the filesystems have been created and mounted:

[root@NWAPPLIANCE24932 ~]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root            20G  2.7G   16G  15% /
tmpfs                                  63G     0   63G   0% /dev/shm
/dev/sda1                             496M   67M  403M  15% /boot
/dev/mapper/VolGroup00-usrhome        3.9G  8.1M  3.7G   1% /home
/dev/mapper/VolGroup00-opt             20G  395M   20G   2% /opt
/dev/mapper/VolGroup00-rsaroot         10G   62M   10G   1% /opt/rsa
/dev/mapper/VolGroup00-tmp             20G   33M   20G   1% /tmp
/dev/mapper/VolGroup00-var             20G  112M   20G   1% /var
/dev/mapper/VolGroup01-rabmq          324G   37M  324G   1% /var/lib/rabbitmq
/dev/mapper/VolGroup00-varlog          16G   51M   16G   1% /var/log
/dev/mapper/VolGroup00-nwhome          30G   39M   30G   1% /var/netwitness
/dev/mapper/VolGroup01-lcol           300G   65M  300G   1% /var/netwitness/logcollector
/dev/mapper/VolGroup01-warec          400G   33M  400G   1% /var/netwitness/warehouseconnector
/dev/mapper/VolGroup00-vartmp         4.0G   33M  4.0G   1% /var/tmp
/dev/mapper/logdecodersmall-decoroot   10G   33M   10G   1% /var/netwitness/logdecoder
/dev/mapper/logdecodersmall-index      30G   33M   30G   1% /var/netwitness/logdecoder/index
/dev/mapper/logdecodersmall-metadb    9.0T   34M  9.0T   1% /var/netwitness/logdecoder/metadb
/dev/mapper/logdecodersmall-sessiondb 1.0T   34M  1.0T   1% /var/netwitness/logdecoder/sessiondb
/dev/mapper/logdecoder-packetdb        20T   34M   20T   1% /var/netwitness/logdecoder/packetdb

 

Within the Core service itself, you can see the storage configuration entries added:

[root@NWAPPLIANCE24932 ~]# NwConsole
RSA Security Analytics Console 10.6.5.0
Copyright 2001-2017, RSA Security Inc.  All Rights Reserved.

Type "help" for a list of commands or "man" for a list of manual pages.
> login localhost:50002 admin

Password: **********
Successfully logged in as session 819
[localhost:50002] /> cd /database/config
[localhost:50002] /database/config
[localhost:50002] /database/config> ls
hash.algorithm (Hash Algorithm) = none
hash.databases (Hash Databases) = session,meta,packet
hash.dir (Hash Directory)
manifest.dir (Manifest Directory)
meta.compression (Meta Compression) = none
meta.compression.level (Meta Compression Level) = 0
meta.dir (Meta Database Directory) = /var/netwitness/logdecoder/metadb=8.51 TB
meta.dir.cold (Cold Meta Database Directory)
meta.dir.warm (Warm Meta Database Directory)
meta.file.size (Meta File Size) = 3 GB
meta.files (Meta Open Files) = auto
meta.free.space.min (Meta Minimum Free Space) = 79 GB
meta.index.fidelity (Meta Index Fidelity) = 1
meta.integrity.flush (Meta Integrity Flush) = sync
meta.write.block.size (Meta Write Block Size) = 64 KB
packet.compression (Packet Compression) = none
packet.compression.level (Packet Compression Level) = 0
packet.dir (Packet/Log Database Directory) = /var/netwitness/logdecoder/packetdb=18.99 TB
packet.dir.cold (Cold Packet/Log Database Directory)
packet.dir.warm (Warm Packet/Log Database Directory)
packet.file.size (Packet File Size) = 5 GB
packet.files (Packet Open Files) = auto
packet.free.space.min (Packet Minimum Free Space) = 178 GB
packet.index.fidelity (Packet Index Fidelity) = 1
packet.integrity.flush (Packet Integrity Flush) = sync
packet.write.block.size (Packet Write Block Size) = 64 KB
session.dir (Session Database Directory) = /var/netwitness/logdecoder/sessiondb=972.32 GB
session.dir.cold (Cold Session Database Directory)
session.dir.warm (Warm Session Database Directory)
session.file.size (Session File Size) = 256 MB
session.files (Session Open Files) = auto
session.free.space.min (Session Minimum Free Space) = 8 GB
session.integrity.flush (Session Integrity Flush) = sync
session.write.block.size (Session Write Block Size) = 32 KB

[localhost:50002] /database/config> cd /index/config
[localhost:50002] /index/config
[localhost:50002] /index/config> ls
index.dir (Index Directory) = /var/netwitness/logdecoder/index=26.98 GB
index.dir.cold (Index Cold Storage Directory)
index.dir.warm (Index Warm Storage Directory)
index.slices.open (Index Open Slice Count) = 42
page.compression (Page Compression) = huffhybrid
save.session.count (Save Session Count) = 0

 

Troubleshooting

The Unity Config program generates a log file, arrayCfg.log, if it encounters an error in the Unity storage setup.  Verbose command output can be found in this file. The arrayCfg.log file is created in the working directory from which the NwUnityConfig.py command is invoked.

 

Common Error Output from NwUnityConfig

ErrorExplanation
Failed!: Ssl may be set opposite of what was attempted

The core service configuration could not be updated.

Verify that the core service is running (NwDecoder, NwLogDecoder, NwConcentrator or NwArchiver)

Failed!: No available VNX LUNs found. Verify VNX configuration before trying again.

The storage LUNs attached to this system have already been allocated.

As a safety precaution, NwUnityConfig will not overwrite any volume that might currently store data.

To reallocate LUNs, you must unmount any filesystems on them and manually remove the Logical Volumes, Volume Groups, and Physical Volumes defined on the LUNs.

Unity-specific Issues You May Encounter During Installation

  1. Multiple drives show up in the ‘Failed State’ resulting in the Pool going offline.  For example, you see messages stating "Storage pool NLPool is offline." The disks toggle between failed and normal state randomly.
    • Resolution: Apply the latest Drive FW
    • Details: 517273 : Dell EMC Unity: DAE X X Disk XX is resynchronizing with the system (Dell EMC Correctable) https://support.emc.com/kb/517273
  2. Rebooting both the storage Processors at (or about) the same time resulting in the following error message:
    The Dirty Cache data for LUN 4 has been lost. Gather service information and contact your service provider.
    • The storage processors must be rebooted one after the other.  Make sure the first storage processor is online before rebooting the second one.

      In some cases, this could reflect a power failure scenario. Requires Dell EMC Support to resolve.

  3. Incorrect SFP used in the host's Emulex cards
    • The SFP must be one of the following:
      • Finisar FTLF8529P3BNV (019-048-045)

      • Avago AFBR-57F5AMZ (no Dell EMC part number)

      • Delivered with new Emulex cards: Emulex Part No: NET-PCI-DELL-EMULXDP-8GB-FC-N NETWITNESS

  4. Dell EMC PowerPath license not applied.  This results in PowerPath not enabling all failover modes.
    • Run the following command on the RSA NetWitness host and the reboot the host:

      emcpreg --add <LicenseCode>

    • Verify the License is applied with the following command:

      emcpreg --list

1 person found this helpful

Attachments

    Outcomes