Storage: Configure Storage Using the REST API

Document created by RSA Information Design and Development on May 29, 2019Last modified by RSA Information Design and Development on Jan 30, 2020
Version 12Show Document
  • View in full screen mode
 

In NetWitness Platform 11.3 and later releases, you use the REST API for all storage configuration operations.

REST API Storage Configuration Commands

Each of the commands listed below has built-in help that describes their function and usage. If you are using the REST interface, select the command from the drop-down menu to see the help text.

Commands for Direct-Attached RAID Volumes

  • raidList - List the RAID controllers and direct-attach enclosures that are present on this host.
  • raidNew - Allocate direct-attached enclosures to block devices.

Commands for Allocating Block Devices as Storage

  • devlist - List available block devices on the host.
  • partNew - Allocate partitions on a block device and create volume groups.
  • vgs - Summarize how block devices are organized into volume groups.

Commands for Allocating Storage to Services

  • srvList - List services on the host and their allocated storage paths.
  • srvAlloc - Allocate a volume group to a service.
  • srvFree - Remove a volume group from a service.

Command to Reconfigure Services to Detect and Use All of the New Storage

  • reconfig - After configuring new storage, detect and use new storage on the associated service and database.

Storage Configuration Tasks

Task 1 - Attach storage to the host and access the REST API storage configuration commands.

Task 2 - (Conditional) Configure RAID if necessary.

Task 3 - Allocate block devices to partitions, volume groups, and logical volumes.

Task 4 - Allocate volume groups to NetWitness services.

Task 5 - Reconfigure services and databases to detect and appropriately use new storage.

Task 1 - Attach Storage to the Host and Access the REST API Storage Commands

Complete the following steps to attach an external storage device to a host and access the storage configuration commands available through the REST API.

  1. Attach the storage and make it available to this host.
    • For direct-attach storage, the RAID manipulation commands construct the hardware RAID volumes and make these volumes appear as drives.
    • For SAN storage, you must allocate the storage through the SAN management tools and present them to this host.
    • After you attach either type of storage, the storage appears as block devices (that is, /dev/sdc for direct-attach drives, or /dev/emcpowera for an EMC SAN drive. Attaching storage to a virtual or cloud instance also presents a block device to the host.
  2. Access the REST API storage commands from either a Browser or the Services > Explore view from the User Interface.
    • From a Browser.
      1. Open a Browser and specify the ip-address of the host with port 50106.
        The following example is the Decoder, but you need to use port 50106 for any host hardware for which you are configuring storage using the REST API.
        https://<decoder-ip-address>:50106
      1. Log in with the admin account credentials.
        The following REST API menu is displayed.
      2. Click on the (*) next to appliance to access the REST command set.
        he Properties for /appliance dialog is displayed under the initial REST menu. The Output (or command manual help) section describes the commands that the REST API can send to the device, their usage, and their parameters.

    • From the User Interface.
      1. In the NetWitness Platform menu, select ADMIN > Services.
      2. Select the service (for example, a Concentrator).
      3. Under (actions), select View > Explore.
      4. Navigate to deviceappliance/appliance, right click, and click Properties.

        You can now access the storage commands from the Properties dialog.
  3. Proceed to:
    • Task 2 if you need to configure RAID for the external storage device.
    • Task 3 if you do not need to configure RAID for the external storage device.

Task 2 - (Conditional) Create RAID If Necessary

NetWitness Platform hardware uses direct-attached SAS drives for storage. These drives are housed in a SAS enclosure. SAS enclosures are shelves of drives attached to the NetWitness node by a cable connected to the SAS host bus adapter.

SAS enclosures are also known as other names, such as "DAC" (Direct-Attached Capacity), or "JBOD" (Jumbo Box of Disks), or "Dell PowerVault".

NetWitness Platform utilizes Dell PERC SAS host bus adapters. NetWitness Platform devices typically include two SAS host bus adapters. One is used for controller drives that are internal to the NetWitness Node, and another is used for controlling drives attached to the SAS enclosures. The internal controller and drives are configured when the node is built, but the external SAS enclosures are not. You execute the raidList and raidNew commands to identify and configure the external SAS enclosures.

These command work with the following SAS enclosure types:

  • EMC ESAS 15-drive enclosures
  • EMC ESAS 60-drive enclosures
  • Dell PowerVault 12-drive enclosures

Note: EMC 60-drive enclosures are logically organized as four separate 15-drive sub-enclosures. They behave as if there are four 15-drive enclosures, each of which can be configured independently.

The raidList and raidNew commands operate on entire enclosures. Execute raidList to identify the enclosues. execute raidNew to configure an enclosure to perform one of the pre-determined roles within a NetWitness Platform node.

After you attach storage to the host and access the REST API storage commands, complete the following steps to create RAID if required.

  1. Execute the raidList command to identify the controllers and enclosures that are attached to the system.
    In the following example, Controller 1 does not display any block devices. This indicates the array is not configured.

  1. Select a RAID layout scheme for the Enclosure.
    The following tables show you the supported allocation schemes.

    Note: For RAID configuration, when you use the decoder for 10G Capture you use decoder for both enclosures for performance reasons. When you do not use the decoder for 10G Capture, you use the decoder and archiver for the enclosures to maximize storage for because the second enclosure is a single RAID under the archiver configuration.

                                           
SchemeDrives RequiredAllocation
decoder 12 or 15 HDDs

3x drives in RAID 5 for decodersmall, all remaining drives in RAID 5 for decoder

logdecoder12 or 15 HDDs

Same as decoder

archiver12 or 15 HDDsAll drives in RAID 6 for archiver or decoder database volume
networkhybrid12 or 15 HDDs 3x drives in RAID 5 for meta expansion, all remaining drives in RAID 5 for packet expansion
loghybrid12 or 15 HDDs

Half of the drives in RAID 5 for meta expansion, half the drives in RAID 5 for packet expansion

concentrator 3 or more SSDs, 3 or more HDDs All SSDs in RAID 5 for index, all HDDs in RAID 6 for meta
  1. After the controller, enclosure, and scheme are identified, execute the raidNew command to create RAID Volumes. For example:
    send /appliance raidNew controller=1 enclosure=82 scheme=decoder preferSecure=false
    Add the commit=1 parameter to actually execute this operation. Execute the raidList command to list the created block devices.
  2. Go to Storage: Configure Storage Using the REST APITask 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes, after you create RAID volumes.

Using SEDs (Self-Encrypting Drives)

If the raidNew command detects self-encrypting drives and a security key has been set on the controller, the raidNew command will attempt to create a secure array. To set a security key on the controller, execute the raidKey command. For example:

send /appliance raidKey controller=1 key=myPasssphrase keyId=1

To create a secured (that is, encrypted) array on physical devices attached to a controller with a security key set, specify preferSecure=true when using raidNew.

To create an unsecured (that is, unencrypted) array on physical devices attached to a controller with a security key set, specify preferSecure=false when using raidNew.

Prepare Virtual or Cloud Storage

Virtual or Cloud NetWitness Hosts need block storage attached. Make sure that the allocated storage meets all of the Storage Requirements. Specifically, make sure that:

  • You have at least two Block Devices are created for Decoders (meta /session and packet volumes)
  • You have at least two block devices are created for Concentrators (index and meta volumes)
  • Ensure block devices can meet the minimum IOPS for expected ingestion rates

Attach the allocated storage to the NetWitness Host by following the hosting platforms native procedure.

  • VmWare – Vsphere Console (add disk to VM)
  • Hyper-V – Manager Console (add disk to VM)
  • Azure – Add Managed Disks to virtual instance.
  • AWS – Add EBS Storage to virtual instance.

After the storage is attached to the virtual host, proceed to Task 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes.

Task 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes

The partNew command prepares a storage device to use in NetWitness Platform. It performs the following tasks.

  • Creates the partition table on the block device.
  • Creates the Linux Volume Manager physical device partition.
  • Creates a volume group containing the physical device.
  • Creates logical volumes in the volume group.
  • Creates XFS filesystems on each logical volume.
  • Creates /etc/fstab entries for each logical volume.
  • Mounts each logical volume.

Complete the following steps to allocate block devices to partitions, volume groups, and logical volumes.

  1. Run the devlist command to locate unused block devices. The following example shows the devlist command output.

    You must provide a name for the service that will be used with the storage, for example, decoder for the Network Decoder service or concentrator for the Concentrator service and you have the option of providing the volume type. The default volume type has the same name as the service.
  2. Execute the partNew command to allocate block devices to partitions, volume groups, and logical volumes.

    By default, the partNew command does not make changes. It displays the actions that will be taken if you commit the command string. To actually make the changes to the system, add the commit=true parameter to the command.
    For example, to assign devices sdd and sde to Decoder:
    send /appliance partNew name=sdc service=decoder volume=decodersmall commit=true
    send /appliance partNew name=sdd service=decoder volume=decoder commit=true

    Caution: For the decoder and concentrator services, you must create storage volumes in a specific order.
    The decoder has the decodersmall and decoder volumes. Create the decodersmall volume before the decoder volume because decodersmall contains the small filesystem mounted at /var/netwitness/decoder.
    The concentrator has the (concentrator and index volumes. Create the concentrator volume before index volume or it will fail and you receive the following message.
    Failed to process message partNew for /appliance com.rsa.netwitness.carlos.transport.TransportException: Volumes for index require mount point /var/netwitness/concentrator to be created and mounted first.

  3. Execute the vgs command to validate that the partNew command created the correct Logical Volumes.
    The output of this command:
    • Enumerates all the volume groups on this host. I
    • Displays the physical volumes that the volume group consists of, and the logical volumes within the volume group.
  4. Go to Task 4 - Allocate Volume Groups to NetWitness Services- srvAlloc.

NetWitness Service Volume Reference

Service Volume Names

                                                

Service

Volume NameFilesystems Created
Network Decoder decoder packetdb
Network Decoderdecodersmalldecoder root, index, sessiondb, metadb
Log Decoderlogdecoder packetdb
Log Decoderlogdecodersmall logdecoder root, index, sessiondb, metadb
Concentratorconcentrator

concentrator root, metadb, sessiondb

Concentrator index index

Archiver

archiver

database

Volume Sizing

                                                                                                         

Volume

FilesystemMount PointSize
decodersmall decoroot /var/netwitness/decoder 10 GB
decodersmall index /var/netwitness/decoder/index 30 GB
decodersmall sessiondb /var/netwitness/decoder/sessiondb 600 GB
decodersmall metadb/var/netwitness/decoder/metadb100% of free space on
decodersmall volume
decoder packetdb/var/netwitness/decoder/packetdb100% of free space on
decoder volume
logdecodersmalldecoroot /var/netwitness/logdecoder 10 GB

logdecodersmall

index

/var/netwitness/logdecoder/index

30 GB

logdecodersmallsessiondb /var/netwitness/logdecoder/sessiond 600 GB

logdecodersmall

metadb

/var/netwitness/logdecoder/metadb

100% of free space on
logdecodersmall volume

logdecoder packetdb /var/netwitness/logdecoder/packetdb 100% of free space on
logdecoder volume

concentrator

root

/var/netwitness/concentrator

30 GB

concentrator sessiondb /var/netwitness/concentrator/sessiondb600 GB

concentrator

metadb

/var/netwitness/concentrator/metadb

100% of free space on
concentrator volume

index index /var/netwitness/concentrator/index 100% of free space on
index volume

archiver

database

/var/netwitness/archiver/database

100% of free space on
archiver volume

Task 4 - Allocate Volume Groups to NetWitness Services - srvAlloc

The srvAlloc command configures services on a host to use storage in a volume group. You must provide the name of the service to configure and the volume group to assign to the service (the service you provide must be installed on the host).

Allocate services in the following order:

  • For the Decoder, allocate decodersmall first then the decoder
  • For a Concentrator, allocate index first then concentrator.

Note: By default, the srvAlloc command does not make changes. You must append the commit=true parameter to the command string to actually make the changes to the system and restart the specified service after making changes.

  1. Execute the srvLst command to see a list of services installed on this host.
    The srvLst command communicates with the service through the SSL port. You install a Category on a host. A Category can be a single service, or multiple related services, located on the same host. The following table lists the services by Category.
                                      
CategoryServicesEncrypted SSL Port
Archiver Archiver56008
ConcentratorConcentrator565005
Log CollectorLog Collector56001
Log DecoderLog Collector
Log Decoder
56001
56002
Network Decoder 56004
  1. Execute the srvAlloc command to configures service on a host to use storage in a volume group. For example:
    service=concentrator volume=index commit=1
    service=concentrator volume=concentrator commit=1
  2. Go to Task 5 - Reconfigure Services and Databases to Detect and Appropriately Use New Storage.

Task 5 - Reconfigure Services and Databases to Detect and Appropriately Use New Storage

You need to reconfigure the following services and databases to detect and appropriately use all of the free space created in Tasks 1 through 4.

Archiver Service

Complete the following steps so that the archiver service detects and uses the new free space and has enough hot storage.

  1. In the NetWitness Platform menu, select ADMIN > Services.
    The Services view is displayed.
  2. Select the archiver service.
  3. Under (actions), select View > Explore.
    The Explore tree for the service is displayed.
  4. Reconfigure space on the service.
    1. Navigate to the archiver , right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.
    2. Go to the Config view, open the Data Retention tab, and confirm the Hot Storage.

Concentrator Service and Its Database

Complete the following steps so that the Concentrator service and its database detect and use new free space.

  1. In the NetWitness Platform menu, select ADMIN > Services.
    The Services view is displayed.
  2. Select the concentrator.
  3. Under (actions), select View > Explore.
    The Explore tree for the service is displayed.
  4. Reconfigure space on the concentrator service.
    1. Navigate to the concentrator), right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.
  5. Reconfigure space on the database.
    1. Navigate to database in the service Explore tree, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

Network Decoder Service and Its Database

Complete the following steps so that the Network Decoder service and its database detect and use new free space.

  1. In the NetWitness Platform menu, select ADMIN > Services.
    The Services view is displayed.
  2. Select the decoder.
  3. Under (actions), select View > Explore.
    The Explore tree for the service is displayed.
  4. Reconfigure space on the decoder service.
    1. Navigate to the decoder, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

  5. Reconfigure space on the database.
    1. Navigate to database in the service Explore tree, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

Log Decoder Service and Its Database

Complete the following steps so that the Log Decoder service and its database detect and use new free space.

  1. In the NetWitness Platform menu, select ADMIN > Services.
    The Services view is displayed.
  2. Select the decoder.
  3. Under (actions), select View > Explore.
    The Explore tree for the service is displayed.
  4. Reconfigure space on the decoder service.
    1. Navigate to the decoder, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

  5. Reconfigure space on the database.
    1. Navigate to database in the service Explore tree, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

Previous Topic:Storage Requirements
You are here
Table of Contents > Configure Storage Using the REST API

Attachments

    Outcomes