Storage: Configure Storage Using the REST API

Document created by RSA Information Design and Development Employee on May 29, 2019Last modified by RSA Information Design and Development Employee on Jul 1, 2020
Version 17Show Document
  • View in full screen mode
 

In NetWitness Platform 11.3 and later releases, you use the REST API for all storage configuration operations. For information about how to use the REST API, see the RESTful API User Guide. Go to the Master Table of Contents to find all RSA NetWitness Platform 11.x documents.

REST API Storage Configuration Commands

Each of the commands listed below has built-in help that describes their function and usage. If you are using the REST interface, select the command from the drop-down menu to see the help text.

Commands for Direct-Attached RAID Volumes

  • raidList - List the RAID controllers and direct-attach enclosures that are present on this host.
  • raidNew - Allocate direct-attached enclosures to block devices.

Commands for Allocating Block Devices as Storage

  • devlist - List available block devices on the host.
  • partNew - Allocate partitions on a block device and create volume groups.
  • vgs - Summarize how block devices are organized into volume groups.

Commands for Allocating Storage to Services

  • srvList - List services on the host and their allocated storage paths.
  • srvAlloc - Allocate a volume group to a service.
  • srvFree - Remove a volume group from a service.

Command to Reconfigure Services to Detect and Use All of the New Storage

  • reconfig - After configuring new storage, detect and use new storage on the associated service and database.

Storage Configuration Tasks

Task 1 - Attach storage to the host and access the REST API storage configuration commands.

Task 2 - (Conditional) Configure RAID if necessary.

Task 3 - Allocate block devices to partitions, volume groups, and logical volumes.

Task 4 - Allocate volume groups to NetWitness services.

Task 5 - Reconfigure services and databases to detect and appropriately use new storage.

Task 1 - Attach Storage to the Host and Access the REST API Storage Commands

Complete the following steps to attach an external storage device to a host and access the storage configuration commands available through the REST API.

  1. Attach the storage and make it available to this host.
    • For direct-attach storage, the RAID manipulation commands construct the hardware RAID volumes and make these volumes appear as drives.
    • For SAN storage, you must allocate the storage through the SAN management tools and present them to this host.
    • After you attach either type of storage, the storage appears as block devices (that is, /dev/sdc for direct-attach drives, or /dev/emcpowera for an EMC SAN drive. Attaching storage to a virtual or cloud instance also presents a block device to the host.
    • If you are connecting to virtual or cloud storage, go to Configure Storage Using the REST API.
  2. Access the REST API storage commands from either a Browser or the Services > Explore view from the User Interface.
    • From a Browser.
      1. Open a Browser and specify the ip-address of the host with port 50106.
        The following example is the Decoder, but you need to use port 50106 for any host hardware for which you are configuring storage using the REST API.
        https://<decoder-ip-address>:50106
      1. Log in with the admin account credentials.
        The following REST API menu is displayed.
      2. Click on the (*) next to appliance to access the REST command set.
        The Properties for /appliance dialog is displayed under the initial REST menu. The Output (or command manual help) section describes the commands that the REST API can send to the device, their usage, and their parameters.

    • From the User Interface.
      1. In the NetWitness Platform menu, go to (Admin) > SERVICES.
      2. Select the service (for example, a Concentrator).
      3. Under (actions), select View > Explore.
      4. Navigate to deviceappliance/appliance, right click, and click Properties.

        You can now access the storage commands from the Properties dialog.
  3. Proceed to:
    • Task 2 if you need to configure RAID for PowerVault or DACs.
    • Task 3 if you do not need to configure RAID and already have a block device available.

Task 2 - (Conditional) RAID Configuration for PowerVault and DACs

NetWitness Platform hardware uses direct-attached SAS drives for storage. These drives are housed in a SAS enclosure. SAS enclosures are shelves of drives attached to the NetWitness node by a cable connected to the SAS host bus adapter.

SAS enclosures are also known as other names, such as "DAC" (Direct-Attached Capacity), or "JBOD" (Jumbo Box of Disks), or "Dell PowerVault".

NetWitness Platform utilizes Dell PERC SAS host bus adapters. NetWitness Platform devices typically include two SAS host bus adapters. One is used for controller drives that are internal to the NetWitness Node, and another is used for controlling drives attached to the SAS enclosures. The internal controller and drives are configured when the node is built, but the external SAS enclosures are not. You execute the raidList and raidNew commands to identify and configure the external SAS enclosures.

These commands work with the following SAS enclosure types:

  • EMC ESAS 15-drive enclosures
  • EMC ESAS 60-drive enclosures
  • Dell PowerVault 12-drive enclosures

Note: EMC 60-drive enclosures are logically organized as four separate 15-drive sub-enclosures. They behave as if there are four 15-drive enclosures, each of which can be configured independently.

The raidList and raidNew commands operate on entire enclosures. Execute raidList to identify the enclosues. execute raidNew to configure an enclosure to perform one of the pre-determined roles within a NetWitness Platform node.

After you attach storage to the host and access the REST API storage commands, complete the following steps to create RAID if required.

  1. Execute the raidList command to identify the controllers and enclosures that are attached to the system.
    In the following example, Controller 1 does not display any block devices. This indicates the array is not configured.

  1. Select a RAID layout scheme for the Enclosure.
    The following tables show you the supported allocation schemes.

    Note: For RAID configuration, when you use the decoder for 10G Capture you use decoder for both enclosures for performance reasons. When you do not use the decoder for 10G Capture, you use the decoder and archiver for the enclosures for which to maximize storage, because the second enclosure is a single RAID under the archiver configuration.

                                           
SchemeDrives RequiredAllocation
decoder 12 or 15 HDDs

3x drives in RAID 5 for decodersmall, all remaining drives in RAID 5 for decoder

logdecoder12 or 15 HDDs

Same as decoder

archiver12 or 15 HDDsAll drives in RAID 6 for archiver or decoder database volume
networkhybrid12 or 15 HDDs 3x drives in RAID 5 for meta expansion, all remaining drives in RAID 5 for packet expansion
loghybrid12 or 15 HDDs

Half of the drives in RAID 5 for meta expansion, half the drives in RAID 5 for packet expansion

concentrator 3 or more SSDs, 3 or more HDDs All SSDs in RAID 5 for index, all HDDs in RAID 6 for meta
  1. After the controller, enclosure, and scheme are identified, execute the raidNew command to create RAID Volumes. For example:
    send /appliance raidNew controller=1 enclosure=82 scheme=decoder preferSecure=false
    Add the commit=1 parameter to actually execute this operation. Execute the raidList command to list the created block devices.
  2. (Optional) Configure SEDs (Self-Encrypting Drives). If the raidNew command detects self-encrypting drives and a security key has been set on the controller, the raidNew command will attempt to create a secure array. To set a security key on the controller, execute the raidKey command. For example:
    send /appliance raidKey controller=1 key=myPasssphrase keyId=1
    • To create a secured (that is, encrypted) array on physical devices attached to a controller with a security key set, specify preferSecure=true when using raidNew
    • To create an unsecured (that is, unencrypted) array on physical devices attached to a controller with a security key set, specify preferSecure=false when using raidNew.
  3. Go to Storage: Configure Storage Using the REST APITask 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes, after you create RAID volumes.

Task 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes

The partNew command prepares a storage device to use in NetWitness Platform. It performs the following tasks.

  • Creates the partition table on the block device.
  • Creates the Linux Volume Manager physical device partition.
  • Creates a volume group containing the physical device.
  • Creates logical volumes in the volume group.
  • Creates XFS filesystems on each logical volume.
  • Creates /etc/fstab entries for each logical volume.
  • Mounts each logical volume.

Complete the following steps to allocate block devices to partitions, volume groups, and logical volumes.

  1. Run the devlist command to locate unused block devices. The following example shows the devlist command output.

    You must provide a name for the service that will be used with the storage, for example, decoder for the Network Decoder service, or concentrator for the Concentrator service. You have the option of providing the volume type. The default volume type has the same name as the service.
  2. Execute the partNew command to allocate block devices to partitions, volume groups, and logical volumes.

    By default, the partNew command does not make changes. It displays the actions that will be taken if you commit the command string. To actually make the changes to the system, add the commit=true parameter to the command.
    For example, to assign devices sdd and sde to Decoder:
    send /appliance partNew name=sdc service=decoder volume=decodersmall commit=true
    send /appliance partNew name=sdd service=decoder volume=decoder commit=true

    Caution: For the decoder and concentrator services, you must create storage volumes in a specific order.
    - The decoder has the decodersmall and decoder volumes. Create the decodersmall volume before the decoder volume because decodersmall contains the small filesystem mounted at /var/netwitness/decoder.
    - The concentrator has the concentrator and index volumes. Create the concentrator volume before index volume or it will fail and you receive the following message.
    Failed to process message partNew for /appliance com.rsa.netwitness.carlos.transport.TransportException: Volumes for index require mount point /var/netwitness/concentrator to be created and mounted first.

  3. Execute the vgs command to validate that the partNew command created the correct Logical Volumes.
    The output of this command:
    • Enumerates all the volume groups on this host. I
    • Displays the physical volumes that the volume group consists of, and the logical volumes within the volume group.
  4. Go to Task 4 - Allocate Volume Groups to NetWitness Services- srvAlloc.

Task 4 - Allocate Volume Groups to NetWitness Services - srvAlloc

The srvAlloc command configures services on a host to use storage in a volume group. You must provide the name of the service to configure and the volume group to assign to the service (the service you provide must be installed on the host). For information about NetWitness Platform service volumes, see "NetWitness Platform Service Volume Reference" in Storage Requirements.

Allocate services in the following order:

  • For the Decoder, allocate decodersmall first then the decoder
  • For a Concentrator, allocate concentrator first then index.

Note: By default, the srvAlloc command does not make changes. You must append the commit=true parameter to the command string to actually make the changes to the system and restart the specified service after making changes.

  1. Execute the srvLst command to see a list of services installed on this host.
    The srvLst command communicates with the service through the SSL port. You install a Category on a host. A Category can be a single service, or multiple related services, located on the same host.
  1. Execute the srvAlloc command to configure a service on a host to use storage in a volume group. For example:
    service=concentrator volume=index commit=1
    service=concentrator volume=concentrator commit=1
  2. Go to Task 5 - Reconfigure Services and Databases to Detect and Appropriately Use New Storage.

Task 5 - (Optional) Reconfigure Storage Configuration for 10G Capture

You need to reconfigure the Decoder service and databases for 10G capture. Complete the following steps so that the Network Decoder service and its database detect and use new free space.

  1. In the NetWitness Platform menu, go to (Admin) > SERVICES.
    The SERVICES view is displayed.
  2. Select the decoder.
  3. Under (actions), select View > Explore.
    The Explore tree for the service is displayed.
  4. Reconfigure space on the decoder service.
    1. Navigate to the decoder, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

  5. Reconfigure space on the database.
    1. Navigate to database in the service Explore tree, right click, and click Properties.

      The Properties dialog is displayed.
    1. Execute the reconfig command by selecting it from the drop-down list, specifiy update=1 in Parameters, and click Send.

You are here
Table of Contents > Configure Storage Using the REST API

Attachments

    Outcomes