NRT: Disaster Recovery (Back Up and Restore)

Document created by RSA Information Design and Development on Sep 12, 2018Last modified by RSA Information Design and Development on Apr 11, 2019
Version 2Show Document
  • View in full screen mode
 

You can use the NetWitness Recovery Tool (NRT) to back up and restore data from the NetWitness Server and component host systems. The NRT is a script that you run from the command line to back up and restore data on hosts for RMAs, hardware refreshes, and general backup and restore requirements. Refer to Disaster Recovery in Azure Deployment for specific steps on how to perform disaster recovery for hosts deployed in Azure VMs.

You must run the NRT on each host system locally. You cannot run it from remote hosts or an external host.

The following types of hosts can be backed up and restored.

In the NRT script, the following terms in bold are referred to as categories.

  • NetWitness Admin Server (may include Respond, Health and Wellness, and Reporting Engine)
  • Archiver Log Archiver (Workbench and Archiver)
  • Broker Stand-alone Broker
  • Concentrator Network or Log Concentrator
  • Decoder Network Decoder (Packets)
  • Endpoint Broker Endpoint Broker
  • Endpoint Log Hybrid Log Collector, Log Decoder, Endpoint Server, and Concentrator

  • Event Stream Analysis (ESA) Primary Entity Behavior Analytics, Contexthub, ESA Correlation, and Incident Management database
  • ESA Secondary Entity Behavior Analytics and ESA Correlation
  • Gateway Cloud Gateway
  • Log Collector Log Collector including Virtual Log Collector if installed
  • Log Decoder Log Decoder including Local Log Collector and Warehouse Connector, if installed
  • Log Hybrid Log Collector, Log Decoder, and Concentrator
  • Malware Malware Analysis and Broker
  • Network Hybrid Concentrator and Decoder
  • UEBA User Entity and Behavior Analytics
  • Warehouse Warehouse Connector

Basic Usage of the NetWitness Recovery Tool

You can use the NRT to back up data by using the export option. To restore data, use the import option. The basic usage of the tool is to run the following command from the root directory level:

nw-recovery-tool [command] [option]

The commands and options that you can use with this tool are described in the following tables.

                                   
Commands and OptionsDescription

-h, --help

Display help on commands and option. For example,

specify: nw-recovery-tool --help-categories to get a list of all the valid category names.

-e, --export Export data or configuration.
-i, --importImport data or configuration.

-d, --dump-dir <path>

Path for the where data will be exported or imported from (for example, /var/netwitness/backup).

-C, --category <name>

Select components by category.

Valid category names are AdminServer, Archiver, Broker, Concentrator, Decoder, EndPointBroker, EndpointLogHybrid, ESAPrimary, ESASecondary, Gateway, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, UEBA, and Warehouse.

You can specify a single category or multiple categories if multiple categories are co-located on the same host. For example:

  • --category AdminServer for the Admin Server exclusively.
    --category AdminServer --category Gateway for the Admin Server and the Cloud Gateway.
  • --category ESAPrimary for the ESA Primary exclusively.
  • --category Broker for the Broker exclusively.
    --category Broker --category EndpointBroker for the Broker and the Endpoint Broker.

-p, --deploy-password <pwd>

Specify deployment password. This is only needed if the selected category or component includes Mongo (for hosts such as Admin Server, Endpoint Log Hybrid, or ESA Primary).

Required Conditions

Make sure that the following conditions are met:

  • Read the entire document before backing up any data. The document covers all deployment scenarios, so you want to make sure you have all the information required to back up and restore your implementation of NetWitness Platform before going through this process.
  • Run the NRT for both backup and recovery locally, on each system being backed up or restored. You cannot run the NRT on an external host, or back up or restore several hosts simultaneously. However, you can back up several components on the same host system simultaneously.
  • Export and import data on the same host. If a host fails and you need to build a new system, the new system must have the same identity parameters (i.e., the same IP address), and must be on the same version of NetWitness Suite
  • Make sure that there is adequate disk space in the backup location (/var/netwitness/backup is the recommended directory) before the export command in the nw-recovery tool is executed. Do not use a tmp directory because it fills up quickly and may cause the system to crash.

  • Check the sizing of the Malware disks and adjust them before you back them up. The following table shows you the maximum size of Malware databases that you can back up by hardware type with the actions you can take to reduce them to the maximum size.
    HostSource
    Hardware
    Target HardwareDatabaseMaximum
    Size for
    Backup
    Actions to Reduce
    Size to Backup
    Maximum
    MalwareSeries 4S Hybrid

    Series 6 Core

    /var/netwitness2.5TBConfigure a rollover.
    Purge data that you do not need from the database.
  • Restore to the exact ISO Image that each host had at the time of backup.
  • If you have multiple services co-located on a single host, include all the services in a single command string for the import and export commands in the nw-recovery tool.

1.) When you run the NRT, the Malware , Reporting Engine, and Postgresql services are stopped and restarted during both the backup (export) and restore (import) processes. Log and packet collection is not stopped.

Disaster Recovery Workflow

The following diagram shows the high-level Disaster Recovery tasks.

You only need to recover a host if it failed. This means that you can recover a single host, or any combination of hosts depending on which host or hosts failed.

The following diagram shows the tasks for:

  • Backup (perform as soon as possible and as frequently as possible).
  • Restore (only required if you need to restore your data).

Back Up and Restore Data for 11.x Hosts

The procedures for backing up and restoring data are different for NetWitness Server host systems and for component systems.

1.) Do not remove component hosts (that is any host other than the NW Server host) from the Hosts View (Admin >Hosts) from the user interface when you are performing the following disaster recovery procedure. 2.) You must retain (restore) the ‘Host name’ that existed prior to performing the disaster recovery procedure. 3.) Make sure that you record your master password and store it in a safe location so you can access the system in the case of Disaster Recovery.

Back Up and Restore Data on the 11.x NetWitness Server

If you are using shared storage to export data from multiple hosts (for example, a shared mount or drive), use host-specific subfolders for the path to the location of the exported files for each host, to avoid overwriting one host’s exported data with another. For example, you could use a path similar to --dump-dir /mnt/storage/<host-specific-name> for the path to the location of the exported files.

Back Up Data on a NetWitness Server Host

Perform this procedure on an existing, functional 11.x NetWitness Server host system.

  1. At the root level, type the following command:

    nw-recovery-tool --export --dump-dir /var/netwitness/backup --category AdminServer

    If a service is co-located with another category on the same host rather than on its own, dedicated host, you must include it in the command string. The Gateway and EndpointBroker can be co-located as show in the following examples:
    nw-recovery-tool--export --dump-dir /var/netwitness/backup --category AdminServer --category Gateway
    nw-recovery-tool--export --dump-dir /var/netwitness/backup --category Broker --category EndpointBroker

  2. Replace /var/netwitness/backup with the path to the location to which the data should be exported.

    1. Ensure that this location has sufficient space to store the backup data.
    2. The backup directory path should be located on the local host. However, the backup files could be located on a network mount or an external device.
  3. When you are prompted for the deployment administration password, enter the password, or include the following additional argument for the nw-recovery-tool command:

    --deploy-password <password>

    Use the existing deploy_admin password that was used when you first installed the host.

    The data is backed up on the NetWitness Server host in the location you set up in step 2 .

  4. Move the backed up data from the local host to an external server or a USB stick.

Restore Data on a NetWitness Server Host

  1. Re-image the NetWitness Server host using the same network configuration settings of the original host. For information about re-imaging the NetWitness Server host, see "Task 1 - Install 11.2 on the NetWitness Server Host" in the Physical Host Installation Guide for Version 11.2 Guide

    1. Optional If you need to establish network connectivity before you can fetch backup data, for example, if it is on a remote host, run the following script using the same IP address, subnet, gateway, DNS and domain information as the original host:

      netconfig --static --interface <name> --ip <address> --netmask <netmask> --gateway <gateway>

      For example:

      netconfig --static --interface eth0 --ip 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1

      Optional: To specify DNS server(s), include the following additional parameter:

      --dns <address>

      Optional: To set the local domain name, include the following additional parameter:

      --domain <name>

    2. (Optional) If you are using DHCP, run the following script:

      netconfig --dhcp --interface <name>

      For example:

      netconfig --dhcp --interface eth0

    3. Add the backup data to the backup directory path on the local host, for example:

      /var/netwitness/backup

  2. Run the nwsetup-tui command. This initiates the Setup program.

    During the Setup program, when you are prompted for the network configuration of the host, be sure to specify the same identical network configuration that was used for the original installation of 11.x on this host.

  3. When you are prompted, select install type option 3: Recover (Reinstall), click OK, and then enter the path to the backup directory containing the backup data.
  4. After the installation completes successfully, ensure that the host is running the exact same release and patch version of the data that was backed up:

    • If the data was on an 11.x system that was updated to a later patch release, update the host by following the instructions for updating systems offline in the update guide for the same patch version as what was previously running on the host (the exact release/patch version for which data was backed up).
    • If the data was on a major release version (for example, 11.x) that had not been updated to a later patch version, you do not need to update the host system.

  5. When the host is running at the correct version, run the following command on the NetWitness Server to restore data:

    nw-recovery-tool --import --dump-dir /var/netwitness/backup --category AdminServer

    If a service is co-located with another category on the same host rather than on its own, dedicated host, you must include it in the command string. The Gateway and EndpointBroker can be co-located as show in the following examples:
    nw-recovery-tool--import --dump-dir /var/netwitness/backup --category AdminServer --category Gateway
    nw-recovery-tool--import --dump-dir /var/netwitness/backup --category Broker --category EndpointBroker

  6. (Conditional) For customers using custom firewall rules (that is, replied "Yes" to the "Disable Firewall" nwsetup-tui prompt during installation), restore the /etc/sysconfig/iptables file from the backup copy located in the <dump-dir>/unmanaged/etc/sysconfig/iptables file.
  7. Reboot the NetWitness Server host.

Back Up and Restore Data on Other Component Hosts

Perform these procedures on each existing, functional 11.x component host system.

Back Up Data on a Component Host

  1. At the root level, type the following command:
    nw-recovery-tool --export --dump-dir /var/netwitness/backup --category <category name>

    where the category name is one of the following:
    AdminServer, Archiver, Broker, Concentrator, Decoder, EndPointBroker, EndpointLogHybrid, ESAPrimary, ESASecondary, Gateway, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, UEBA,or Warehouse

  2. 1.) Use the category that matches the host type. 2.) If services are co-located on a Component Host rather than on its own dedicated host, you must include it in the command string. For example, a Warehouse Connector resides on a Log Decoder host. The following is an example of this command string.
    nw-recovery-tool--export --dump-dir /var/netwitness/backup --category LogDecoder --category Warehouse

  3. (Optional) Replace /var/netwitness/backup with the path to the location to which the data should be exported
    1. Ensure that this location has sufficient space to store the backup data.
    2. The backup directory path should be located on the local host. However, the backup files could be located on a network mount or an external device.
  4. For Endpoint Log Hybrid and ESA Primary hosts, you can export application data that is stored in the database by running the following command:
    nw-recovery-tool --export --dump-dir /var/netwitness/backup --component mongo
    You can replace /var/netwitness/backup with the path to the location to which the data should be exported.
  5. 1.) Make sure that there is enough space in the export location for the files from the Mongo database. 2.) You can back up the Endpoint Log Hybrid or ESA Primary host data and Mongo database in a single command string. For example, nw-recovery-tool --export --dump-dir /var/netwitness/backup --category EndpointLogHybrid --component mongo

    When you are prompted for the deployment administration password, enter the password, or include the following additional argument for the nw-recovery-tool command:
    --deploy-password <password>

  6. For Malware, you can export application data from the Malware application database by running the following command:
    nw-recovery-tool --export --dump-dir /var/netwitness/backup --component postgresql
    You can replace /var/netwitness/backup with the path to the location to which the data should be exported.
  7. Ensure that there is enough space in the export location for the files from the Malware database.

  8. Move the backed up data from the local host to an external server or a USB stick.

Restore Data on a Component Host

  1. Re-image the component host using the same network configuration settings of the original host. For information about re-imaging a component host, see "Task 2 - Install 11.x on Other Component Hosts" in the Physical Host Installation Guide for Version 11.x Guide
  2. Optional If you need to establish network connectivity before you can fetch backup data, for example, if it is on a remote host, run the following script using the same IP address, subnet, gateway, DNS and domain information as the original host:
    netconfig --static --interface <name> --ip <address> --netmask <netmask> --gateway <gateway>
    For example:
    netconfig --static --interface eth0 --ip 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1
    Optional: To specify DNS server(s), include the following additional parameter:
    --dns <address>
    Optional: To set the local domain name, include the following additional parameter:
    --domain <name>

    1. (Optional) If you are using DHCP, run the following script:
      netconfig --dhcp --interface <name>
      For example:
      netconfig --dhcp --interface eth0
    2. Add the backup data to the backup directory path on the local host, for example, /var/netwitness/backup.
  3. Run the nwsetup-tui command. This initiates the Setup program.
  4. During the Setup program, when you are prompted for the network configuration of the host, be sure to specify the same identical network configuration that was used for the original installation of 11.x on this host.

  5. When you are prompted, select install type option 3: Recover (Reinstall), click OK, and then enter the path to the directory containing the backup data.
  6. After completing the nwsetup-tui command setup, you must re-install the appropriate categories (except EndpointLogHybrid) on the host using the Install command from the Hosts View in the NetWitness Platform User Interface.
    For the EndpointLogHybrid category, you must use the orchestration-cli-client on the Admin Server to install the Endpoint Log Hybrid services. Run the following command:

    orchestration-cli-client --hostaddr-as-id -i -o <host IP Address> --category EndpointLogHybrid --version <version>

    For example:

    orchestration-cli-client --hostaddr-as-id -i -o 192.168.200.83 --category EndpointLogHybrid --version 11.2.0.0

    The version number must match the version of the media that was used to re-image the host.

  7. After the service installation completes, ensure that the host is running the exact same release and patch version of the data that was backed up:
    • If the data was on an 11.x system that was updated to a later patch release, update the host by following the instructions for updating systems offline for the same patch version as what was previously running on the host (the exact release/patch version for which data was backed up).
    • If the data was on a major release version (for example, 11.x) that had not been updated to a later patch version, you do not need to update the host system.
  8. When the host is running at the correct version, return to the root level of the component host and run the following command to restore data:
    nw-recovery-tool --import --dump-dir /var/netwitness/backup --category <category name>

    If services are co-located on a Component Host rather than on its own dedicated host, you must include it in the command string. For example, a Warehouse Connector resides on a Log Decoder host. The following is an example of this command string.
    nw-recovery-tool--import --dump-dir /var/netwitness/backup --category LogDecoder --category Warehouse

  9. For EnpointLogHybrid and ESAPrimary systems, you can import application data to be restored by running the following command:
    nw-recovery-tool --import --dump-dir /var/netwitness/backup --component mongo
  10. When you are prompted for the deployment administration password, enter the password, or include the following additional argument for the nw-recovery-tool command:
    --deploy-password <password>

  11. For Malware, you can import application data from the Malware application database to be restored by running the following command:
    nw-recovery-tool --import --dump-dir /var/netwitness/backup --component postgresql
  12. For a Decoder, Log Decoder , Concentrator, Archiver, Network Hybrid, or Log Hybrid configured with external storage (JBOD / SAN /Unity / Powervault):
    1. Scan the <dump-dir>/unmanaged/etc/fstab file for devices with mount points that do not exist in the system /etc/fstab file.

    Important: If you are migrating to new host hardware (that is a new Decoder, Log Decoder , Concentrator, Archiver, Network Hybrid, or Log Hybrid host), before you proceed to the next step you must:
    1. Power off the old hardware host and the external storage device attached to it.
    2. Attach the external storage device to the new host hardware.
    3. Power on both the new host hardware and the external storage device attached to it.

    1. Complete the following steps for each device in the backup copy of <dump-dir>/unmanaged/etc/fstab.
      1. Verify that the corresponding device is present and attached. If it not attached, attach it. If the device is no longer applicable, skip it and go to the next device.
      2. Verify that the mount point directory exists on the file system. If it does not exist, create the directory with the mkdir <path> command.

      3. Add the fstab entry from the backup copy to the system /etc/fstab file.
    2. Run the following command on each host.
      mount -a
  13. (Conditional) For customers using custom firewall rules (that is, replied "Yes" to the "Disable Firewall" nwsetup-tui prompt during installation), restore the /etc/sysconfig/iptables file from the backup copy located in the <dump-dir>/unmanaged/etc/sysconfig/iptables file.

  14. Reboot the component host.

Hardware Refresh Only - Use Additional Space in New Hardware Hosts

Refer to the RSA NetWitness Platform Core Tuning Guide (https://community.rsa.com/docs/DOC-95938

) for instructions on how to use all the space you have available on your new hardware.

You are here
Table of Contents > Disaster Recovery

Attachments

    Outcomes