AWS Upgrade: Backup Instructions

Document created by RSA Information Design and Development Employee on Apr 12, 2019Last modified by RSA Information Design and Development Employee on Jul 20, 2020
Version 14Show Document
  • View in full screen mode

Backing up your configuration data for all your hosts from 10.6.6.x is the first step in upgrading from 10.6.6.x releases to

Note: It is important that you place Custom Certificate files and any other certificate authority (CA) files in the /root/customcerts folder to make sure that these certificate files are backed up. Your custom certificate files will be automatically restored during the upgrade process. After upgrading to, the custom certificate files will be located in /etc/pki/nw/trust/import. For more information about backing up these types of files, see step 1 in For All Host Types

Caution: 1) These services are not supported in the 10.6.6.x backup and upgrade process.
- All in One servers
- Malware Analysis Co-Located on the NetWitness Server
- Standalone Warehouse Connector
2) There is a known issue if you have Active Directory users configured in 10.6.6.x. You have two options to address this issue:
• Apply the patch before you back up your data for the 11.4 upgrade.
• If you failed to apply the patch, you can apply the patch immediately after you upgrade to 11.4.

The following types of hosts can be backed up and are automatically restored during the upgrade process:

  • NetWitness Server (may include Malware Analysis, NetWitness Respond, Health and Wellness, and Reporting Engine)
  • Archiver
  • Broker
  • Event Stream Analysis (including Context Hub and NetWitness Respond database)
  • Concentrator
  • Log Decoder
  • Packet Decoder
  • Virtual Log Collector

The following types of files are automatically backed up but must be restored manually after the upgrade process:

  • PAM configuration files: For information about restoring the PAM configuration files, see "Task 5 - Reconfigure Plugable Authentication Module (PAM) in", in the "Global" section of the Post Upgrade Tasks.

  • /etc/pfring/mtu.conf and /etc/init.d/pf_ring: To restore these files, you must manually retrieve them. The /etc/pfring/mtu.conf files will be located in /var/netwitness/database/nw-backup/restore/etc/pfring/mtu.conf, and the /etc/init.d/pf_ring files will be located in /var/netwitness/database/nw-backup/restore/etc/init.d/pf_ring. For information about how to restore these files, see "(Conditional) Task 2 - Restore Files for 10G Decoder" in the "Hardware Related Tasks" section of Post Upgrade Tasks.

Note: If you have problems during the backup or upgrade processes and you lose data, you can recover the data and start the process again. For information about recovering lost data, see "Recover Data After System Failure" in the System Maintenance Guide.

The following diagram shows the high-level task flow of the steps you perform to back up your hosts.


Task 1 - Set up an External Host for Backing up Files

You must set up an external host to use for backing up files. The host must be running Centos 6 with connectivity through SSH to the NetWitness Platform stack of hosts.

Make sure that the host names for systems to be backed up are resolvable on the backup host machine, either by DNS or listed in the /etc/hosts file.

Note: These scripts are designed to run on CentOS 6 only. You must execute these scripts on CentOS 6 machines.

There are several scripts that you run during the backup process. You must download the zip file that contains the scripts ( or later) from RSA Link at this location: and copy it over to your CentOS 6 backup system. Extract the zip file to access the scripts. The scripts are:

  • Creates the all-systems file, which contains a list of all your NetWitness Servers and host systems to be backed up.
  • Automates sharing keys between systems you are backing up and the backup host system so that you are not prompted for passwords multiple times.
  • Performs the backup of your hosts.

Note: The backup scripts do not support backing up data for STIG-hardened hosts.

Task 2 - Create a List of Hosts to Back up

The script that you use to back up your files depends on the all-systems and all-systems-master-copy files, which contain a list of hosts that you want to back up. The all-systems-master-copy file contains a list of all your hosts. The all-systems file is used for each backup session, and contains only those hosts which are being backed up for a particular session. You must run the script to generate these files. RSA recommends that you back up your hosts in groups, and not all at once. The recommended order and grouping of hosts for backup sessions is shown in the following diagram:


Limit each backup session to five hosts to make sure that you do not run out of space for the backup files. You create all-systems files for your backup sessions by using the all-systems-master-copy file as a reference, and then manually edit the all-systems file to contain specific hosts.

To generate the all-systems and all-systems-master-copy files:

  1. From the host on which you are running the backup process, make the script executable by running the following command:

    chmod u+x

  2. At the root level, run the following script:

    ./ <IP-Address-of-NetWitness-Admin-Server>

    You will be prompted for the password for each host system once per host.
    This script saves the all-systems file and the all-systems-master-copy file to /var/netwitness/database/nw-backup/.

  3. Validate that the all-systems and all-systems-master-copy files were generated and that they contain the right hosts.
  4. Edit the all-systems file to contain only the systems you are backing up. You can do this by using the all-systems-master-copy file as a reference, and then opening the all-systems file in an editor (such as vi) and modifying it to include only the systems you want to back up.

    Note: If you use vi, be sure to include the path to the location of the all-systems file.

Here is an example of an all-systems-master-copy file:


And here is an example of an all-systems file based on the all-systems-master-copy file that could be used in the first backup session:


Troubleshooting Information

  • Be sure to save copies of the all-systems and all-systems-master-copy files in a safe location.

    Follow these recommendations:

    • Do not edit the all-systems-master-copy file.
    • If you create several different versions of the all-systems file (for example, for several backup sessions), be sure to remove pre-existing entries from the file so that the file contains only those hosts that are currently being backed up.

      For more information, see Post Backup Tasks.

  • If any host systems are down while you are running the script, the script creates a list of hosts for which it cannot find information. After the script completes and the all-systems file is created, you must edit the all-systems file manually and add the missing information for these hosts.
  • The script generates a list of hosts that were defined in the NetWitness Platform user interface. Make sure that all hosts and services are provisioned properly. If any hosts or services are not provisioned properly, they will not be backed up. RSA recommends that when you add hosts and services to NetWitness Platform, you use the NetWitness Platform user interface to Msure that they are provisioned properly. However, if there are any hosts or services that were not defined in the user interface, you must add them to the all-systems file manually.
  • At the end of the script, the script will check for any differences between the systems that the NetWitness Server has listed, and the ones for which the script was able to find all the required information. If any Node ID’s or system names are listed as missing, verify the existence of those systems, that their services are all running, and that they are properly communicating with the NetWitness Server. (Any Windows Legacy Collectors or AWS Cloud Collectors will not be added to the all-systems file, and may account for discrepancies. DO NOT add these items to the all-systems file manually.)
  • If the syntax in the all-systems file is incorrect, the script will fail. For example, if there is an extra space at the beginning or the end of a host entry, the script will fail.

Task 3 - Set up Authentication Between Backup and Target Hosts

RSA recommends that you run the script to automate sharing keys between the backup host and the host systems.

Note: If you have SSH keys that are protected with pass phrases, you can use ssh-agent to save time. For more information, refer to the man page for ssh-agent.

  1. On the external backup host system, make the script executable by running the following command:

    chmod u+x

  2. At the root directory, run the following command, where <path-to-all-systems-file> is the path to the directory where the all-systems file is stored: <path-to-all-systems-file>

  3. You are prompted for the password once per host, but you will not need to enter it repeatedly later during the backup process.

Task 4 - Check for Backup Requirements for Specific Types of Hosts

After you create the all-systems file to use for backup, you must check to see if any of the hosts listed in the file have requirements that must be met before you run the backup process.

For All Host Types

Perform the following steps for all host types:

  1. On the NetWitness Server, place Custom Certificate files and any other certificate authority (CA) files in the /root/customcerts folder to make sure that these certificate files are backed up. Your custom certificate files that are placed in this directories will be automatically restored during the upgrade process. After upgrading to, your custom certificate files will be located in /etc/pki/nw/trust/import.
    You can convert CA certificates and keys to different formats to make them compatible with specific types of servers or software using OpenSSL. For example, you can convert a normal PEM file that would work with Apache to a PFX (PKCS#12) file and use it with Tomcat or IIS. To convert the files, SSH to the NetWitness Server and run the following command strings to perform the conversions listed.
    Convert a DER file (.crt .cer .der) to PEM
    openssl x509 -inform der -in certificate.cer -out certificate.pem
    Convert a PEM file to DER
    openssl x509 -outform der -in certificate.pem -out certificate.der
    Convert a PEM Certificate File and a Private Key to PKCS#12 (.pfx .p12)
    openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt
    Convert a PKCS#12 File (.pfx .p12) Containing a Private Key and Certificates to PEM
    openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes

Note: Add the following qualifier to the command string to:
-nocerts convert private keys exclusively.
-nokeys convert certificates exclusively.

  1. Manually record any custom configurations made to CentOS 6 (for example, driver customizations) for restoration after you update to CentOS 7. Custom configurations to CentOS 6 are not automatically backed up and restored.

For Decoder, Concentrator, or Broker Hosts: Stop Data Capture and Aggregation

In addition to the tasks described in For All Host Types, for Decoder, Concentrator, or Broker hosts, stop data capture and aggregation on all the systems that you are backing up. For instructions, refer to Appendix B. Stopping and Restarting Data Capture and Aggregation

Log Collectors (LC) and Virtual Log Collectors (VLCs): Run

Caution: This task stops log collection so you must perform this step immediately before you upgrade to minimize the loss of event collection. Complete this task in accordance with the backup and upgrade tasks in this guide.


You need the following information before you prepare LCs and VLCs for upgrade.

  • If Lockbox was initialized on the LC and VLC, you must know the Lockbox password. It is required to reconfigure the Lockbox after upgrade.
  • If you set the password for logcollector user for RabbitMQ , you must know the password so you can set it again after the upgrade.

Prepare LCs and VLCs for Upgrade

  1. SSH to the Log Collector.
  2. Submit the following command string.

    # /opt/rsa/nwlogcollector/nwtools/ --prepare

    This command:

    • Stops the Puppet Agent service.

    • Disables the file collection accounts (“sftp” and all users in the group “upload”) used for uploading log files to the Log Collector. The log files accumulate on the event sources until the Log Collector has been upgrade to

    • Stops all the collection protocols in the Log Collector service.

    • Saves the list of Plugin accounts and RabbitMQ accounts.

    • Configures the RabbitMQ server so that new events cannot be published to it any longer. Consumers of events in the queues, such as shovels and Log Decoder Event Processors, will continue to run.

    • Waits until the Log Collector queues are empty.

    • Stops the Log Collector service.

    • Creates a marker file indicating that the Log Collector has been successfully prepared for upgrade.

Troubleshooting Information

The script:

  • Sends informational, warning, and error messages to the console.
  • Saves a session log in the /var/log/backup/ directory.

You must fix any of the following errors and resume the preparation. Contact RSA Customer Support ( for assistance.

  • Log Collector queues with events but without consumers are found.
  • Unable to stop the Puppet Agent service.
  • Unable to stop a collection protocol in the Log Collector service.
  • Unable to block event publishers to the RabbitMQ server.
  • Unable to or taking too long for queue events to be consumed. The script makes 30 attempts waiting for the events to be consumed. After each attempt, it sleeps for 30 seconds.
  • Unable to stop the Log Collector service.

For more information about troubleshooting, see Appendix A. Troubleshooting

For Integrations with Web Threat Detection, Archer Cyber Incident & Breach Response or NetWitness Endpoint: List RabbitMQ Usernames and Passwords

On the 10.6.6.x host, on the NetWitness Server host, you must get a list of all RabbitMQ usernames and passwords so that after you perform the upgrade, you can restore RabbitMQ user accounts.

To get a list of RabbitMQ usernames and passwords, run the following command:

rabbitmqctl list_users >> /root/rabbitmq_users.txt

To restore RabbitMQ user accounts, refer to Task 2 - For Integrations with Web Threat Detection, NetWitness SecOps Manager or NetWitness Endpoint Configure Mutually Authenticated SSL in Post Upgrade Tasks.

For Bluecoat Event Sources

Bluecoat ProxySG event sources use FTPS protocol to upload log files to the Log Collector (LC) and Virtual Log Collector (VLC). The event source documentation contains the steps to configure VSFTPD service on the LC and VLC.

  • If key material exists in /root/vsftpd/ directory in 10.6.6.x, this material area will be backed up and restored. If the material was in another location, you must back it up and restore it manually.
  • If the /etc/vsftpd/vsftpd.conf file exits in 10.6.6.x, it is backed up and restored.

Task 5 - Check for Adequate Space for the Backup

You can run the backup test script to check the amount of disk space that is required for the backup using the -t option described in Test Options. You run the script without actually backing up files or stopping any services. RSA recommends that you perform this step to make sure that you provide adequate space for the backup so that the backup captures all your data.

To check for adequate disk space:

  1. Make the backup script executable by running the following command:
    chmod u+x
  2. Run the following command at the root directory level:
    ./ -t
    The output displays the amount of disk space that is required for the backup.

Note: The ./ –t command runs with the -d option by default. However, if you are looking for more accurate disk space results, you can override the -d option by using -D. Using the -D option will show how much space is required on each host for the data that will be backed up, but does not show how much space is available. If there is not enough space available, the -D option will throw an error. If you want to know how much space is available on the target host, you must run the df -h command on the host.

The following figure shows an example of the output from using the -t option.


Task 6 - Back up Your Host Systems

Before you run the backup script to do the actual backup, be sure that you have plenty of space. To back up your hosts, you run the script using the -u option. This option is required for upgrading to

Note: The script will stop services as it runs. However, you can stop services manually before you run the script if needed.

When you run the backup script, you can choose from several options that are described in the following sections.

./ [-u -t -d -D -l -x -e] <external-mnt> -b <backup file path>

General Options

-u : This option is required for upgrading to 11.4. Enables the upgrade flag to run backup for upgrading to 11.4. It also enables disk space check (-d), backing up reporting engine reports (-r) and stores backup content locally (-l). Default: (no)

-d : enables disk space check in 'fast' mode (quick estimate of space using uncompressed data). Default: (no)

-D : enables disk space check in 'full' mode (estimate of space using compressed data, ~10X slower). Default: (no)

-l : stores backup content locally on each host (automatically set if -u is used). Default: (no)

-e <path to mount point> : copies backup files of all devices onto an external mount point. Default: (/mnt/external_backup)

-x : move all backup files to an external mount point. Default: (no) - COPY

-b <path to write backups> : path to the location for storing backup files on a backup server. For upgrading to 11.4, please use the default location! Default: (/var/netwitness/database/nw-backup)

Note: Do not change the backup path in upgrade (-u) mode.

Advanced Content Selection Options

-c : back up Colocated Malware Analysis on SA servers. Default: (no)

-i : back up IPDB data (/var/netwitness/ipdbextractor). Default: (no)

-m : back up Malware Analysis File Repository. Default: (no)

-r : back up Reporting Engine Report Repository (automatically set if -u is used). Default: (no)

-v : back up system logs (/var/log). Default: (no)

-y : back up YUM Web Server & RPM Repository. Default: (no)

-S : If set: DISABLES back up of SMS RRD files. Default: (not-set)

-C : If set: DISABLES back up of Context-Hub configuration and database. Default: (not-set)

-E : If set: DISABLES back up of ESA Mongo database. Default: (not-set)

Test Options

-t : performs script test run for disk space check only. Services are not stopped and excludes execution of backup. Can be combined with (-d) or (-D) and other flags. Default: (-t)

For example, the command:
would run the backup with options as set in the Header of the script itself.
OR, the command:
./ -ue /mnt/external_backup
would run a normal backup using the backup path defined in the script, with the following options:

-u : enables the upgrade flag to run backup for upgrading to 11.4. It also enables disk space check (-d), backing up reporting engine reports (-r) and stores backup content locally (-l). Default: (no)

-e : Copy the backup files to external mount point, mounted on /mnt/external_backup

For Help: ./ –h

When you run the script, the following text is displayed at the top of the script:

Caution: RSA nw-backup script backs up configuration files, data, and logs on the options provided in the script. It tars the content, with options to store the backup files on the backup server, move or copy them to external storage on a mount point (USB/NFS/SMB), or SCP them back to the target host.
This backup script has been qualified on the following versions of Security Analytics:
Use of this script on any other versions of the product may not give expected results and may not be supported by RSA Customer Service.Note: All non-RSA custom files, scripts, Cronjobs and other important files should be placed in /root, /home/'user', OR /etc to be included in the backup.

To run the backup script to back up your hosts:

  1. Make sure that the all-systems file contains only the hosts to back up. For information, see Task 2 - Create a List of Hosts to Back up.
  2. Make the backup script executable by running the following command:
    chmod u+x
  3. Begin the backup process by running the following command at the root directory level:
    ./ -u <additional options as needed>

    Note: You must use the -u option so that your files will be restored correctly during the upgrade to

    When the text "Backup completed with no errors" is displayed, the backup has completed successfully.

    A log file, with a name similar to the following example, is created in the backup directory which provides information on the files being backed up:

  4. When the backup has completed, to make sure that the intended files were backed up, you can run the following command to see a list of all the files that were backed up:
    tar -tzvf hostname-ip-address-backup.tar.gz
    The following archive files are created:
    For all hosts:

    tar checksum files
    For NetWitness Servers:

    tar checksum files
    For ESA Hosts:

    tar checksum files

The archived files are located in the /var/netwitness/database/nw-backup directory. If any of the tar files appear smaller than expected, open them to be sure that the files were properly backed up.

Post Backup Tasks

Task 1 - Save a Copy of the all-systems File and the Backup Tar files

Make copies of the all-systems file, the all-systems-master-copy file, and the backup tar files and put the copies in a secure location. You cannot regenerate these files after you upgrade the NetWitness Server (specifically the Admin service) to

Task 2 - Ensure Required Backup Files Were Generated

After you run the backup scripts, several files are generated. These files are required for the upgrade process. Before you begin the upgrade process, you must make sure that the required backup files are on the hosts that you are upgrading, and that you perform the following tasks.

The following files are generated on all hosts by the backup scripts:

  • all-systems
  • all-systems-master-copy
  • appliance_info
  • service_info
  • <hostname>-<host-IP-address>-backup.tar.gz
  • <hostname>-<host-IP-address>-backup.tar.gz.sha256
  • <hostname>-<host-IP-address>-root.tar.gz
  • <hostname>-<host-IP-address>-root.tar.gz.sha256
  • <hostname>-<host-IP-address>

In addition to the files listed above, the following files will be generated on NetWitness Server and ESA hosts:

  • <hostname>-<host IP address>-mongodb.tar.gz
  • <hostname>-<host IP address>-mongodb.tar.gz.sha256

The backup script will also generate the following controldata-mongodb.tar.gz files.

Note: The backup script copies the following files from all ESA hosts to the NetWitness Server host's backup path .

  • <esa hostname>-<esa hostip>-controldata-mongodb.tar.gz
  • <esa hostname>-<esa hostip>-controldata-mongodb.tar.gz.sha256

(Conditional) Task 3 - For Multiple ESA Hosts, Copy mongodb tar files to Primary ESA Host

If you have multiple ESA host systems in your enterprise, copy the following two files from each ESA host to the /opt/rsa/database/nw-backup/ directory on the Primary ESA host system (the host that has the ContextHub service running on it) :

  • <hostname>-<host-IP-address>-mongodb.tar.gz
  • <hostname>-<host-IP-address>-mongodb.tar.gz.sha256

Task 4 - Ensure All Required Backup Files are on Each Host

Before you upgrade to, make sure that the appropriate files exist on the hosts that you are upgrading as described in the following lists.

There should be note here mentioning default backup path locations for that user knows where to go and check these files.

Note: The default paths for backup files are:
- NetWitness Server hosts:/var/netwitness/database/nw-backup
- ESA hosts:/opt/rsa/database/nw-backup
- Malware hosts: /var/lib/rsamalware/nw-backup

Required Files for NetWitness Servers

  • all-systems-master-copy
  • <hostname>-<host-IP-address>-backup.tar.gz
  • <hostname>-<host-IP-address>-backup.tar.gz.sha256
  • <hostname>-<host-IP-address>-root.tar.gz
  • <hostname>-<host-IP-address>-root.tar.gz.sha256
  • <hostname>-<host-IP-address>
  • <hostname>-<host-IP-address>-mongodb.tar.gz
  • <hostname>-<host-IP-address>-mongodb.tar.gz.sha256
  • <esa-hostname>-<esa-host-IP-address>-controldata-mongodb.tar.gz
  • <esa-hostname>-<esa-host-IP-address>-controldata-mongodb.tar.gz.sha256

Required Files for ESA Hosts

  • all-systems-master-copy
  • <hostname>-<host-IP-address>-backup.tar.gz
  • <hostname>-<host-IP-address>-backup.tar.gz.sha256
  • <hostname>-<host-IP-address>-root.tar.gz
  • <hostname>-<host-IP-address>-root.tar.gz.sha256
  • <hostname>-<host-IP-address>
  • <hostname>-<host-IP-address>-mongodb.tar.gz
  • <hostname>-<host-IP-address>-mongodb.tar.gz.sha256

Required Files for All Other Hosts

  • all-systems-master-copy
  • <hostname>-<host-IP-address>-backup.tar.gz
  • <hostname>-<host-IP-address>-backup.tar.gz.sha256
  • <hostname>-<host-IP-address>-root.tar.gz
  • <hostname>-<host-IP-address>-root.tar.gz.sha256
  • <hostname>-<host-IP-address>

Note: The following files are located in the <hostname>-<host-IP-address>-backup.tar.gz tar on all hosts:

Note: The paths to the location of the backup and restore files for iptables, NAT configurations, user accounts, and crontab entries are shown in the following list:
Backup paths:
BUPATH=/opt/rsa/database/nw-backup for the ESA Correlation Engine
BUPATH=/var/lib/rsamalware/nw-backup for the Malware Service
BUPATH=/var/netwitness/database/nw-backup for all other services
Restore locations:
BUPATH/restore/etc/sysconfig for Iptable rules
BUPATH/restore/etc/sysconfig for NAT configurations
BUPATH/restore/etc for Crontab entries
BUPATH/restore/etc for User Accounts (users are located in the passwd file, and groups are located in the group file. These are not restored during the upgrade process but can be restored manually.
BUPATH/restore/etc/ntp.conf for NTP configurations (must be restored using the NetWitness Platform UI)

You are here
Table of Contents > Backup Instructions