Updated for Version 11.5+
Need to remotely backup your NetWitness hosts to a central location, to satisfy Disaster Recovery Requirements, perform a Tech Refreshes, or to be prepared for RMA replacement of a device.
Solution – A Wrapper for NRT
Building off the framework of the original nw-backup scripts written for 10.x backup/restore and migration to 11.x, a new set of version 11 scripts has been written as a "wrapper" to the built in NRT functionality of NetWitness Version 11.2+.
(Please note that this is not an officially supported solution by NetWitness, but can be used by customers as a possible solution, at their own risk)
The solution consists of 3 scripts (all run from the NW Admin server (node-zero)) and an example nw-base.nrt file:
- get-all-systems11.sh – Generates an inventory file of all hosts in the NW server deployment.
- ssh-propagate11.sh – Generates an ECDSA key pair on the NW Server and propagates the public key to all hosts.
- nw-backup11.sh – The main backup script.
- nw-base.nrt – An NRT script to add custom files to the backup directory as part of the backup process.
Process Walkthrough –
Copy the attached zip file to the NW Admin Server host (node-zero) and unzip:
unzip nw-backup11.zip -d /root/scripts
chmod +x *.sh
Step 1: Run get-all-systems11.sh
./get-all-systems11.sh [ -u <USER> ][ -p <PASSWORD> ][-d /home/path ][ -b </backup/path> ][-n <hrs>][-h]
Note: All command-line options are optional.
With no options selected, the script will use options set within the file itself.
-b </backup/path> : Path to store the all-systems file and logs. Default:(/var/netwitness/nw-backup)
-u <username> : User acct to use for non-root SSH access to hosts during backups. Default:(root)
-p <password> : Password for <username> Default: (ask)
-d </home/path> : Base home directory path (/home) for <username> Default:(/var/netwitness/nw-backup)
-n <hrs> : Skip check for new systems if current new-systems is less than <hrs> old.
-h : Print this help information.
Run on the NW Admin Server. Creates the /var/netwitness/nw-backup dierctory (or the directory passed with the -b option), then using a combination of mongo and salt queries, will create the all-systems file in that directory. The all-systems file is used by the other scripts , with entries that contain the following in comma separated format:
DeviceType = NRT Category Type (i.e. AdminServer,Broker,Concentrator,Decoder, etc.)
Hostname = Short hostname (not FQDN)
IPAddress = Management Interface IP address of host
MinionID = Unique Salt MinionID of host
SerialNumber = Device Serial Number for Reference and Support
The script is designed to run from the cron on a regular basis (if you are in a dynamic environment were systems are added/removed on a regular basis), it has a 30 second timeout on the one question it asks.
If changes to the environment are detected, generation of new-systems and/or old-systems files will occur. The new-systems file can be used by other scripts for running specific targeted actions against the newly installed hosts. If a system is “offline” or has been removed from the UI, the old-systems file will have the entries for those hosts, so information about them is not lost.
Step 2: Run ssh-propagate11.sh
Note: All command-line options are optional.
Run with no options, script uses the /var/netwitness/nw-backup/all-systems file to target
all hosts and copy the root users ecdsa-521 bit public ssh key to all hosts
./ssh-propagate.sh [ -u <user> ] [-c] [ -b <path> ] [ -t <target> ]
-b <path> : path to the location of all-systems file. Default: (/var/netwitness/nw-backup)
-u <user> : User to propagate keys to on all nodes (must exist on NW Server). Default: (root)
-c : Used with -u <user> for non-root user, create user on remote hosts, if needed.
-t <target>: Target hosts (new-systems or anything greppable from all-systems. Default: (ALL)
Run on the NW Admin Server. Performs the following (depending on options):
- Verifies the designated user account already exists on the NW Server
- Updates the /etc/hosts file with host shortnames using entries from all-systems file
- Generates an ecdsa-521bit ssh key for root (or the specified user) if it does not exist
- Iterates through the target list (default is ALL hosts in the all-systems file) and performs the following:
- Tests ssh connectivity to the host (host responding on port 22)
- Verifies the user exists on the remote host, if not and the -c option is selected
- Creates remote user acct
- Sets password to same as user acct on NW server
- Tests SSH key authentication to host as the user specified, if auth fails:
- Copies the user ecdsa public key string to the ~/.ssh/authorized_keys file
- Adds the target host fingerprint to the user's ~/.ssh/known_hosts file on the NW server
- Verifies ssh connectivity via ssh-key authentication
Step 3: Modify the nw-base.nrt file
The default /etc/netwitness/recoverytool/nw-base.nrt file, distributed file with 11.x systems, only contains the following entries:
# unmanaged files
# for azure
The “STASH” entries are NOT restored during an NRT import (recovery), but are available for reference in the /var/netwitness/backup/unmanaged folder, after the import. The included nw-base.nrt file has an expanded list of stash files and directories to include files used at several customer installations. To Fully restore functionality, these files need to be available after the restore.
Extended nw-base.nrt supplied with the scripts:
# unmanaged files
stash /root/ (note: this backs up ALL file/folders in /root including /root/.ssh)
# for azure
Edit the file to include any additional locations (files or directories) that contain customizations in your deployment, then save the file in the same directory as the nw-backup11.sh script. The backup script will verify the file on each system matches your modified file and if not, will automatically copy the modified file to each host before running NRT on that host.
Step 4: Run nw-backup11.sh
Note: All command-line options are optional.
With no options selected, the script backup ALL devices listed in the
/var/netwitness/nw-backup/all-systems file(except any commented(#) out)
and copy the backup files to the /var/netwitness/nw-backup/<date>/ directory
on the NW Server.
./nw-backup11.sh [-b <Local backup path>] [-m <remote transfer mode>]
[-l <local mount point (NFS only)>] [-p <remote backup path>] [-s <remote server IP>]
[-d <destination path>] [-t <DeviceType>] [-u <scp user>] [-M]
-b <local NRT backup path> : path for NRT backup files. Default: (/var/netwitness/backup)
-m <mode> : remote transfer mode (scp or nfs). Default: (scp)
-p <NetWitness Server backup path> : Path on NW server for logs and location of all-systems
file. Default: (/var/netwitnes/nw-backup)
-d <Destination backup path> : Path on destination server to move completed backup files to
via nfs or scp. Default: (/var/netwitness/nw-backup)
-s <remote server IP> : Destination server IP address for storing completed backup files,
transferred via nfs or scp. Default: (NW Server IP)
-u <SCPUser> : User acct for SCP transfers of completed backups, user must exist on all
target systems and on Destination server. Default: (root)
-l <mount_point> : local mount point for NFS share. Default: (/mnt/backup)
-t <devType> : backup ONLY specific device type or device Default: (all)
core (Broker, Concentrator, Decoder, LogDecoder, Archiver,
LogCollector(vlc), NetworkHybrid, LogHybrid)
nonw (all devices except AdminServer)
nwonly (AdminServer only)
esaonly (all ESA devices only)
endpoint (all endpoint devices EndpointHybrid, EndpointLogHybrid, Gateway)
-M : Exclude the Malware Analysis File Store.
-S : Stop LogCollector and/or Reporting Engine during backup. Default: (don't stop)
Note: NRT normal operation would stop these services, not stopping will affect:
LogCollector - Will not have latest tracking data for some logs.
Reporting Engine - Some live chart data and alert status data may be lost.
- If using SCP to a server other than the NW Admin Server, the copy uses the "-3" option and makes the transfer of the backup file via the NW Admin Server, so ONLY the NW Admin server needs to have SSH key authentication configured to all hosts.
- Make sure the modified nw-base.nrt file is in the same directory you are running the nw-backup11.sh script from, the script will hash that file, and verify that hash against the file on each server, if they do not match, it will copy the file in the script directory (where you ran the nw-backup11.sh script from) to the remote host, before triggering the NRT backup on that host. If you do not want to make any modifications to the default file, don’t include the file in the script run directory.
- Deploy Admin password is programmatically called, so no exposure of password in the scripts
- The /var/netwitness/nw-backup/all-systems file can be used for a myriad of other scripting calls, or for targeting a specific type of host, especially when using “salt” commands.
- If there are issues with backups failing due to "0" sized files, this is due to the way the tar command in the nw-recovery-tool script checks the return codes (i.e. If files being backed up change during the tar process). Please refer to the nw-recovery-tool fix.txt file for a temporary workaround until 11.5 version, which has the fix applied.
RESTORE - Follow the published backup/restore document for the version of NW you are running, The only change will be that the file is archived in a tar file, and after copying the file to the /var/netwitness/backup directory on a NEW host, run tar -xvf <backup-file-name>.tar to extract it there. Then follow the remaining instructions for running nwsetup-tui and nw-recovery-tool --import
- Previous versions of the backup script were dependant on the Device-Type, listed in the all-systems file, being a single type (i.e. AdminServer, Broker, Decoder, etc.) and was based off the single "installed services" entry in mongo for each host.
- NetWitness 11.4+ introduced the ability to add additional services to hosts (i.e. add New Health & Wellness(Search) to a Broker, or Endpoint Server to a LogHybrid), so now multiple "installed services" are listed for these hosts.
- The get-all-systems11.sh script, has been updated to handle this scenario by concatenating the installed services together, separated by a '+'. In the above examples the all-systems would show the Device Type listed as Broker+Search and Endpoint+LogHybrid. This allows continued use of the entries in all-systems to be used as a reference by other scripts when looking for specific services.
Non-root User for SCP
- The backup scripts now support using a non-root user for the SCP copy of the backup files from the host being backed up, to the NW Server (or to an external destination server). If your environment does not allow direct login as root via ssh, or the host you are copying the backups too requires a non-root user, then this will allow for that scenario.
- The backup user is required to have full access to "read" the files written by NRT during the backup, and the ability to create directories and files on the destination server.
- Run with the -u <username> option, if the user account does not exist on the NW Server, will prompt to create the account.
Default Settings for the account will be the following:
Example: using nwbackup as the username and not specifying analternate home directory (-d)
password:(you will be prompted to enter)
group: nwbackup (same name as the username)
home: /var/netwitness/nw-backup (or what the -b <bupath> option is set to)
Using the -d <homepath> will create and alternate home directory path (ex: /home)
- If preferred, the user account can be created manually before running the script or use an existing account. Owner:Group of the backup directories will be set to username:username
If copying the backup to a remote (non-netwitness) destination server, the user MUST exist on that server and have write permissions to the destination path backup files will be stored.
- Run with the -u <username> option, create and propagate the ecdsa-521 ssh key for this user to all other nodes in the all-systems file.
If user does not exist on a node, adding the -c option will allow the script to create the remote user and set the password the same as the local user on the NW server, as well as copying the ssh key to that host
- Run with the -u <username> uses designated username for all copying of backup files from nw nodes being backed up, to the designated destination (NW Server or external host).
User MUST exist on the nw nodes and have read permissions to the backup files and write/create permission on the destination host.
This does not change the fact that the actual backup scripts MUST be run as the root user, it just allows the use of a NON-ROOT user to do the file copies between systems.
/etc/hosts updates (ssh-propagate11.sh)
- Pre 11.5 - checks for IP entry in /etc/hosts for each host in the all-systems file, and if missing, adds an entry for <ipaddress> <hostname>.
- Post 11.5 - /etc/hosts is controlled by a chef process to handle entries for known nw hosts, the new entries will look like this:
<IPaddress> <UUID> <UUID.netwitness>
- Any entries manually added will be wiped out by the chef process. To add <hostname> (short hostname) to these entries, a new file /etc/hosts.user (format: <ipaddress> <hostname> ) must be created so the chef recipe can update the /etc/hosts file with the hostnames.
ssh-propagate11.sh will automatically create a /etc/hosts.user file from the entries in the all-systems file, and run the chef recipe to update the /etc/hosts, so the entries will read:
<IPaddress> <UUID> <UUID.netwitness> <hostname>
- In addition, if you need to add other hosts to the /etc/hosts that are not part of the Netwitness Deployment (and not resolvable by DNS), you can create a hosts.custom file in the directory where the ssh-propagate11.sh is run from, and it will append those entries to the /etc/hosts.user file before running the chef recipe.
- Previously, nw-backup11.sh relied on the Device-Type entry in all-systems to match an NRT Category, NRT would then use the category.sequence file to get a list of components to backup for that Device Type. Post 11.4, with the ability added to install additional services on hosts, this process would no longer work effectively.
- The script no longer depends on the Device-Type entry in the all-systems to run the backups, (it will still use it for filtering on specific types if the -t option is used, but is not used to determine what services are running on a particular host). It now queries each host for a list of enabled processes, and parses that list to determine what components are enabled on the host, and dynamically creates a component list to pass to the NRT script when launched.
- Any changes to a host (adding services) will not require a change to the backup process, as the new services will be discovered automatically when the backup is run.
- NFS use (for copying backup files to another host) has been updated to give better feed back if there are issues.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.