There two sections in this appendix.
- Section 1 - General Troubleshooting Information
- Section 2 - Hardware-Related Troubleshooting Information
This section describes solutions to problems that you may encounter during installations and upgrades. In most cases, NetWitness Platform creates log messages when it encounters these problems.
This section has troubleshooting documentation for the following services, features, and processes.
- Command Line Interface (CLI)
- Backup Script
- Event Stream Analysis
- Concentrator Service
- Log Collector Service (nwlogcollector)
- NW Server
- Reporting Engine
- NetWitness UEBA
Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.
|Error Message|| |
Command Line Interface (CLI) displays: "Orchestration failed."
Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource in/var/log/netwitness/config-management/chef-solo.log
|Cause||Entered the wrong deploy_admin password in nwsetup-tui.|
Retrieve your deploy_admin password password.
|Error Message||ERROR com.rsa.smc.sa.admin.web.controller.ajax.health.|
AlarmsController - Cannot connect to System Management Service
|Cause||NetWitness Platform sees the Service Management Service (SMS) as down after successful upgrade even though the service is running.|
|Solution||Restart SMS service.|
systemctl restart rsa-sms
|Error Message|| |
You receive a message in the User Interface to reboot the host after you update and reboot the host offline.
|Cause||You cannot use CLI to reboot the host. You must use the User Interface.|
Reboot the host in the Host View in the User Interface.
|Error Message||WARNING: Incorrect ESA Mongo admin password for host <hostname>.|
|Cause||ESA Mongo admin password contains special characters (for example, ‘!@#$%^qwerty’).|
|Solution||Change the ESA Mongo admin password back to the original default of ‘netwitness’ before running backup.|
|Error||Error creating Network Configuration Information file due to duplicate or bad entries in primary network configuration file:|
Verify contents of /var/netwitness/logdecoder/packetdb/nw-backup/2018-02-23/S5-BROK-36-10.25.53.36-network.info.txt
|Cause||There are incorrect or duplicate entries for any one of the following fields: DEVICE, BOOTPROTO, IPADDR, NETMASK or GATEWAY, that were found from reading the primary Ethernet interface configuration file from the host being backed up.|
|Solution||Manually create a file at the backup location on the external backup server, as well as the backup location local to the host where other backups have been staged. The file name should be of the format <hostname>-<hostip>-network.info.txt, and should contain the following entries:|
DEVICE=<devicename> ; # from the host's primary ethernet interface config file
BOOTPROTO=<bootprotocol> ; # from the host's primary ethernet interface config file
IPADDR=<value> ; # from the host's primary ethernet interface config file
NETMASK=<value> ; # from the host's primary ethernet interface config file
GATEWAY=<value> ; # from the host's primary ethernet interface config file
search <value> ; # from the host's /etc/resolv.conf file
nameserver <value> ; # from the host's /etc/resolv.conf file
- For ESA Correlation troubleshooting information, see the Alerting with ESA Correlation Rules User Guide.
- For ESA Analytics troubleshooting information, see the Automated Threat Detection Configuration Guide.
After you upgrade to 18.104.22.168, pivot to navigate query fails if the Concentrator service version is 10.6.x.
|Cause||Pivot to Navigate query fails as it contains meta entities and 10.6.x Concentrator service does not support meta entities.|
|Solution||You must edit the query and remove meta entities. For example, if query is for user then remove the user.all meta entity and re-run the query.|
Log Collector logs are posted to /var/log/install/nwlogcollector_install.log on the host running the nwlogcollector service.
|Error Message||<timestamp>.NwLogCollector_PostInstall: Lockbox Status : Failed to open lockbox: The lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the lockbox using the passphrase.|
|Cause||The Log Collector Lockbox failed to open after the update.|
|Solution||Log in to NetWitness Platform and reset the system fingerprint by resetting the stable system value password for the Lockbox as described in the "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.|
|Error Message||<timestamp> NwLogCollector_PostInstall: Lockbox Status : Not Found|
|Cause||The Log Collector Lockbox is not configured after the update.|
|Solution||If you use a Log Collector Lockbox, log in to NetWitness Platform and configure the Lockbox as described in the "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.|
|Error Message||<timestamp>: NwLogCollector_PostInstall: Lockbox Status : Lockbox maintenance required: The lockbox stable value threshold requires resetting. To reset the system fingerprint, select Reset Stable System Value on the settings page of the Log Collector.|
|Cause||You need to reset the stable value threshold field for the Log Collector Lockbox.|
|Solution||Log in to NetWitness Platform and reset the stable system value password for the Lockbox as described in "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.|
|Problem||You have prepared a Log Collector for upgrade and no longer want to upgrade at this time.|
|Cause||Delay in upgrade.|
Use the following command string to revert a Log Collector that has been prepared for upgrade back to resume normal operation.
# /opt/rsa/nwlogcollector/nwtools/prepare-for-migrate.sh --revert
These logs are posted to /var/netwitness/uax/logs/sa.log on the NW Server Host.
After upgrade, you notice that Audit logs are not getting forwarded to the configured Global Audit Setup;
The following message seen in the sa.log.
|Cause||NW Server Global Audit setup migration failed to migrate from 10.6.6.x to 22.214.171.124.|
The orchestration server logs are posted to /var/log/netwitness/orchestration-server/orchestration-server.log on the NW Server Host.
You will see the following message in the orchestration-server.log.
|Cause||Salt minion may have been upgraded and never restarted on failed non-NW Server host|
Reporting Engine Update logs are posted to to/var/log/re_install.log file on the host running the Reporting Engine service.
|Error Message||<timestamp> : Available free space in /var/netwitness/re-server/rsa/soc/reporting-engine [ ><existing-GB ] is less than the required space [ <required-GB> ]|
|Cause||Update of the Reporting Engine failed because you do not have enough disk space.|
|Solution||Free up the disk space to accommodate the required space shown in the log message. See the "Add Additional Space for Large Reports" topic in the Reporting Engine Configuration Guide for instructions on how to free up disk space.|
The User Interface is not accessible.
|Cause||You have more than one NetWitness UEBA service existing in your NetWitness deployment and you can only have NetWitness UEBA service in your deployment.|
Complete the following steps to remove the extra NetWitness UEBA service.
The mtu.conf and pf_ring files for the 10G Decoder were not restored from the ./etc/init/pfring_bkup directory after upgrade.
If you use the 10G Decoder hardware driver and you customized the /etc/init.d/pf_ring script to use MTU from the /etc/pf_ring/mtu.conf file, the mtu.conf and pf_ring files from the ./etc/init/pfring_bkup directory are not restored after upgrade.
Complete the following steps to restore the files.