This section describes solutions to problems that you may encounter during installations and upgrades. In most cases, NetWitness Platform creates log messages when it encounters these problems.
Note: If you cannot resolve an upgrade issue using the following troubleshooting solutions, contact Customer Support (https://community.rsa.com/docs/DOC-1294).
This section has troubleshooting documentation for the following services, features, and processes.
- Command Line Interface (CLI)
- Backup Script
- Event Stream Analysis
- Concentrator Service
- Log Collector Service (nwlogcollector)
- Orchestration
- NW Server
- Reporting Engine
- NetWitness UEBA
Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.
Command Line Interface (CLI)
| Error Message | Command Line Interface (CLI) displays: "Orchestration failed." Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource in/var/log/netwitness/config-management/chef-solo.log |
| Cause | Entered the wrong deploy_admin password in nwsetup-tui. |
| Solution | Retrieve your deploy_admin password password.
|
| Error Message | ERROR com.rsa.smc.sa.admin.web.controller.ajax.health. AlarmsController - Cannot connect to System Management Service |
| Cause | NetWitness Platform sees the Service Management Service (SMS) as down after successful upgrade even though the service is running. |
| Solution | Restart SMS service. systemctl restart rsa-sms |
| Error Message | You receive a message in the User Interface to reboot the host after you update and reboot the host offline. |
| Cause | You cannot use CLI to reboot the host. You must use the User Interface. |
| Solution | Reboot the host in the Host View in the User Interface. |
Backup (nw-backup script)
| Error Message | WARNING: Incorrect ESA Mongo admin password for host <hostname>. |
| Cause | ESA Mongo admin password contains special characters (for example, ‘!@#$%^qwerty’). |
| Solution | Change the ESA Mongo admin password back to the original default of ‘netwitness’ before running backup. |
| Error | Backup errors caused by the immutable attribute setting. Here is an example of an error that can be displayed: |
| Cause | If you have any files that have the immutable flag set (to keep the Puppet process from overwriting a customized file), the file will not be included in the backup process and an error will be generated. |
| Solution | On the host that contains the files with the immutable flag set, run the following command to remove the immutable setting from the files: chattr -i <filename> |
| Error | Error creating Network Configuration Information file due to duplicate or bad entries in primary network configuration file: /etc/sysconfig/network-scripts/ifcfg-em1 Verify contents of /var/netwitness/logdecoder/packetdb/nw-backup/2018-02-23/S5-BROK-36-10.25.53.36-network.info.txt |
| Cause | There are incorrect or duplicate entries for any one of the following fields: DEVICE, BOOTPROTO, IPADDR, NETMASK or GATEWAY, that were found from reading the primary Ethernet interface configuration file from the host being backed up. |
| Solution | Manually create a file at the backup location on the external backup server, as well as the backup location local to the host where other backups have been staged. The file name should be of the format <hostname>-<hostip>-network.info.txt, and should contain the following entries: DEVICE=<devicename> ; # from the host's primary ethernet interface config file BOOTPROTO=<bootprotocol> ; # from the host's primary ethernet interface config file IPADDR=<value> ; # from the host's primary ethernet interface config file NETMASK=<value> ; # from the host's primary ethernet interface config file GATEWAY=<value> ; # from the host's primary ethernet interface config file search <value> ; # from the host's /etc/resolv.conf file nameserver <value> ; # from the host's /etc/resolv.conf file |
Event Stream Analysis
- For ESA Correlation troubleshooting information, see the Alerting with ESA Correlation Rules User Guide.
- For ESA Analytics troubleshooting information, see the Automated Threat Detection Configuration Guide.
Concentrator Service
| Problem | After you upgrade to 11.3.0.0, pivot to navigate query fails if the Concentrator service version is 10.6.x. |
| Cause | Pivot to Navigate query fails as it contains meta entities and 10.6.x Concentrator service does not support meta entities. |
| Solution | You must edit the query and remove meta entities. For example, if query is for user then remove the user.all meta entity and re-run the query. |
Log Collector Service (nwlogcollector)
Log Collector logs are posted to /var/log/install/nwlogcollector_install.log on the host running the nwlogcollector service.
| Error Message | <timestamp>.NwLogCollector_PostInstall: Lockbox Status : Failed to open lockbox: The lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the lockbox using the passphrase. |
| Cause | The Log Collector Lockbox failed to open after the update. |
| Solution | Log in to NetWitness Platform and reset the system fingerprint by resetting the stable system value password for the Lockbox as described in the "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. |
| Error Message | <timestamp> NwLogCollector_PostInstall: Lockbox Status : Not Found |
| Cause | The Log Collector Lockbox is not configured after the update. |
| Solution | If you use a Log Collector Lockbox, log in to NetWitness Platform and configure the Lockbox as described in the "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. |
| Error Message | <timestamp>: NwLogCollector_PostInstall: Lockbox Status : Lockbox maintenance required: The lockbox stable value threshold requires resetting. To reset the system fingerprint, select Reset Stable System Value on the settings page of the Log Collector. |
| Cause | You need to reset the stable value threshold field for the Log Collector Lockbox. |
| Solution | Log in to NetWitness Platform and reset the stable system value password for the Lockbox as described in "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. |
| Problem | You have prepared a Log Collector for upgrade and no longer want to upgrade at this time. |
| Cause | Delay in upgrade. |
| Solution | Use the following command string to revert a Log Collector that has been prepared for upgrade back to resume normal operation. # /opt/rsa/nwlogcollector/nwtools/prepare-for-migrate.sh --revert |
NW Server
These logs are posted to /var/netwitness/uax/logs/sa.log on the NW Server Host.
| Problem | After upgrade, you notice that Audit logs are not getting forwarded to the configured Global Audit Setup; or, The following message seen in the sa.log. |
| Cause | NW Server Global Audit setup migration failed to migrate from 10.6.6.x to 11.3.0.0. |
| Solution |
|
Orchestration
The orchestration server logs are posted to /var/log/netwitness/orchestration-server/orchestration-server.log on the NW Server Host.
| Problem |
You will see the following message in the orchestration-server.log. |
| Cause | Salt minion may have been upgraded and never restarted on failed non-NW Server host |
| Solution |
|
Reporting Engine Service
Reporting Engine Update logs are posted to to/var/log/re_install.log file on the host running the Reporting Engine service.
| Error Message | <timestamp> : Available free space in /var/netwitness/re-server/rsa/soc/reporting-engine [ ><existing-GB ] is less than the required space [ <required-GB> ] |
| Cause | Update of the Reporting Engine failed because you do not have enough disk space. |
| Solution | Free up the disk space to accommodate the required space shown in the log message. See the "Add Additional Space for Large Reports" topic in the Reporting Engine Configuration Guide for instructions on how to free up disk space. |
NetWitness UEBA
| Problem | The User Interface is not accessible. |
| Cause | You have more than one NetWitness UEBA service existing in your NetWitness deployment and you can only have NetWitness UEBA service in your deployment. |
| Solution | Complete the following steps to remove the extra NetWitness UEBA service.
|
