on Apr 12, 2019This section describes problems that you may encounter during the upgrade with solutions. In most cases, NetWitness Platform creates log messages when it encounters these problems.
Note: If you cannot resolve any upgrade issue using the following troubleshooting solutions, contact Customer Support (How to contact RSA Customer Supporthttps://community.rsa.com/docs/DOC-1294)
This section has troubleshooting documentation for the following services, features, and processes.
- 11.3.0.2 Setup Program (nwsetup-tui)
- Backup
- Event Stream Analysis
- General
- Log Collector Service (nwlogcollector)
- NW Server
- Reporting Engine
11.3.0.2 Setup Program (nwsetup-tui)
| Problem | Host Setup Program (nwsetup-tui) exits and creates the following error message in /var/log/netwitness/bootstrap/launch/ security-server/security-server.log: <yyyy-mm-dd hh:mm:ss,nnn> [ main] ERROR SystemOperation|Service startup failed. Running in safe mode org.h2.jdbc.JdbcSQLException: The database is read only [90097-193] at org.h2.message.DbException. getJdbcSQLException(DbException.java:345) ... at org.springframework.jdbc.datasource. AbstractDriverBasedDataSource.getConnection (AbstractDriverBasedDataSource.java:159) at com.rsa.asoc.security.upgrade.legacy. MigrationDatabase.<init>(MigrationDatabase.java:113) |
| Cause | The H2 database needs write permission to complete the host setup. |
| Solution | From the NW Server command line, provide write permission to H2.db, restart the NW Server, and restart nwsetup-tui Setup Program. chmod o+w /var/lib/netwitness/uax/db/platform.h2.db |
Backup (nw-backup script)
| Message | WARNING: Incorrect ESA Mongo admin password for host <hostname>. |
| Cause | ESA Mongo admin password contains special characters (for example, ‘!@#$%^qwerty’). |
| Solution | Change the ESA mongo admin password back to the original default of ‘netwitness’ before running backup. See "ESA Config: Change MongoDB Password for admin Account" the the RSA NetWitness Platform Event Stream Analysis Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents. |
Event Stream Analysis
| Problem | ESA service crashes after you upgrade to 11.3.0.2 from a FIPS enabled setup. |
| Cause | ESA service is pointing to an invalid keystore. |
| Solution |
|
General
Logs referred to in this section are posted to /var/log/install/install.log on the NW Server Host.
| Message | ERROR com.rsa.smc.sa.admin.web.controller.ajax.health. AlarmsController - Cannot connect to System Management Service |
| Cause | NetWitness Platform sees the Service Management Service (SMS) as down after successful upgrade even though the service is running. |
| Solution | Restart SMS service using below command. systemctl restart rsa-sms |
| Message | <timestamp> <host>: SMS_PostInstall: INFO: Free disk space on /opt is nGB <timestamp> <host>: SMS_PostInstall: WARN: Disk space check failed on /opt. The available disk space nGB is less than the recommended minimum disk space of 10GB. |
| Cause | Low or insufficient disk space allocated for the SMS service. |
| Solution | RSA recommends that you provide a minimum of 10 GB of disk space for the SMS service to run optimally. |
| Problem | After you run the Setup Program for a non-NW Server host, you must go in to the UI, enable the host, and install the service on the host from the Hosts View. If you see "Install error View Details" in the Status column of the Hosts view, the host lost connectivity due to network issues. |
| Solution | Re-install the service on the host from the Hosts view. |
Log Collector Service (nwlogcollector)
Log Collector logs are posted to /var/log/install/nwlogcollector_install.log on the host running the nwlogcollector service.
| Message | <timestamp>.NwLogCollector_PostInstall: Lockbox Status : Failed to open lockbox: The lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the lockbox using the passphrase. |
| Cause | The Log Collector Lockbox failed to open after the update. |
| Solution | Log in to NetWitness Platform and reset the system fingerprint by resetting the stable system value password for the Lockbox as described in the "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents. |
| Message | timestamp NwLogCollector_PostInstall: Lockbox Status : Not Found |
| Cause | The Log Collector Lockbox is not configured after the update. |
| Solution | (Conditional) If you use a Log Collector Lockbox, log in to NetWitness Platform and configure the Lockbox as described in the"Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.. |
| Message | <timestamp>: NwLogCollector_PostInstall: Lockbox Status : Lockbox maintenance required: The lockbox stable value threshold requires resetting. To reset the system fingerprint, select Reset Stable System Value on the settings page of the Log Collector. |
| Cause | You need to reset the stable value threshold field for the Log Collector Lockbox. |
| Solution | Log in to NetWitness Platform and reset the stable system value password for the Lockbox as described in "Reset the Stable System Value" topic under "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents. |
| Problem | You have prepared a Log Collector for upgrade and no longer want to upgrade at this time. |
| Cause | Delay in upgrade. |
| Solution | Use the following command string to revert a Log Collector that has been prepared for upgrade back to resume normal operation. # /opt/rsa/nwlogcollector/nwtools/prepare-for-migrate.sh --revert |
NW Server
These logs are posted to /var/netwitness/uax/logs/sa.log on the NW Server Host.
| Problem | After upgrade, you notice that Audit logs are not getting forwarded to the configured Global Audit Setup; or, The following message seen in the sa.log. |
| Cause | NW Server Global Audit setup migration failed to migrate from 10.6.6 to 11.3.0.2. |
| Solution |
|
Reporting Engine Service
Reporting Engine Update logs are posted to to/var/log/re_install.log file on the host running the Reporting Engine service.
| Message | <timestamp> : Available free space in /home/rsasoc/rsa/soc/reporting-engine [ existing-GB ] is less than the required space [ required-GB ] |
| Cause | Update of the Reporting Engine failed because you do not have enough disk space. |
| Solution | Free up the disk space to accommodate the required space shown in the log message. See the "Add Additional Space for Large Reports" topic in the Reporting Engine Configuration Guide for instructions on how to free up disk space. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents. |
