Azure 10.6.6.x to 11.3 Upgrade: Troubleshooting

Document created by RSA Information Design and Development on Apr 12, 2019Last modified by RSA Information Design and Development on Sep 26, 2019
Version 6Show Document
  • View in full screen mode
 

This section describes problems that you may encounter during the upgrade with solutions. In most cases, NetWitness Platform creates log messages when it encounters these problems.

Note: If you cannot resolve any upgrade issue using the following troubleshooting solutions, contact Customer Support (How to contact RSA Customer Supporthttps://community.rsa.com/docs/DOC-1294)

This section has troubleshooting documentation for the following services, features, and processes.

11.3.0.2 Setup Program (nwsetup-tui)

                 
ProblemHost Setup Program (nwsetup-tui) exits and creates the following error message in /var/log/netwitness/bootstrap/launch/
security-server/security-server.log
:
<yyyy-mm-dd hh:mm:ss,nnn> [ main] ERROR
SystemOperation|Service startup failed. Running in safe mode

org.h2.jdbc.JdbcSQLException: The database is read only [90097-193]
at org.h2.message.DbException.
getJdbcSQLException(DbException.java:345) ...

at org.springframework.jdbc.datasource.
AbstractDriverBasedDataSource.getConnection
(AbstractDriverBasedDataSource.java:159
)
at
com.rsa.asoc.security.upgrade.legacy.
MigrationDatabase.<init>(MigrationDatabase.java:113)
CauseThe H2 database needs write permission to complete the host setup.
Solution

From the NW Server command line, provide write permission to H2.db, restart the NW Server, and restart nwsetup-tui Setup Program.

chmod o+w /var/lib/netwitness/uax/db/platform.h2.db
systemctl restart rsa-nw-security-server.service
nwsetup-tui

Backup (nw-backup script)

                 
MessageWARNING: Incorrect ESA Mongo admin password for host <hostname>.
CauseESA Mongo admin password contains special characters (for example, ‘!@#$%^qwerty’).
SolutionChange the ESA mongo admin password back to the original default of ‘netwitness’ before running backup. See "ESA Config: Change MongoDB Password for admin Account" the the RSA NetWitness Platform Event Stream Analysis Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.

Event Stream Analysis

                 
ProblemESA service crashes after you upgrade to 11.3.0.2 from a FIPS enabled setup.
CauseESA service is pointing to an invalid keystore.
Solution
  1. SSH to the ESAPrimary host and log in.
  2. In the /opt/rsa/esa/conf/wrapper.conf file, replace the following line:
    wrapper.java.additional.5=-Djavax.net.ssl.keyStore=/opt/rsa/esa/../carlos/keystore
    with:
    wrapper.java.additional.5=-Djavax.net.ssl.keyStore=/opt/rsa/carlos/keystore
  3. Submit the following command to restart ESA .
    systemctl restart rsa-nw-esa-server

    Note: If you have multiple ESA hosts and you encounter that same problem, repeat steps 1 through 3 inclusive on each secondary ESA host.

General

Logs referred to in this section are posted to /var/log/install/install.log on the NW Server Host.

                 
MessageERROR com.rsa.smc.sa.admin.web.controller.ajax.health.
AlarmsController - Cannot connect to System Management Service
CauseNetWitness Platform sees the Service Management Service (SMS) as down after successful upgrade even though the service is running.
SolutionRestart SMS service using below command.
systemctl restart rsa-sms

 

                 
Message<timestamp> <host>: SMS_PostInstall: INFO: Free disk space on /opt is nGB

<timestamp> <host>: SMS_PostInstall: WARN: Disk space check failed on /opt. The available disk space nGB is less than the recommended minimum disk space of 10GB.

CauseLow or insufficient disk space allocated for the SMS service.
SolutionRSA recommends that you provide a minimum of 10 GB of disk space for the SMS service to run optimally.

 

             
ProblemAfter you run the Setup Program for a non-NW Server host, you must go in to the UI, enable the host, and install the service on the host from the Hosts View. If you see "Install error View Details" in the Status column of the Hosts view, the host lost connectivity due to network issues.
Solution

Re-install the service on the host from the Hosts view.

Log Collector Service (nwlogcollector)

Log Collector  logs are posted to /var/log/install/nwlogcollector_install.log on the host running the nwlogcollector service.

                 
Message<timestamp>.NwLogCollector_PostInstall: Lockbox Status : Failed to open lockbox: The lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the lockbox using the passphrase.
CauseThe Log Collector Lockbox failed to open after the update.
SolutionLog in to NetWitness Platform and reset the system fingerprint by resetting the stable system value password for the Lockbox as described in the "Reset the Stable System Value" topic under  "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.

 

                 
Messagetimestamp NwLogCollector_PostInstall: Lockbox Status : Not Found
CauseThe Log Collector Lockbox is not configured after the update.
Solution(Conditional) If you use a Log Collector Lockbox, log in to NetWitness Platform and configure the Lockbox  as described in the"Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents..

 

                 
Message<timestamp>: NwLogCollector_PostInstall: Lockbox Status : Lockbox maintenance required: The lockbox stable value threshold requires resetting. To reset the system fingerprint, select Reset Stable System Value on the settings page of the Log Collector.
CauseYou need to reset the stable value threshold field for the Log Collector Lockbox.
SolutionLog in to NetWitness Platform and reset the stable system value password for the Lockbox  as described in "Reset the Stable System Value" topic under  "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.

 

                 
ProblemYou have prepared a Log Collector for upgrade and no longer want to upgrade at this time.
CauseDelay in upgrade.
Solution

Use the following command string to revert a Log Collector that has been prepared for upgrade back to resume normal operation.

# /opt/rsa/nwlogcollector/nwtools/prepare-for-migrate.sh --revert

NW Server

These logs are posted to /var/netwitness/uax/logs/sa.log on the NW Server Host.

                 
Problem

After upgrade, you notice that Audit logs are not getting forwarded to the configured Global Audit Setup;

or,

The following message seen in the sa.log.
Syslog Configuration migration failed. Restart jetty service to fix this issue

CauseNW Server Global Audit setup migration failed to migrate from 10.6.6 to 11.3.0.2.
Solution
  1. SSH to the NW Server.
  2. Submit the following command.
    orchestration-cli-client --update-admin-node

Reporting Engine Service 

Reporting Engine Update logs are posted to to/var/log/re_install.log file on the host running the Reporting Engine service.

                 
Message<timestamp> : Available free space in /home/rsasoc/rsa/soc/reporting-engine [ existing-GB ] is less than the required space [ required-GB ]
CauseUpdate of the Reporting Engine failed because you do not have enough disk space. 
SolutionFree up the disk space to accommodate the required space shown in the log message. See the "Add Additional Space for Large Reports" topic in the Reporting Engine Configuration Guide for instructions on how to free up disk space. Go to the Master Table of Contents to find all NetWitness Platform Logs & Network 11.x documents.

Previous Topic:Post Upgrade Tasks
You are here
Table of Contents > Appendix A. Troubleshooting

Attachments

    Outcomes