Upgrade Guide 11.4.1: Appendix B. Troubleshooting Version Installations and Upgrades

Document created by RSA Information Design and Development Employee on Apr 23, 2020•Last modified by RSA Information Design and Development Employee on Apr 28, 2020
Version 2Show Document
  • Comment • 0
  • View in full screen mode
 

This section describes the error messages displayed in the Hosts view when it encounters problems updating host versions and installing services on hosts in the Hosts view. If you cannot resolve an update or installation issue using the following troubleshooting solutions, contact Customer Support (https://community.rsa.com/docs/DOC-1294).

Troubleshooting instructions for the following errors that may occur during the upgrade are described in this section.

Troubleshooting instructions are also provided for errors for the following hosts and services that may occur during or after an upgrade.

deploy_admin User Password Has Expired Error

                 
Error Message

Cause The deploy_admin user password has expired.
Solution

Reset your deploy_admin password password.

  1. On all component hosts (not including the NW Server host), run the following command.
    /opt/rsa/saTools/bin/set-deploy-admin-password
  2. After all the component hosts have been updated, run this command on the NW Server host.
    /opt/rsa/saTools/bin/set-deploy-admin-password

  3. On the host that failed installation or orchestration, run the nwsetup-tui command and use the new deploy_admin password in response to the Deployment Password prompt.

Downloading Error

                     
Error Message

ProblemWhen you select an update version and click Update >Update Host, the download starts but fails to complete.
CauseVersion download files can be large and take a long time to download. If there are communication issues during the download it will fail.
Solution
  1. Try to update again.

  2. If it fails again with the same error, try to update using the offline methods as described in "Offline Method from Hosts View" or "Offline Method Using Command Line Interface" in the Upgrade Guide for NetWitness Platform 11.4. Go to the Master Table of Contents to find all RSA NetWitness Platform 11.x documents.

  3. If you are still not able to update, contact Customer Support (https://community.rsa.com/docs/DOC-1294).

Error Deploying Version <version-number> Missing Update Packages

                 
Error Message


Problem

Error deploying version <version-number> is displayed in the Initialize Update Package for RSA NetWitness Platform dialog after you click on Initialize Update if the update package is corrupted.

Solution

  1. Click Close to close the dialog.

  2. Remove the version folder from staging folder.

  3. Make sure that the salt-master service is running.

  4. Recopy the update package zip file to the staging folder.

  5. In the Hosts view toolbar, select Check for Updates again.

  6. Click Initialize Update.
  7. Click Update > Update Hosts from the toolbar.
  8. Click Begin Update from the Update Available dialog.
    After the host is updated, it prompts you to reboot the host.
  9. Click Reboot from the toolbar.

External Repo Update Error

                 
Error Message

Received an error similar to the following error when trying to update to a new version from the :
.Repository 'nw-rsa-base': Error parsing config: Error parsing "baseurl = 'https://nw-node-zero/nwrpmrepo /<version-number>/RSA'": URL must be http, ftp, file or https not ""

Cause There is an error the path you specified.
Solution

Make sure that:

  • the URL does exist on the NW Server host.
  • you used the correct path and remove any spaces from it.

Host Installation Failed Error

                 
Error Message

ProblemWhen you select a host and click Install the install service process fails.
Solution
  1. Try to install the service again.
    Often this is all you need to do.
  2. If you still cannot install the service:
    1. Monitor the following logs on NW Server as it progresses (for example, submit the tail -f command string from the command line'):
      /var/netwitness/uax/logs/sa.log
      /var/log/netwitness/orchestration-server/orchestration-server.log
      /var/log/netwitness/deployment-upgrade/chef-solo.log
      /var/log/netwitness/config-management/chef-solo.log
      /var/lib/netwitness/config-management/cache/chef-stacktrace.out
      The error appears in one or more of these logs.
    2. Try to resolve the issue and reinstall the service.
      • Cause 1 - Entered the wrong deploy_admin password in the nwsetup-tui.
        Solution - Reset your deploy_admin password password.
        1. On the NW Server host and all other hosts on 11.x, run the following command.
          /opt/rsa/saTools/bin/set-deploy-admin-password

        2. On the host that failed installation or orchestration, run the nwsetup-tui command and use the new deploy_admin password in response to the Deployment Password prompt.

      • Cause 2 -The deploy_admin password has expired.
        Solution - Reset your deploy_admin password password.
        1. On the NW Server host and all other hosts on 11.x, run the following command.
          /opt/rsa/saTools/bin/set-deploy-admin-password

        2. On the host that failed installation or orchestration, run the nwsetup-tui command and use the new deploy_admin password in response to the Deployment Password prompt.

  1. If you still cannot apply the update, gather the logs from step 2 and contact Customer Support (https://community.rsa.com/docs/DOC-1294).

Host Update Failed Error

                 
Error Message


ProblemWhen you select an update version and click Update > Update Host, the download process is successful, but the update process fails.
Solution
  1. Try to apply the version update to the host again.
    Often this is all you need to do.
  2. If you still cannot apply the new version update:
    1. Monitor the following logs on NW Server as it progresses (for example, run the tail -f command from the command line):
      /var/netwitness/uax/logs/sa.log
      /var/log/netwitness/orchestration-server/orchestration-server.log
      /var/log/netwitness/deployment-upgrade/chef-solo.log
      /var/log/netwitness/config-management/chef-solo.log
      /var/lib/netwitness/config-management/cache/chef-stacktrace.out
      The error appears in one or more of these logs.
    2. Try to resolve the issue and reapply the version update.
      • Cause 1 - deploy_admin password has expired.
        Solution - Reset your deploy_admin password .
        Complete the following steps to resolve Cause 1.
        1. In the NetWitness Suite menu, select ADMIN > Security > Users tab.
        2. Select the deploy_admin and click Reset Password.
        3. (Conitional) If NetWitness Suite does not allow you to expired deploy_admin password in the Reset Password dialog, complete the following steps.
          1. Reset deploy_admin to use a new password.
          2. On all non-NW Server hosts on 11.x , run the following command using the matching deploy_admin password from NW Server host.
            /opt/rsa/saTools/bin/set-deploy-admin-password
        • Cause 2 -The deploy_admin password was changed on NW Server host but not changed on non-NW Server hosts.
          Complete the following step to resolve Cause 2.
          •  On all non-NW Server hosts on 11.x , run the following command using the matching deploy_admin password from NW Server host.
            /opt/rsa/saTools/bin/set-deploy-admin-password
  1. If you still cannot apply the update, gather the logs from step 2 and contact Customer Support (https://community.rsa.com/docs/DOC-1294).

Missing Update Packages Error

                 
Error Message


ProblemMissing the following update package(s) is displayed in the Initialize Update Package for RSA NetWitness Platform dialog when you are updating a host from the Hosts view offline and there are packages missing in the staging folder.
Solution

  1. Click Download Packages from RSA Link in the Initialize Update Package for RSA NetWitness Platform dialog.
    The RSA Link page that contains the update files for the selected version is displayed.

  2. Select missing packages from staging folder (for example, 11.4.0.0, 11.4.0.x, and 11.4.x.x).

    The Initialize Update Package for RSA NetWitness Platform dialog is displayed telling you that it is ready to initialize the update packages.

OpenSSL 1.1.x

                 
Error Message

The following example illustrates an ssh error that can occur when the ssh client is run from a host with OpenSSL 1.1.x installed:
$ ssh root@10.1.2.3
ssh_dispatch_run_fatal: Connection to 10.1.2.3 port 22: message authentication code incorrect

Problem

Advanced users who want to ssh to a NetWitness Platform host from a client that is using OpenSSL 1.1.x encounter this error because of incompatibility between CENTOS 7.x and OpenSSL 1.1.x. For example:

$ rpm -q openssl
openssl-1.1.1-8.el8.x86_64

Solution

Specify the compatible cipher list on the command line. For example:

$ ssh -oCiphers=aes128-ctr,aes192-ctr,aes256-ctr root@10.1.2.3

I've read & consent to terms in IS user agreement.

root@10.1.2.3's password:

Last login: Mon Oct 21 19:03:23 2019

Patch Update to Non-NW Server Error

                 
Error Message

The /var/log/netwitness/orchestration-server/orchestration-server.log has an error similar to the following error:
API|Failure /rsa/orchestration/task/update-config-management [counter=10 reason=IllegalArgumentException::Version '11.x.x.n' is not supported

Problem After you update the NW Server host to a version, you must update all non-NW Server hosts to the same version. For example, if you update the NW Server from 11.4.0.0 to 11.4.x.x , the only update path for the non-NW Server hosts is the same version (that is, 11.4.x.x). If you try to update any non-NW Server host to a different version (for example, from 11.4.0.0 to an 11.4.0.x) you will get this error.
Solution

You have two options:

  • Update the non-NW Server host to 11.4.x.x, or
  • Do not update the non-NW Server host (keep it at its current version)

Reboot Host After Update from Command Line Error

                 
Error Message

You receive a message in the User Interface to reboot the host after you update and reboot the host offline.

Cause You cannot use CLI to reboot the host. You must use the User Interface.
Solution

Reboot the host in the Host View in the User Interface.

Reporting Engine Restarts After Upgrade

                 
Problem

In some cases, after you upgrade to 11.4 from versions of 11.x, such as 11.2 or 11.3, the Reporting Engine service attempts to restart continuously without success.

Cause

The database files for live charts, alert status, or report status may not be loaded successfully as the files may be corrupted.

Solution

To resolve the issue, do the following:

  1. Check which database files are corrupted:

    Navigate to the file located at /var/netwitness/reserver/rsa/soc/reporting-engine/logs/reporting-engine.log and check the following blocks:

    • If the live charts db file is corrupted, the following logs are displayed:

      Live charts error

    • If the alert status db file is corrupted, the following logs are displayed:

      Alert status error

    • If the report status db file is corrupted, the following logs are displayed:

      org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-196]

  2. To resolve the live charts database file corruption, perform the following steps:

    1. Stop the Reporting Engine service.

    2. Move the livechart.mv.db file from /var/netwitness/reserver/rsa/soc/reporting-engine/livecharts folder to a temporary location.

    3. Restart the Reporting Engine service.

      Note: Some live charts data may be lost on performing the above steps.

     

    To resolve the alert status or report status database file corruption, perform the following steps:

    1. Stop the Reporting Engine service.
    2. Replace the corrupted db file with the latest alertstatusmanager.mv.db or reportstatusmanager.mv.db file from /var/netwitness/reserver/rsa/soc/reporting-engine/archives folder.
    3. Restart the Reporting Engine service.

    For more information, see the Knowledge Base article Reporting Engine restarts After upgrade to RSA NetWitness Platform 11.4.

     

 

Log Collector Service (nwlogcollector)

Log Collector logs are posted to /var/log/install/nwlogcollector_install.log on the host running the nwlogcollector service.

                 
Error Message <timestamp>.NwLogCollector_PostInstall: Lockbox Status : Failed to open lockbox: The lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the lockbox using the passphrase.
Cause The Log Collector Lockbox failed to open after the update.
Solution Log in to NetWitness Platform and reset the system fingerprint by resetting the stable system value password for the Lockbox as described in the "Reset the Stable System Value" topic under  "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.

 

                 
Error Message <timestamp> NwLogCollector_PostInstall: Lockbox Status : Not Found
Cause The Log Collector Lockbox is not configured after the update.
Solution If you use a Log Collector Lockbox, log in to NetWitness Platform and configure the Lockbox  as described in the "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.

 

                 
Error Message <timestamp>: NwLogCollector_PostInstall: Lockbox Status : Lockbox maintenance required: The lockbox stable value threshold requires resetting. To reset the system fingerprint, select Reset Stable System Value on the settings page of the Log Collector.
Cause You need to reset the stable value threshold field for the Log Collector Lockbox.
Solution Log in to NetWitness Platform and reset the stable system value password for the Lockbox  as described in "Reset the Stable System Value" topic under  "Configure Lockbox Security Settings" topic in the Log Collection Configuration Guide.

NW Server

These logs are posted to /var/netwitness/uax/logs/sa.log on the NW Server Host.

                 
Problem

After upgrade, you notice that Audit logs are not getting forwarded to the configured Global Audit Setup;

or,

The following message seen in the sa.log.
Syslog Configuration migration failed. Restart jetty service to fix this issue

Cause NW Server Global Audit setup migration failed to migrate from 11.2.x.x or 11.3.x.x. to 11.4.0.0.
Solution
  1. SSH to the NW Server.
  2. Submit the following command.
    orchestration-cli-client --update-admin-node

Orchestration

The orchestration server logs are posted to /var/log/netwitness/orchestration-server/orchestration-server.log on the NW Server Host.

                 
Problem
  1. Tried to upgrade a non-NW Server host and it failed.
  2. Retried the upgrade for this host and it failed again.

 

You will see the following message in the orchestration-server.log.
"'file' _virtual_ returned False: cannot import name HASHES""

Cause Salt minion may have been upgraded and never restarted on failed non-NW Server host
Solution
  1. SSH to the non-NW Server host that failed to upgrade.
  2. Submit the following commands.
    systemctl unmask salt-minion
    systemctl restart salt-minion
  3. Retry the upgrade of the non-NW Server host.

Reporting Engine Service 

Reporting Engine Update logs are posted to to/var/log/re_install.log file on the host running the Reporting Engine service.

                 
Error Message <timestamp> : Available free space in /var/netwitness/re-server/rsa/soc/reporting-engine [ ><existing-GB ] is less than the required space [ <required-GB> ]
Cause Update of the Reporting Engine failed because you do not have enough disk space. 
Solution Free up the disk space to accommodate the required space shown in the log message. See the "Add Additional Space for Large Reports" topic in the Reporting Engine Configuration Guide for instructions on how to free up disk space.

 

You are here
Table of Contents > Appendix B. Troubleshooting Version Installations and Upgrades

Attachments

    Outcomes

      Visibility: RSA NetWitness Platform Online Documentation545 Views
      Last modified on Apr 28, 2020 12:53 PM
      Ratings: