Skip navigation
All Places > Products > RSA NetWitness Platform > RSA NetWitness Platform Online Documentation > Documents
Log in to create and rate content, and to follow, bookmark, and share content with other members.

Upgrade Guide 11.5: Post Upgrade Tasks

Document created by RSA Information Design and Development Employee on Sep 9, 2020Last modified by RSA Information Design and Development Employee on Sep 21, 2020
Version 8Show Document
  • View in full screen mode
 

After you upgrade to 11.5, NetWitness Platform has several new features in the user interface. Administrative tasks are consolidated as icons in the upper right corner to keep administration, configuration, notifications, jobs, and user preferences together. The following figure shows the new top-level navigation.
the top-level menu

The default view is the Springboard landing page. You can change the settings to customize the landing page view. For additional information, refer to "NetWitness Platform Basic Navigation" in the NetWitness Platform Getting Started Guide.

Complete the tasks that apply to the hosts in your environment.

General

(Conditional) Configure NAT-Based IP Addresses

If you have a host, such as a VLC, that requires a NAT-based IP address in order to connect to the NW Server host, you must update the host configuration with the following steps.

  1. Log in to the host that requires the use of NAT IP addresses, using the console or SSH.
  2. Run the following command:
    nw-manage --enable-nat-usage
  3. To set the NAT address for the NW Server:
    1. Log into the NW Server using the console or SSH.
    2. Run the following command:
      nw-manage -update-host --host-id <UUID of NW Server> --ipv4-public <NAT IP of NW Server>
    3. Note: You can find the UUID and view the current NAT IP address of the host by running nw-manage --list-hosts.

(Conditional - For Warm-Standby Hosts Only) Register the Secondary IP Address of Warm-Standby Hosts

The Warm-Standby server must be upgraded to 11.5 before completing the following steps.

  1. Log in to the NW Server using the console or SSH.
  2. Run the following command:
    nw-manage --add-nws-secondary-ip --ipv4 <ip address of Warm/Standby Server>
  3. Note: If the Warm-Standby server requires a NAT-based IP address (IPv4-public) for any host to access it during failover, the NAT IP address must also be registered by running the following command: nw-manage --add-nws-secondary-ip --ipv4 <NAT-based IP address of Warm Standby Server>

  4. Verify the correct Warm Standby host IP address value by running the following command:
    nw-manage --get-nws-secondary-ip

Review Contents of /etc/hosts.user for Obsolete Host Entries

After upgrading the NW Server host or a component host, review the contents of the /etc/hosts.user file for any obsolete host entries. The /etc/hosts.user file contains system and user-generated entries that are not managed by NetWitness Platform. However, entries from /etc/hosts.user are merged with NetWitness Platform-generated host mappings to create and update /etc/hosts. To avoid conflicts with NetWitness Platform-generated mappings, and to avoid generating connectivity errors resulting from an IP address change, RSA recommends that you remove any entries in /etc/hosts.user that include a non-loopback IP address of a NetWitness Platform host.

After updating /etc/hosts.user, you must refresh the system by running the following command:
nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>

Reconfigure DNS Servers

By default, a component host upgraded from 11.4 or earlier is configured with the same system DNS server as the NW Server. If this component host requires a different system DNS address, see "Change Host Network Configuration" in the System Maintenance Guide for instructions.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

  • Decoder
  • Log Decoder
  • Broker
  • Concentrator
  • Archiver

Start Network Capture

  1. In the NetWitness Platform menu, go to (Admin) > Services.
    The Services view is displayed.
  2. Select each Decoder service.
  3. Under (actions), select View > System.

  4. In the toolbar, click

Start Log Capture

    1. In the NetWitness Platform menu, go to (Admin) > Services.
      The Services view is displayed.
    2. Select each Log Decoder service.
    3. Under (actions), select View > System.
    4. In the toolbar, click
  • Start Aggregation

    1. In the NetWitness Platform menu, go to (Admin) > Services.

      The Services view is displayed.

    2. For each Concentrator, Broker, and Archiver service:

      1. Select the service.
      2. Under (actions), select View > Config.
      3. In the toolbar, click

    Event Stream Analysis (ESA)

    Note: Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.5 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

    There are no required post-upgrade tasks for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

    If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide.

    New Health and Wellness

    Note: New Health and Wellness in 11.5 replaces Next GEN Health and Wellness (BETA) in 11.4.x.x.

    Deploy the New Health and Wellness Content from Live

    After you upgrade from version 11.4.x.x to 11.5, New Health and Wellness content is not updated. To use the latest (default) content, you must deploy the content through NetWitness Live Services.

    Note: RSA recommends you to take a copy of 11.4.x.x Health and Wellness content before you deploy the content from NetWitness Live Services, as it overwrites the existing content.

    1. Log in to NetWitness Platform UI.

    2. Click (CONFIGURE) > LIVE CONTENT.

    3. In the Search Criteria panel, select the Resource Types as:

      • Health and Wellness Dashboards
      • Health and Wellness Monitors
    4. Click Search.

    5. In the Matching Resources view, select the checkbox to the left of the resources that you want to deploy.

    6. In the Matching Resources toolbar, click .

    7. In the Deployment Wizard > Resources tab, click Next.

    8. In the Services tab, select the Metrics Server service.

    9. Click Next.

    10. Click Deploy.
      The Deploy page is displayed. The Progress bar turns green when you have successfully deployed the resources to the selected services.

    11. Click Close.

    (Optional) Update UUID of New Health and Wellness Host to Update Service Configuration Documents

    If you have configured services for New Health and Wellness from nw-shell using set-config API and upgrade NetWitness Platform version from 11.4.x.x to 11.5, you must update IP with UUID for a host on which New Health and Wellness is installed.

    1. SSH to Admin Server.

    2. Check the UUID of a host on which New Health and Wellness is installed using the command:
      orchestration-cli-client --list-hosts

      This lists NetWitness Platform hosts along with the respective UUIDs. Make a note of the UUID of host on which New Health and Wellness is installed.

    3. Identify the services on which set-config is invoked using the command:
      mongo localhost/metrics-server -u deploy_admin -p <deployment_password> --authenticationDatabase admin --eval 'db.metric_config.find({ "createdBy": { $ne: "system" }})'
      This will list the configuration documents of the services on which set-config is invoked.

    Note: If no service documents are listed which means no services are configured before the upgrade, so you can ignore the remaining steps.

    1. In the configuration file, update the service document “host” field by replacing IP with the UUID of the host on which New Health and Wellness is installed.
      For example, update the host IP with UUID as below:

    "host" : “e086511c-121c-4e66-95a3-e87e27b12acb”

    1. Log in to nw-shell using the command:
      nw-shell
    2. Connect to metrics-server service using the command:
      connect --service metrics-server
    3. Enter the log in command:
      login

    4. Enter the admin username and password.
    5. Go to /rsa/metrics/elastic/set-config and invoke configuration files using the command:
      invoke --file /<absolute_path_of_service_config_file>

    (Optional) Uninstall New Health and Wellness

    To uninstall New Health and Wellness, perform the following:

    1. Take a backup of NetWitness Server host. For more information, see “Disaster Recovery (Back Up and Restore)” topic in the NetWitness Recovery Tool User Guide.

      nw-recovery-tool --export --dump-dir /some/folder --category AdminServer --category Search

      Note: If New Health and Wellness is not installed on NetWitness Server, you must take a backup of the host on which New Health and Wellness is installed.

    2. Make sure that the installation or upgrades are not in progress and stop the orchestration server on NetWitness Server host:

      systemctl stop rsa-nw-orchestration-server

    3. Remove the New Health and Wellness service category (“Search") from the host:

      1. SSH to Admin server

      2. Fetch host details where New Health and Wellness is installed using the following command:
        mongo localhost/orchestration-server -u deploy_admin -p <deploy_admin-password> --authenticationDatabase admin --eval 'db.host.find({ "installedServices": /.*Search.*/i })'

        Sample output

        { "_id" : "56f2a90b-1f03-d09a-fb71-42c2a93958a8", "hostname" : "10.10.10.11", "ipv4" : "10.10.10.11", "ipv4Public" : "", "displayName" : "adminserver", "version" : { "major" : 11, "minor" : 5, "servicePack" : 0, "patch" : 0, "snapshot" : false, "rawVersion" : "11.5.0.0" }, "lastFailedRefreshAttempt" : NumberLong(0), "refreshAttemptDelayFactor" : 0, "thirdParty" : false, "installedServices" : [ "Search", "AdminServer" ], "meta" : { "node-zero" : true }, "_class" : "com.rsa.asoc.orchestration.host.HostEntity" }

      3. Remove the "Search" from the installedServices.

      IMPORTANT: Do not remove any other category names.

      1. Replace <LIST-OF-CATEGORIES-EXCEPT-SEARCH> with a comma-delimited AND double-quoted list of all the existing installed services found earlier EXCEPT "Search":
        mongo localhost/orchestration-server -u deploy_admin -p <deploy_admin-password> --authenticationDatabase admin --eval 'db.host.update({ "_id" : "<hw-node-uuid>" },{$set: {"installedServices" : [ <LIST-OF-CATEGORIES-EXCEPT-SEARCH> ]}})'

        Example
        mongo localhost/orchestration-server -u deploy_admin -p netwitness --authenticationDatabase admin --eval 'db.host.update({ "_id" : "56f2a90b-1f03-d09a-fb71-42c2a93958a8" },{$set: {"installedServices" : [ "AdminServer" ]}})'

        Sample output

        MongoDB shell version v4.0.19

        connecting to: mongodb://localhost:27017/orchestration-server?authSource=admin&gssapiServiceName=mongodb

        Implicit session: session { "id" : UUID("04e32380-347e-4b7d-a63e-a094536d7242") }

        MongoDB server version: 4.0.19

        WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

      2. Make sure that the "Search" category is removed in the updated host record in the installedServices :

        mongo localhost/orchestration-server -u deploy_admin -p <deploy_admin-password> --authenticationDatabase admin --eval 'db.host.find({ "_id" : "<hw-node-uuid>" })'

        Example

        mongo localhost/orchestration-server -u deploy_admin -p netwitness --authenticationDatabase admin --eval 'db.host.find({ "_id" : "56f2a90b-1f03-d09a-fb71-42c2a93958a8" })'

      Note: Any inconsistencies can result in unrecoverable errors.

      Sample output

      { "_id" : "56f2a90b-1f03-d09a-fb71-42c2a93958a8", "hostname" : "10.10.10.11", "ipv4" : "10.10.10.11", "ipv4Public" : "", "displayName" : "adminserver", "version" : { "major" : 11, "minor" : 5, "servicePack" : 0, "patch" : 0, "snapshot" : false, "rawVersion" : "11.5.0.0" }, "lastFailedRefreshAttempt" : NumberLong(0), "refreshAttemptDelayFactor" : 0, "thirdParty" : false, "installedServices" : [ "AdminServer" ], "meta" : { "node-zero" : true }, "_class" : "com.rsa.asoc.orchestration.host.HostEntity" }

    4. Stop the New Health and Wellness services:

      systemctl stop rsa-nw-metrics-server elasticsearch opendistro-performance-analyzer kibana

    5. Disable the New Health and Wellness services:

    systemctl disable rsa-nw-metrics-server elasticsearch opendistro-performance-analyzer kibana

    1. Uninstall the New Health and Wellness packages using the command:

      yum erase -y rsa-nw-metrics-server opendistroforelasticsearch opendistroforelasticsearch-kibana

      Note: rsa-nw-shell (installed with metrics server) is a shared package and should not be removed.

    2. Remove the configuration folders or files:
      • /etc/netwitness/metrics-server
      • /etc/netwitness/platform/elasticsearch
      • /etc/netwitness/platform/nodeinfo/metrics-server
      • /etc/netwitness/platform/nodeinfo/elasticsearch-open-distro
      • /etc/netwitness/platform/nodeinfo/kibana-open-distro
      • /etc/systemd/system/rsa-nw-metrics-server.service.d
      • /etc/systemd/system/elasticsearch.service.d
      • /etc/pki/nw/service/bootstrap/metrics-server.completed
      • /etc/pki/nw/service/rsa-nw-metrics-server-cert.pem
      • /etc/pki/nw/service/rsa-nw-metrics-server.chain
      • /etc/pki/nw/elastic
      • /etc/pki/nw/kibana
      • /var/log/netwitness/metrics-server
      • /var/log/kibana
      • /etc/collectd.d/rsa-metrics-server.conf
      • /etc/logrotate.d/kibana
      • /etc/elasticsearch
      • /etc/kibana
      • /var/lib/elasticsearch
      • /var/lib/kibana
      • /var/netwitness/elasticsearch
    3. Start the orchestration Server on NetWitness Server:
      systemctl start rsa-nw-orchestration-server

    4. Unregister the New Health and Wellness from the installedService:

      1. Find the service IDs for metrics-server, elasticsearch-open-distro, and kibana-open-distro

        Note: Make sure you look for service IDs for the correct host; do not unregister elastic or kibana on an UEBA host.

        orchestration-cli-client --list-services | grep <hw-node-IP-address>

        Sample output

        ID=50082d04-320c-4ce2-8379-00f38ae2d1df, NAME=metrics-server, HOST=192.168.1.2:7018, TLS=true

        ID=530ff46a-8793-4e8e-be9c-742193d1705a, NAME=elasticsearch-open-distro, HOST=192.168.1.2:9200, TLS=true

        ID=4bad6ea8-e3a4-46ab-a342-34356bea65bb, NAME=kibana-open-distro, HOST=192.168.1.2:5601, TLS=true

        ... (other services) ...

      2. Remove the service IDs returned above for metrics-server, elasticsearch-open-distro, and kibana-open-distro (associated with New Health new Wellness host):

        orchestration-cli-client --remove-service --id <metrics-server-service-id>

        orchestration-cli-client --remove-service --id <elasticsearch-open-distro-service-id>

        orchestration-cli-client --remove-service --id <kibana-open-distro-service-id>

      3. Verify if the services are removed:

        orchestration-cli-client --list-services | grep <hw-node-IP-address>

    5. On all hosts, except for UEBA, stop and disable metricbeat:

      systemctl stop metricbeat

      systemctl disable metricbeat

      Note: For NetWitness Platform without UEBA, you can stop and disable metricbeat on all hosts through salt:
      salt '*' cmd.run 'systemctl stop metricbeat && systemctl disable metricbeat'

    6. (Optional) - If you are not reinstalling New Health and Wellness (on same or other hosts), you can also remove metricbeat package and configuration:
      1. Package to uninstall:
        metricbeat
      2. Service configurations to uninstall:
        • /etc/metricbeat

        • /var/log/metricbeat

        • mongo account

        • systemd configuration

    1. Refresh the New Health and Wellness host:
      nw-manage --refresh-host --host-key <node-ip>
      Make sure that the New Health and Wellness service is not installed or running and metricbeat service is not active on the New Health and Wellness host.

    2. If you are not reinstalling New Health and Wellness on another host, you must refresh UI hosts (NetWitness Server host and Analyst UI) to update NGNIX:

      nw-manage --refresh-host --host-key <node-ip>

    Note: After uninstalling New Health and Wellness, if you want to install New Health and Wellness again, see "New Health and Wellness" in the Deployment Guide.

    Investigate

    (Conditional - For Custom Roles Only) Adjust investigate-server Permissions for Custom User Roles

    After upgrading to Version 11.5, the built-in user roles for analysts (and others) using Investigate have the investigate-server.event.filter permission enabled, but the upgrade process does not enable the permission for custom user roles. Users who are assigned a custom user role that does not have this permission enabled cannot see the Filter Events panel, a new panel in 11.5 where they can drill into metadata.

    Note: When upgrading from Version 11.3.x.x or earlier, the built-in user roles for analysts using Investigate have three additional permissions added in Version 11.4 enabled, but the upgrade process does not enable the permissions for custom user roles. Users who are assigned a custom user role that does not have these permissions cannot see the Navigate view and Legacy Events view in the Investigate menu. The three permissions that need to be enabled for custom user roles are:
    investigate-server.columngroup.read, investigate-server.metagroup.read, and investigate-server.profile.read

    To enable the permissions for a user role:

    1. Go to (Admin) > Security and click the Roles tab.
    2. Select the custom user role that needs to be edited and click (edit icon).
    3. In the Edit Role dialog, ensure that these four permissions are enabled:
      investigate-server.event.filter
      investigate-server.columngroup.read
      investigate-server.metagroup.read
      investigate-server.profile.read
    4. Click Save to save your changes. When analysts with the custom user role log in to the NetWitness Platform, the changes are in effect.

    Respond

    The Primary ESA server must be upgraded to 11.5 before you can complete these tasks.

    Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 11.5. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

    (Conditional) Restore Any Respond Service Custom Keys in the Aggregation Rule Schema

    Note: If you did not manually customize the incident aggregation rule schema, you can skip this task.

    If you added custom keys in the var/lib/netwitness/respond-server/data/aggregation_ rule_schema.json file for use in the groupBy clause for 11.x, modify the /var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file and add the custom keys from the automatic backup file.

    The backup file is located in /var/lib/netwitness/respond-server/data and it is in the following format:
    aggregation_rule_schema.json.bak-<time of the backup>

    (Conditional) Restore any Customized Respond Service Normalization Scripts

    Note: If you did not manually customize any alert normalization scripts, you can skip this task.
    This procedure applies to upgrades from 11.3.x to 11.5. (For upgrades from 11.4.x to 11.5, there are no backups of the normalization script files since customizations are in separate custom_normalize script files, which are not overwritten during the upgrade.

    To prevent overwriting future customizations, custom normalization script files are available in NetWitness Platform 11.4 and later. Add any custom logic to the custom_normalize_<alert type>.js files.

    1. Locate any custom logic from the backup Respond normalization scripts located in the /var/lib/netwitness/respond-server/scripts.bak-<timestamp> directory, where <timestamp> is the time that the backup completed:
      data_privacy_map.js
      normalize_alerts.js
      normalize_core_alerts.js
      normalize_ecat_alerts.js
      normalize_ma_alerts.js
      normalize_ueba_alerts.js
      normalize_wtd_alerts.js
      utils.js

    2. Edit the new 11.4 or later script files in the /var/lib/netwitness/respond-server/scripts directory to include any logic from the back up files. If you have any customizations in the normalization files, add them to the normalization files with the custom prefix.
      data_privacy_map.js
      custom_normalize_alerts.js
      custom_normalize_core_alerts.js
      custom_normalize_ecat_alerts.js
      custom_normalize_ma_alerts.js
      custom_normalize_ueba_alerts.js
      custom_normalize_wtd_alerts.js
      utils.js

      For Example, the custom_normalize_core_alerts.js is the normalization script for ESA to add any custom logic. This java script file has a function ‘normalizeAlert’ with the parameters headers, rawAlert, and normalizedAlert. The variable ‘normalized’ is an immutable copy object, which has an embedded object of a list of normalized events. So if you have any custom meta keys configured for the events then you have to iterate through the ‘normalized.events’ to populate the appropriate meta keys with values from the ‘rawAlert.events’ object. Below is the sample code.

       

      normalizeAlert = function (headers, rawAlert, normalizedAlert) {

      // normalizedAlert is the immutable copy of ootb normalizer alert, make sure you use

      // normalized object to update/set the values in your scripts

      var normalized = Object.assign(normalizedAlert);

      var custom_events;

      if(normalized.events !== undefined) {

      custom_events = normalized.events;

       

      } else {

      custom_events = new Array([]);

       

      }

      for (var i = 0; i < rawAlert.events.length; i++) {

      custom_events[i].legalentity=Utils.stringValue(rawAlert.events[i].isgs_legalentity);

      custom_events[i].companycode=Utils.stringValue(rawAlert.events[i].isgs_companycode);

       

      }

       

      if(normalized.events === undefined){

      normalized.events = custom_events;

      }

      return normalized;

      };


      You can also look at the built-in Respond normalization script files for reference, such as normalize_alerts.js. For more information, see "Configure Custom Respond Server Alert Normalization" in the NetWitness Respond Configuration Guide.

    Reference Log Decoder

    For full functionality, make sure your reference Log Decoder is at 11.5 or later. If you never set up a reference Log Decoder, there is no need to take action. For details, see the Log Parser Customization Guide.

    Windows Log Collector

    Update the Windows Log Collector UUID

    After upgrading to 11.5, for each Windows Log Collector configured in your environment, run the following command on the NW Server:

    wlc-cli-client --update-to-uuid --host <WLC host address>

    User Entity Behavior Analytics

    IMPORTANT: Every UEBA deployment when upgraded requires additional steps to complete the upgrade process. When you upgrade from 11.3.0.0 to 11.3.2.0 to 11.5.0.0, you must follow UEBA instructions in the Upgrade Guide for 11.3.2.0, before you upgrade to 11.5.0.0.

    Note: When you upgrade to 11.5.0.0 from 11.4.x, you don't need to rerun the UEBA system for the last 28 days, if you don't update the current processing schemas. When you upgrade to 11.5.0.0 from a version prior to 11.4.x, the UEBA system runs a rerun automatically.

    1. (For Virtual Machines Only) Update the airflow parallelism on VM.
      If the UEBA system is running on VM, update the airflow parallelism to be 64 by running the following command as root from the UEBA host.

      sed -i "s|parallelism = 256|parallelism = 64|g" /var/netwitness/presidio/airflow/airflow.cfg

    2. Update the UEBA configuration using the following command from the UEBA machine.
      python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py
    1. (Optional) Update the UEBA processing schema, using the following script:

      python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/utils/airflow/reset_presidio.py script.

      RSA recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

      For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

    2. Run the airflow upgrade DAG.

      • Go to Airflow main page https://<UEBA-host-name>/admin

      • Enter the admin username and password.
      • Click the Play in presidio_upgarde_dag_from_<previous_version> to_11.5.0.0.

        Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

    3. Set the appropriate "Boot Jar Pools" slots:

      • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

      • Virtual Appliance: Update the spring_boot_jar_pool and the retention_spring_boot_jar_pool slot values to 5.
        To update the “Spring Boot Jar Pools” slots, Go to the Airflow main page, tap the “Admin” tab at the top bar and tap “Pools”.
      1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
        User: admin
        Password: The environment deploy admin password.
      1. Click on the pencil mark of the Pools to update the slot values.

     

    • Edit the spring_boot_jar_pool and update the slots amount to 5.

     

    Previous Topic:Upgrade Tasks
    You are here
    Table of Contents > Post Upgrade Tasks

    Attachments

      Outcomes