Skip navigation
All Places > Products > RSA NetWitness Platform > RSA NetWitness Platform Online Documentation > Documents
Log in to create and rate content, and to follow, bookmark, and share content with other members.

UEBA Configuration: Troubleshooting

Document created by RSA Information Design and Development Employee on Nov 11, 2020•Last modified by RSA Information Design and Development Employee on Jan 6, 2021
Version 4Show Document
  • Comment • 0
  • View in full screen mode
 

This section provides information about possible issues when using RSA NetWitness UEBA.

Adapter logs are not written on upgrade

                   
Problem

When you upgrade from 11.2 or 11.3 to 11.5, flume is using a wrong library to write logs. The logs are written to slf4j-log4j12-1.7.25.jar instead of logback-classic-1.2.3.jar due to which the adaptor logs are not written.

Cause This happens because the flume logs library is not updated.
Solution

To solve this issue, you must delete the slf4j-log4j12-1.7.25.jar libraries from the flume library directory available on the UEBA machine using the following commands

  • rm /var/netwitness/presidio/flume/plugins.d/PresidioStreamingSource/libext/slf4j-log4j12-1.7.25.jar
  • rm /var/netwitness/presidio/flume/lib/slf4j-log4j12-1.7.25.jar

Files are not deleted from Elasticsearch DB

                   
Problem

Metricbeat and Packetbeat documents are not deleted from Elasticsearch DB due to an issue in version 11.4.x. As a result, the Elasticsearch DB stopped working properly and is marked with a “red” health status.

To verify whether the environment is affected by this issue, run the following APIs from the UEBA machine:

curl -s http://localhost:9200/_aliases?pretty=true | grep -E 'metricbeat|packetbeat'

If the returned results contain Metricbeat or Packetbeat with dates that are older than 30 days, the environment retains old and unwanted data and is affected by this issue.

If you get empty results, stop the following services, run the above command again and validate the results again.

systemctl stop packetbeat

systemctl stop metricbeat

Cause UEBA failed to delete Metricbeat and Packetbeat documents from Elasticsearch DB, as a result of using incorrect API.
Solution

Complete the following steps to delete the documents from Elasticsearch DB:

  1. Remove the Packetbeat and the Metricbeat indexes from Elasticsearch using the following commands on the UEBA machine:

    curl -X DELETE http://localhost:9200/packetbeat-*-*

    curl -X DELETE http://localhost:9200/metricbeat-*-*

  2. Wait for the operation to complete and make sure the result - {"acknowledged": true} is returned.

  3. Update the indexes URLs at the metrics cleanup builder job using the following commands on the UEBA machine:

    sed -i "s|packetbeat-6.1.2-|packetbeat-*-|g" /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/builders/maintenance/presidio_metrics_cleanup_builder.py

    sed -i "s|metricbeat-6.0.0-|metricbeat-*-|g" /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/builders/maintenance/presidio_metrics_cleanup_builder.py

  4. Make sure that the next run of the maintenance_flow_dag.presidio-metrics-cleanup job is completed successfully by performing the following steps:

    • Go to Airflow home page.
    • On the main page tap on maintenance_flow_dag.
    • Click on presidio-metrics-cleanup.
    • Click Zoom into sub DAG on the pop-up window.
    • Click clean_presidio_system_metrics and then click View log.
      Make sure that the responses for both delete requests are:{"acknowledged": true}

    [2020-10-08 09:10:26,576] {logging_mixin.py:112} INFO - [2020-10-08 09:10:26,576] {base.py:83} INFO - DELETE http://localhost:9200/%3Cmetricbeat-*-%7Bnow%2Fd-15d%7D%3E [status:200 request:0.004s]

    [2020-10-08 09:10:26,576] {logging_mixin.py:112} INFO - [2020-10-08 09:10:26,576] {presidio_metrics_cleanup_builder.py:74} INFO - response: {"acknowledged": true}

    [2020-10-08 09:10:26,578] {logging_mixin.py:112} INFO - [2020-10-08 09:10:26,578] {base.py:83} INFO - DELETE http://localhost:9200/%3Cpacketbeat-*-%7Bnow%2Fd-15d%7D%3E [status:200 request:0.001s]

    [2020-10-08 09:10:26,579] {logging_mixin.py:112} INFO - [2020-10-08 09:10:26,578] {presidio_metrics_cleanup_builder.py:83} INFO - response: {"acknowledged": true}

User Interface Inaccessible

                   
Problem

The User Interface is not accessible.

Cause You have more than one NetWitness UEBA service existing in your NetWitness deployment and you can only have NetWitness UEBA service in your deployment.
Solution

Complete the following steps to remove the extra NetWitness UEBA service.

  1. SSH to NW Server and run the following commands to query the list of installed NetWitness UEBA services.
    # orchestration-cli-client --list-services|grep presidio-airflow
    ... Service: ID=7e682892-b913-4dee-ac84-ca2438e522bf, NAME=presidio-airflow, HOST=xxx.xxx.xxx.xxx:null, TLS=true
    ... Service: ID=3ba35fbe-7220-4e26-a2ad-9e14ab5e9e15, NAME=presidio-airflow, HOST=xxx.xxx.xxx.xxx:null, TLS=true

  2. From the list of services, determine which instance of the presidio-airflow service should be removed (by looking at the host addresses).

  3. Run the following command to remove the extra service from Orchestration (use the matching service ID from the list of services):
    # orchestration-cli-client --remove-service --id <ID-for-presidio-airflow-form-previous-output>

    4. Note: Run the following command to update NW Server to restore NGINX:
    # orchestration-cli-client --update-admin-node

  4. Log in to NetWitness Platform, go to (Admin) > Hosts, and remove the extra NetWitness UEBA host.

Get UEBA Configuration Parameters

                 
Issue

How to get UEBA configuration parameters?

Explanation

In order to get the UEBA configuration main parameters, run the curl http://localhost:8888/application-default.properties command from the UEBA machine.
Command to get UEBA Configuration Parameters
The main parameters which will be returned are the following:

  • uiIntegration.brokerId: The Service ID of the NW data source (Broker / Concentrator)
  • dataPipeline.schemas: List of schemas processed by the UEBA
  • dataPipeline.startTime: The date the UEBA started consuming data from the NW data source
  • outputForwarding.enableForwarding: The UEBA Forwarder status
Resolution

See the resolution for these statistics in the Troubleshooting UEBA Configurations section.

Scaling Limitation Issue

When installed on a Virtual Machine, UEBA can process up to 20 million network events per day. Based on this limitation, you may encounter the following issues.

             
Issue

How to determine the scale of network events currently available, to know if it exceeds the UEBA limitation.

Solution

To know the network data limit, perform the following :

  • Run the query on the Broker or Concentrator that connects to UEBA using NetWitness UI:

service=443 && direction='outbound' && analysis.service!='quic' && ip.src exists && ip.dst exists && tcp.srcport!=443

Calculate the total number of events for the selected days (including weekdays with standard workload). If the average is above 20 million per day then it indicates that UEBA’s supported scale is exceeded.

 

             
Issue

Can UEBA for Packets be used if UEBA's supported scale is exceeded?

Solution

You must create or choose a Broker that is connected to a subset of Concentrators that does not exceed the supported limit.

To know the network data limit, perform the following :

  • Run the query on the Concentrator that connects to UEBA using NetWitness UI:

service=443 && direction='outbound' && analysis.service!='quic' && ip.src exists && ip.dst exists && tcp.srcport!=443

Calculate the total number of events for the selected days (including weekdays with standard workload). If the average is above 20 million per day then it indicates that UEBA’s supported scale is exceeded.

Note: The Broker must query all the available and needed data needed such as logs, endpoint and network (packets). UEBA packets models are based on the whole environment. Hence, make sure that the data parsed from the subset of Concentrators is consistent.

UEBA Policy Issue

             
IssueAfter you create a rule under UEBA policy, duplicate values are displayed in the Statistics drop-down.
Solution

To remove the duplicate values, perform the following:

  1. Log in to MongoDB using following command:mongo admin -u deploy_admin -p {Enter the password}
  2. Run the following command on MongoDB:
    use sms;
    db.getCollection('sms_statdefinition').find({componentId :"presidioairflow"})
    db.getCollection('sms_statdefinition').deleteMany({componentId :"presidioairflow"})

Troubleshoot Using Kibana

             
Issue

After you deploy NetWitness UEBA, the connection between the NetWitness Platform and NetWitness UEBA is successful but there are very few or no events in the Users > OVERVIEW tab.

  1. Log in to Kibana.
  2. Go to Table of Content > Dashboards > Adapter Dashboard.
  3. Adjust the Time Range on the top-right corner of the page and review the following:
    • If the new events are flowing.
    • In the Saved Events Per Schema graph, see the number of successful events per schema per hour.
    • In the Total Events vs. Success Events graph, see the total number of events and number of successful events. The number of successful events should be more every hour.

    For example, in an environment with 1000 users or more, there should be thousands of authentication and file access events and more than 10 Active Directory events. If there are very few events, there is likely an issue with Windows auditing.

Solution

You must identify the missing events and reconfigure the Windows auditing.

  1. Go to INVESTIGATE > Navigate.
  2. Filter by devide.type= device.type “winevent_snare” or “winevent_nic”.
  1. Review the events using reference.id meta key to identify the missing events.
  2. Reconfigure the Windows auditing. For more information, see NetWitness UEBA Windows Audit Policy topic.

 

             

Issue

The historical load is complete and the events are coming from Adapter dashboard but no alerts are displayed in the Users > OVERVIEW tab.
Solution
  1. Go to Kibana > Table of content > Scoring and model cache.
  2. Adjust the Time Range from the top-right corner of the page, and see if the events are scored.

 

             

Issue

The historical load is complete but no alerts are displayed in the Investigate > Users tab.
Solution

  1. Go to Kibana > Dashboard > Overview.

  2. Adjust the Time Range from the top-right corner of the page, and see how many users are analyzed and if any anomalies are found.

Troubleshoot Using Airflow

             
IssueAfter you start running the UEBA it is not possible to remove a data source during the run process else the process stops.
Solution

You must either continue the process till it completes or remove the required data source from UEBA and rerun the process.

 

             
IssueAfter you deploy UEBA and if there are no events displayed in the Kibana > Table of content > Adapter dashboard and Airflow has already processed the hours but there are no events. This is due to some communication issue.
Solution

You must check the logs and resolve the issue.

  1. Log in to Airflow.
  2. Go to Admin > REST API Plugin.
  3. In the Failed Tasks Logs, click execute.
    A zip file is downloaded.
  4. Unzip the file and open the log file to view and resolve the error.
  5. In the DAGs > reset_presidio, click Trigger Dag.
    This deletes all the data and compute all the alert from the beginning.

Note: During initial installation, if the hours are processed successfully but there are no events, you must click reset_presidio after fixing the data in the Broker. Do not reset if there are alerts.

Previous Topic:UEBA Configuration
You are here
Table of Contents > UEBA Configuration Troubleshooting

Attachments

    Outcomes

      Visibility: RSA NetWitness Platform Online Documentation154 Views
      Last modified on Jan 6, 2021 5:30 AM
      Tags:
      Ratings:

        Recommended Content

        Incoming Links