Skip navigation
Log in to create and rate content, and to follow, bookmark, and share content with other members.

Endpoint Config: Troubleshooting

Document created by RSA Information Design and Development Employee on Mar 16, 2018Last modified by RSA Information Design and Development Employee on Nov 2, 2020
Version 30Show Document
  • View in full screen mode
 

This section provides information about possible issues when using RSA NetWitness Endpoint.

Agent Communication Issues

                 
Issue

Agent Last Seen Time column is not updated in the UI.

Explanation

The issue could be due to any one of the following:

  • Agent is inactive
  • Agent data is not processed if the Endpoint.Health.Overall-Health statistic shows Unhealthy due to which all the agent data including agent last seen time is not updated.
Resolution

See the resolution for these statistics in the Health and Wellness Issues section.

 

                 
Issue

Agent is unable to communicate with the Endpoint Server.

Explanation

This could be due to one of the following reasons:

  •  

  • Agent is inactive.
  •   
  • Endpoint Server settings is incorrect in the agent packager or policy configuration, or not available for communication.
  • Endpoint Server or Nginx Server is not running .
  • Firewall or IP table rules are blocking the connection between the host and Endpoint Server.
Resolution
  • Check if the Endpoint Server and Nginx Server are reachable.
  • If the Endpoint Server settings are incorrect, uninstall the agent, download the agent packager, and reinstall the agent.
  • Update firewall or IP table rules, if required.
                 
Issue

Agent takes a long time to scan.

Explanation

Sometimes, the NetWitness Endpoint scan takes a long time to complete. This is because of the CPU usage by other antivirus programs (such as Windows Defender, McAfee, Norton, and so on) that may be installed on the agent machines.

Resolution

It is recommended to whitelist the <service.exe> (name provided in the packager, by default, the service name is NWEAgent.exe) file in the antivirus suite.

 

                 
Issue

You want to change the responsiveness of the Agent.

Explanation

Depending on your installation, you can adjust Beaconing intervals to change how responsive your agents are.

Resolution

If resources are not a concern, you can lower the HTTPS Beacon Interval and UDP Beacon Intervals. If resources are a concern and responsiveness of the agent is not, you can increase these intervals.

 

                 
IssueAgent is unable to generate network tracking events in Insights mode.
ExplanationVerify that Windows Management Instrumentation (WMI) service is running.
Resolution
  • Run Services.msc and look for Windows Management Instrumentation (WMI) service.
  • Go to properties and change the Startup type to Automatic.

Packager Issues

                     
Message

Failed to load the client certificate.

Issue

Incorrect certificate password.

Explanation

While generating the agent installer, the certificate password does not match with the one provided while downloading the agent packager from the UI.

Resolution

Specify the correct certificate password.

Health and Wellness Issues

Endpoint Issues

                 
BehaviorThe health check of the Endpoint.Health.Overall-Health statistic shows Unhealthy.
Issue

Endpoint Server service or required resources are not available or not in a usable state. This could be due to one of the following reasons:

  • Unable to forward Endpoint meta data to the Log Decoder.

  • Endpoint Log Hybrid disk usage reaches the specified limit.

  • Mongo DB is down or excessive read and write errors during processing.

Resolution

Disk Usage and Mongo Issues

                 
Behavior

The health check of the Data.Application.Connection-Health Application, Data Store Disk Usage or Data Persistence for Endpoint Server shows Unhealthy.

Issue
  • Data.Application.Connection-Health Application or Data Persistence shows Unhealthy, if Mongo service is down or fails due to authentication.
  • Data Store Disk Usage shows Unhealthy, if Endpoint Server Mongo storage size has exceeded the threshold. By default, the server automatically delete the old data when it reaches 80% of the disk space.
Resolution
  • For Data.Application.Connection-Health Application or Data Persistence issue, you must check the Endpoint server logs (/var/log/netwitness/endpoint-server/endpoint-server.log) and Mongo logs (/var/log/mongodb/mongod.log), and:
    • If the issue is due to authentication, you must reissue the certificate. For more information, see "Service Certificate Reissue" section in the System Maintenance Guide.
    • If the issue is due to Mongo service is down, you must restart the Mongo.
  • For Data Store Disk Usage issue, you must increase the storage or configure data retention settings to clear the old data. For more information, see Configuring Data Retention Policy.

Log Decoder Issues

                     
BehaviorEndpoint metadata is not available in the Investigate > Navigate or Events view.
Issue

The health check of the Log Decoder Buffer and Meta Forward shows Unhealthy in the Health and Wellness.

Explanation

The issue could be due to any of the following reasons:

  • Log Decoder capture is not started.
  • Concentrator aggregation is not started.
  • Log Decoder connection issue.
  • Log Decoder buffer usage is beyond the specified limit.
Resolution

Make sure that:

  • Capture is enabled on the Log Decoder.
  • Aggregation is enabled on the Concentrator.
  • Meta forwarding is configured properly.

Note: Make sure Capture Autostart is enabled in the Service Config view for Log Decoder and Aggregate Autostart is enabled in the Service Config view for Concentrator.

File Log Policy Issues

Invalid Policy or Bad Connection Issues

                 
Issue

Policies can be invalid for a variety of reasons. Some examples:

  • No sources found if the policy is enabled.
  • Invalid or missing typespec file
  • No destination is reachable for a file log policy event source type

Additionally, if capture is stopped on the destination Log Decoder, Endpoint Agents will send an error to the Endpoint Server saying that they failed to connect.

Also, if there is a lot of data to be processed for Agents collecting File data (when File Policy is enabled) , there is a possibility that Log Decoder buffer becomes full. If this happens, the Log Decoder cannot process any requests from the Agents communicating via EPS.

Explanation

The system is dynamic in nature, which means its state can change: event sources can lose their connection, typespec files can be altered or deleted, and other changes can occur that can invalidate a previously valid policy.

Resolution

To help identify the specific issue, check the log file on the Endpoint Server that reports the error:

/var/log/netwitness/endpoint-server/endpoint-server.audit.log

Relevant errors will be listed as FileLogError in the log file.

If you experience this issue, you can do the following:

  1. Try to identify and target higher-value data, thus limiting the total amount of data being processed.
  2. Enable throttling in the File policy to smooth out the peaks in usage.
  3. If you really do need to process more data on a regular basis, consider server-side hardware upgrades.

Reset File Collection Bookmarks

                 
Issue

If the system is not configured correctly, NetWitness Platform might collect logs and not be able to parse them. Or, files might get sent, but for some reason, not make it to the Log Decoder (for example if communication is via UDP and there is a network connectivity issue).

In these and other cases, you can reprocess these "missing" log files.

Explanation

For whatever reason, you may need to reprocess logs from the beginning of the file.

Resolution

Reset bookmarks for an event source type using the procedure described here: Reset File Collection Bookmarks.

Missing Log Collectors and Event Sources in the User Interface

                 
Issue

Some log collectors or event sources seem to be missing from the list of available items.

Explanation

The Filter drop-down menus (types, log collectors, and log decoders) only show values that are in the event sources database, rather than all possible values. For example, if you have a log collector that has not yet collected any logs, then it is missing from the list.

Resolution

Collect logs from a specific log collector and event source, and then they should appear as items in the appropriate menu.

Relay Server Issues

Test Connection Issues

             
Issue

Relay Server test connection failed.

Resolution
  1. Check if the hostname or IP and port of the Relay Server are correct.
  2. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint Server. Perform the following:
    1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3 <relayhost> <relayport>
      If the Relay Server is not reachable contact your Administrator.
    2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/relay-install.log) and check the Endpoint Server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep <Endpoint Server Revision ID>
    3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/relay-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-relay-server

 

                 
IssueRelay Server installer generation fails with an error message ‘Unable to download the installer. Retry after sometime’.
ExplanationDependencies of the Relay Server are not resolved or downloaded completely.
Resolution

You must retry the download after 5-10 minutes. If the download still fails even after all dependencies are downloaded in the Endpoint Server, contact the RSA Customer Support.

Note: You can check ‘Finished downloading all Relay Server dependencies’ message in the Endpoint Server logs at /var/log/netwitness/endpoint-server/endpoint-server.log, to see if the dependencies are downloaded. If the download fails due to yum related issues, then you must clean yum repo using the command yum clean all and restart the Endpoint Server.

Installation Issues

             
IssueRelay Server installation fails due to missing or corrupted dependencies.
Resolution

Re-download the installer dependencies, perform the following:

  1. Go to (Admin) > Endpoint Server service > select > View > Explore.
  2. In the Endpoint server configuration, make sure endpoint.relay.installer.download-on-restart boolean is set to true (by default it is true).
  3. Restart the Endpoint server using the following command:
    systemctl restart rsa-nw-endpoint-server
    Fresh dependencies will be downloaded to the local directory in the Endpoint Server. This may take few minutes.
  4. Download the Relay Installer.
  5. Run the Relay Server Installation Script.
    For more information, see (Optional) Installing and Configuring Relay Server.

You are here
Table of Contents > Troubleshooting

Attachments

    Outcomes