Endpoint Config: Troubleshooting

Document created by RSA Information Design and Development on Mar 16, 2018Last modified by RSA Information Design and Development on Jul 9, 2019
Version 14Show Document
  • View in full screen mode
  

This section provides information about possible issues when using the RSA NetWitness Endpoint.

Agent Communication Issues

                 
Issue

Agent is unable to communicate with the Endpoint server.

Explanation

This could be due to one of the following reasons:

  • In the agent packager:

    • Server IP is incorrect
    • Port specified is not available for communication with the Endpoint server
  • Endpoint Server or Nginx Server is not running
  • Firewall or IP table rules are blocking the connection between the host and Endpoint Server
  • Agent is inactive or manually deleted from the UI
Resolution
  • Check if the Endpoint Server and Nginx Server are reachable
  • Uninstall the agent, reboot the host, and reinstall the agent
  • Update Firewall or IP table rules, if required
                 
Issue

Agent takes a long time to scan.

Explanation

Sometimes, the NetWitness Endpoint scan takes a long time to complete. This is because of the CPU usage by other antivirus programs (such as Windows Defender, McAfee, Norton, and so on) that may be installed on the agent machines.

Resolution

It is recommended to whitelist the NWEAgent.exe file in the antivirus Windows Suite.

                 
Issue

You want to change the responsiveness of the Agent.

Explanation

Depending on your installation, you can adjust Beacon intervals to change how responsive your agents are.

Resolution

If resources are not a concern, you can lower the HTTPS Beacon Interval and UDP Beacon Intervals. If resources are a concern and responsiveness of the agent is not, you can increase these intervals.

Packager Issues

                     
Message

Failed to load the client certificate.

Issue

Incorrect certificate password.

Explanation

While generating the agent installer, the certificate password does not match with the one provided while downloading the agent packager from the UI.

Resolution

Specify the correct certificate password.

 

                     
Message

An unexpected error has occurred attempting to retrieve this data.

Issue

When attempting to access the Packager tab, it opens with the message.

Explanation

Endpoint Server might be down or not reachable.

Resolution

Check the status of the Endpoint Server under ADMIN > Service. If the service is not running, start the Endpoint Server.

Scan Schedule Issues

                     
Message

An unexpected error has occurred attempting to retrieve this data.

Issue

When attempting to access the Scan Schedule tab, it opens with the message.

Explanation

Endpoint Server might be down or not reachable.

Resolution

Check the status of the Endpoint Server under ADMIN > Service. If the service is not running, start the Endpoint Server.

Health and Wellness Issues

                 
BehaviorEndpoint metadata is not available in the INVESTIGATE > Navigate or Event Analysis view.
Issue

The health check of the Meta-Ld-Buffer shows Unhealthy in the Health and Wellness with the following exceptions:

dataprocessor-5] WARN MetaManagement|Meta Forwarding waiting for free buffer in Log decoder

Resolution

Make sure that:

  • Capture is enabled on the Log Decoder
  • Metadata is configured properly

 

                 
Behavior

For the NetWitness Endpoint 4.4.0.2 or later, metadata is not reaching the Endpoint Server.

Issue

The health of the Meta-Ld-Buffer shows Unhealthy in the Health and Wellness with the following exceptions:

dataprocessor-5] WARN MetaManagement|Meta Forwarding waiting for free buffer in Log decoder

Explanation

Make sure that:

  • Certificate is obtained and imported to the NetWitness 4.4.0.2 or later Console Server.
  • NetWitness Investigate option is enabled in the NetWitness Endpoint UI.
  • Metadata forwarding is configured in the NetWitness 4.4.0.2 or later Console server.

 

                     
Behavior

The health check of the Data.Application.Connection-Health for Endpoint Server shows Unhealthy.

Issue

Either Mongo or Endpoint Server service is down.

Explanation

For error details, check the Endpoint Server logs in /var/log/netwitness/endpoint-server/endpoint-server.log.

Resolution

Restart the Mongo or Endpoint Server service.

 

                     
Behavior

The health check of the Endpoint.Health.Overall-Health statistic shows Unhealthy.

Issue

Either Mongo or Endpoint Server service is down.

Explanation

Check the other Endpoint Server health statistics (such as, Data.Application.Connection-Health, Endpoint.Health.Ld-Buffer-Health) to identify which stats shows Unhealthy. If one of them is Unhealthy, the overall health of the Endpoint Server shows Unhealthy.

Resolution

See the resolution for these statistics in the Health and Wellness Issues section.

 

                 
Issue

Agent rejection count is more than the alarm threshold.

Explanation

The agent rejected count is more than a specific limit and your custom policy is triggered. For example, agent rejected count for the last 5 hours is 10 percent of the deployed agents.

Resolution

Check the overall health of the Endpoint Server and the sizing guidelines.

 

                 
Issue

Storage size of the Data application statistic has exceeded the alarm threshold.

Explanation

The storage size of the Data application has exceeded the threshold (for example, 75%), and the custom policy is triggered.

Note: By default, the server automatically deletes the older data when it reaches 80% of the disk space.

Resolution

Check the threshold set in the data retention policy.

 

                 
Issue

The health check of the Data.Application.Connection-Health shows Unhealthy or Fatal.

Explanation

The Mongo service is down.

Resolution

Check if the Mongo service is running and the Endpoint Server logs for error details.

 

                 
Issue

The agent request count shows 0 for a alarm threshold.

Explanation

The agent request count shows 0 for the entire day or week. This could be due to one of the following reasons:

  • In the agent packager:

    • Server IP is incorrect .
    • Port specified is not available for communication with the Endpoint server.
  • Endpoint Server or Nginx Server is not running .
  • Firewall or IP table rules are blocking the connection between the host and Endpoint Server.
  • Agent is inactive or manually deleted from the UI.
Resolution
  • Check if the Endpoint Server and Nginx Server are reachable.
  • Uninstall the agent, reboot the host, and reinstall the agent.
  • Update Firewall or IP table rules, if required.

Installation Issue

                     
Behavior

NetWitness Platform allows multiple instances of Endpoint Hybrid or Endpoint Log Hybrid to be installed.

Issue

Only one instance of the Endpoint Hybrid or Endpoint Log Hybrid can be used for endpoint data.

Explanation

While the installation of Endpoint Hybrid or Endpoint Log Hybrid is in-progress, you can install another instance and the installation will be successful.

Resolution

You must delete all instances of Endpoint Hybrid or Endpoint Log Hybrid except the one that you want to use for endpoint data.

Finding Inactive Agents Issue

                 
Issue

Agent might be inactive or has not communicated with the Endpoint Server for a long time.

Explanation

A list of inactive agents is available in the Mongo database with the agent ID. Using this information, you can search for further details of the inactive agents.

Resolution

To find inactive agents in your deployment, perform the following:

  1. Open the Endpoint Server log file from /var/log/netwitness/endpoint-server/endpoint-server.log and search for Agent <ID> does not exist string.
  2. Copy the agent ID displayed in the log file.
  3. Search for the agent ID in the NGINX access log file (/var/log/nginx/access.log) to retrieve the following details of an inactive agent:

    • IP Address
    • Date and time that the agent became inactive
    • Location

NGINX Issue

                 
Issue

Nginx rejects post requests exceeding request size 100 GB.

Explanation

By default, the payload size in the NGINX server is set to 100 GB. This causes any data post request exceeding 100 GB to fail.

Resolution

Add the following setting to the Nginx configuration file (/etc/nginx/conf.d/nginx.conf) and restart the Nginx server.

client_max_body_size xx

where, xx is the size that you want to set.

 

             
Issue

Endpoint server unable to communicate with the Relay Server.

Resolution

Check the Nginx error logs (/var/log/nginx/error.log). If the issue is Ngnix server not able to resolve the hostname.
Perform the following:

  1. Map an IP to hostname of the Reley Server (For example, 121.0.0.0 john-PC) in the /etc/hosts.dnsmasq file of Node 0.
  2. Restart the dnsmasq service.
    systemctl restart dnsmasq
  3. Restart the Endpoint Log hybrid Nginx server.
    systemctl restart nginx

Relay Server Issues

             
Issue

Relay Server test connection failed.

Resolution
  1. Check if the hostname or IP and port of the Relay Server are correct.
  2. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint server. Perform the following:
    1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3 rahost rarport
      If the Relay Server is not reachable contact your Administrator.
    2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/rar-install.log) and check the Endpoint server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep <Endpoint Server Revision ID>
    3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/rar-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-endpoint-rar-server

 

                 
IssueRelay Server installer generation fails with an error message ‘Unable to download the installer. Retry after sometime’.
ExplanationDependencies of the Relay Server are not resolved completely.
Resolution

You must retry the download after 5-10 minutes. If the download still fails even after all dependencies are downloaded in the Endpoint server, contact the RSA Customer Support.

Note: You can check ‘Finished downloading all RAR dependencies’ message in the Endpoint server logs at /var/log/netwitness/endpoint-server/endpoint-server.log, to see if the dependencies are downloaded.

 

             
IssueRelay Server installation fails due to missing or corrupted dependencies.
Resolution

Re-download the installer dependencies, perform the following:

  1. Go to ADMN > Endpoint server service > select > View > Explore.
  2. In the Endpoint server configuration, make sure endpoint.rar.installer.download-on-restart boolean is set to true (by default it is true).
  3. Restart the Endpoint server using the following command:
    systemctl restart rsa-nw-endpoint-server
    Fresh dependencies will be downloaded to the local directory in the Endpoint server. This may take few minutes.
  4. Download the Relay Installer.
  5. Run the Relay Server Installation Script.
    For more information, see (Optional) Installing and Configuring Relay Server.

You are here
Table of Contents > Troubleshooting

Attachments

    Outcomes