Sys Maintenance: Troubleshoot Health and Wellness (Kibana)

Document created by RSA Information Design and Development Employee on Jan 30, 2020Last modified by RSA Information Design and Development Employee on Jul 24, 2020
Version 11Show Document
  • View in full screen mode

This topic describes how to troubleshoot Health & Wellness issues related to the third-party tool Kibana.


Unable to view data in the Kibana UI.

  1. Go to https://<admin-server-ip>:9200
    JSON response is displayed.

  2. If the JSON response is not displayed, there is some issue with the Elasticsearch. You must check logs on host on which you have deployed Health and Wellness at /var/log/netwitness/elasticsearch.log

  1. SSH to the Host on which you have deployed Health and wellness. For example, Admin Server.
  2. Execute the following command to check the health of Elasticsearch:
    curl https://localhost:9200/_cat/health -k -u username:password
    The Elasticsearch Status should be Green.



Unable to load kibana UI.

  1. Go to https://<admin-server-ip>:5601/status
  2. Check the status of Kibana. The Kibana Status should be Green.



An error ‘n of m shards failed’ or ‘unknown field in the index’ in the Kibana UI.


You must refresh the index patterns, perform the following:

1. Log in to Kibana and go to Management > Index Patterns.

2. Click nw* index pattern.

3. Click Refresh to refresh the index pattern on top right corner.

Note: If the issue still persists, refresh other index patterns such as nw-metricbeat* or nw-concentrator* and so on.



Time out error occurs when you reset or restore OOTB content.


You must ignore the error as reset or restore OOTB content will be updated successfully.



Few Kibana visualizations fails.


After you install Health and Wellness Beta, few Kibana visualizations fails with an error “Could not locate that index-pattern (id: nw-metricbeat), [click here to re-create it]”.


You must import missing OOTB content, perform the following:

  1. SSH to Admin Server and connect to nw-shell.
  2. Login to nw-shell and enter the username and password.
  3. Connect to metric server using the following command:
    connect metrics-server
  4. Navigate to the restore option using the following command:
    cd /rsa/metrics/content/restore-content
  5. To invoke the method, using the following command:

This will import the missing OOTB content and any changes made to the existing content is not affected.



Issue with disable-all for launch services.


If you have customized the configuration for a launch service, metrics reporting is not stopped even if disable-all is enabled.


You must restart the launch service for which configuration is customized. For example, if the configuration is customized for Context Hub Server, restart contexthub-server.



Unable to send data to elastic search once disk usage reaches 80% or above.


If the Elasticsearch disk usage reaches 85% or above,

the saved objects (index patterns, dashboards, visualizations etc) becomes read-only mode.

And, services does not write new metrics to Elasticsearch or allow to edit any saved objects.

ResolutionTo change the indexes to write mode, you must execute the following command on the host in which Elasticsearch is installed:
curl -k --cert /etc/pki/nw/elastic/elasticsearch-cert.pem --key /etc/pki/nw/elastic/elasticsearch-key.pem -X PUT -H "Content-Type: application/json" -d '{"index.blocks.read_only_allow_delete": null }' https://localhost:9200/_all/_settings

Note: This command is supported only with certificates.

You are here
Table of Contents > Health and Wellness > Monitor Health and Wellness using Kibana (BETA) > Troubleshooting Health and Wellness (BETA)