NwConsole provides the following commands that are helpful when troubleshooting Security Analytics:
- whatIsWrong: Provides a snapshot of a service's configuration, stats, and failure and warning logs for a specified past period of time.
- dbcheck: Performs consistency checking of database files.
- topQuery: Helps pinpoint queries that are taking an excessively long time to run.
- netbytes: Troubleshoots the network connections on the current host
- netspeed: Troubleshoots the connection between the host computer running NwConsole and the remote computer connected to it using the
The following sections as well as the NwConsole help and topic information (man) pages, provide additional information.
When a service is not working correctly, the reason is usually somewhere in the logs that the service has written. You can use the
whatIsWrong console command to obtain a snapshot of a service's configuration, stats, and failure and warning logs (with surrounding context logs) for a specified past period of time, which defaults to the previous seven days. You can save the results of running whatIsWrong into a specified plain text file. The output of this command can be a useful starting point to help determine what is currently wrong with a service.
To use the whatIsWrong console command, log on to the service to troubleshoot using the
login command, and run the
help whatIsWrong to see all of the available parameters, including the number of days/hours to look back for events, the pathname to store results, whether or not to append or overwrite the results file, and the delimiter to use for log fields. You can also limit the number of most recent logs used to find context, and you can specify how many context logs per warning/failure log to retrieve.
Whenever you receive a request for logs for a Core service, you should run the whatIsWrong command first and use the results collected as a starting point.
dbcheck command is used to perform consistency checking of database files (session, meta, packets, logs, stats, and so on). This might be necessary when a service cannot start because of errors in the consistency of the database files. Normally a service would automatically recover and correct any consistency issues on startup, but there are times when this does not occur. When a service starts (like Decoder), it typically does not read or open most database files in order to start quickly. It assumes most files are in a consistent state and only does a cursory check of the most recently written files. If there are problems,
dbcheck can perform those consistency checks, but ONLY if the service is not running.
For example, you can check a single file:
You can also use wildcards to check multiple files:
topQuery command can help pinpoint queries that are taking an excessively long time to run. This command parses the audit logs of a service and returns the top N longest running queries for the specified time period.
The easiest way to run it is to log on to the service (usually a Broker or Concentrator) and type
topQuery. The default behavior is to return the top 100 longest running queries for the last seven days.
help topQuery for the list of parameters. Here are some additional examples with explanations:
topQuery hours=12 top=10
This command returns the top 10 queries for the last 12 hours.
topQuery time1="2015-03-01 00:00:00" time2="2015-03-14 00:00:00"
This command returns the top 100 queries between March 1, 2015 and March 14, 2015. Times are in UTC, not local.
topQuery input=/var/log/messages output=/tmp/top20.txt top=20 user=sauser1
Instead of connecting to a service, it parses the syslog audit messages for the top 20 queries in the last 7 days, but only for queries executed by user sauser1. It writes the top 20 queries to /tmp/top20.txt instead of the console screen. The parameter user is a regex, so you can specify multiple usernames by writing something like
netbytes command is very useful for troubleshooting the network connections on the current host. It displays continuous send and receive statistics for all network interfaces. Once executed, you must press Ctrl-C to exit this command, which also exits NwConsole.
netspeed command is used to troubleshoot the connection between the host computer running NwConsole and the remote computer connected to it through the
login command. You must supply the amount of bytes to transfer and it will time the speed of the connection. The netspeed command is very useful for troubleshooting Aggregation performance issues that might be network related.
login somedecoder:50004 admin ...
To troubleshoot the connection between a Concentrator and a Decoder, SSH into the Concentrator, run NwConsole, and then log on to the Decoder and run
netspeed. The output from the command gives you an indication of the maximum network throughput. If it is much less than the standard 1 Gbps interface, it could indicate a network issue.