000012093 - How to troubleshoot issues using sendlogs

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000012093
Applies ToenVision 3.7.x, 4.0.0, 4.1.0
IssueThe purpose for this kb is to give more understanding of the sendlogs information and how we use it to troubleshoot an issue.
Customer has reported a complex issue which you can't resolve through the solutions provided in the KB or there are no documented solutions on it
CauseSometimes by just looking at the symptoms of the problem couldn't help us to determine the suitable cause of action. Other times, problems have not being resolved through applying patches. During these times, more information is required to diagnose the issue. Thus there is a need to understand more of the information collected of sendlogs and know how to use it to troubleshoot customer's issue.
ResolutionThis is not the absolute all in one solution to fix all customer's issue, rather it gives more idea how you can find out more about customer's environment to infer on the possible causes of issue

Things not included in sendlogs:

NIC System messages. You have to run lsdata to retreive additional logs from the NIC System

Debug messages for each of the NIC services. You will need to run the services in debug mode and obtain the log seperately

License file. You will need to obtain the licenses file separately from the CSD folder

Logs data for each of the event source. You will need to run lsdata to retrieve any logs collected from event source.

DSET report - You would need to run this seperately by obtaining the DSET tool from dell

Kernel memory dump - You would need to collect seperately in c:\windows (You would also need to first enable this in system properties)

Log files explanation and its usage:

Logs folder. This contains a repoistory of logs from the OS level as well as from the enVision application levels.

yyyy-mm-dd_deviceeps.csv - Contains the day average EPS for each of the event source. Can help to troubleshoot which event sources are using up resources to aid in troubleshoot collector high utilisation.

yyyy-mm-dd_stiestats.txt - contains the daily sitestat information

yyyy-mm-dd_IP_<event type>.evt (e.g: 2011.1.5_18.1.32_Application.evt) - contains the different types of windows event, those you see from the event viewer. This is useful to see any abnormal shutdown of NIC services, or finding patterns with other 3rd party devices when you experience system instability problems (e.g: you would get an error code or an explanation of the services if it crashes suddenly)

alerterService.html, checkpointService.html, collector.html, eaManagerConfig.html, filereaderservice.html, idsxmlService.html, issService.html, locatorService.html, logger.html, lsServerService.html, odbcService.html, packagerService.html, schedulerService.html, sdee.html, storageLocations.html, vacollector.html, vaprocessor.html, vulnerabilityService.html, webServerService.html, windowsService.html - contains parameters used to run each of the NIC services. e.g: logging level, thread priority. Good for troubleshooting performance issue (over utilised or under utilised). The best way to see if customer is having valid parameters is to compare it with your own testing environment containing the standard parametes. If these are not created, it also means the service hasn't completed starting or it has problem starting

More html - allsitedirectory.html & sitedirectory.html - These contains records of each of the event sources and the location of the log files. Good for troubleshooting when customer report some logs are not being retrieved. You can see from there on which time range, where are the logs are stored and trace whether the location still valid. Also good to determine if the locator service was running properly, as the locator servie is responsible for building this upon starting it. If you see these files being updated continously after you start the service, it means the repository is still building. During this time, you might not able to get all the logs when viewing them through event viewer or lsdata. The sitedirectory contains information of the event sources managed by the current site of the appliance you are colllecting. While the allsitedirectory contains rest of the sites within the domain, e.g: you have a master and slave site.

One more html! - datastorage.htm. This contains the correct storage locations available. If this file hasn't being produced, it would mean the NIC locator service has not started or not yet completed finish scanning the log directory.

appserver_install.log - check if the NIC App server has being installed properly. One common issue with NIC App Server not running properly is that the A-Srv have undergo a IP address change. After that, the NIC App Server needs to be reinstall for the change to take effect. Check the line starting "Installing "NIC App Server"... and then check the parameter of "-Djava.rmi.server.hostname=" along that line to make sure the IP address is correct

autoupdatelog.log - contains messages of applying event source update. Check on the end of the file for the line: "INFO  10 09 2010 16:08:34,491 %NIC-6-701001: AutomaticUpdate, AutoUpdateMain, -, package, 20100902-144020, Detail: 1: 2010/09/10 16:08:34, Automatic update completed." To see which is the latest event source update and whether it has being completed update. This file need to be exist on each of the appliances in the site to ensure completion of event source update

chassis_tag.txt - contains the Dell service tag number

*_traphandler.txt - contains audit log for all the SNMP trap handlers. It's useful when you configured event source to use SNMP for log collections. Each type of trap handlers will have its seperate .txt for tracking the audit messages

collector.txt - contains audit message on the collector service during start up and shutdown. This gives an easy way to track when services are started and stopped. It also helps to check for license validity. If you see "License expired" or "License is invalid", then you know the LC licenses has not applied correctly, thus no log collection. Finally also good to find out if the number of manage devices exceed license limit. Look for the "Loading x devices" to see how many event sources are managing at the moment.

contentupdate.log - contains audit message for applying VAM update. Look for the end of file for the line: INFO  10 09 2010 16:20:55,920 %NIC-6-701003: AutomaticUpdate, AutoUpdateMain, -, patch, 20100907-100940/service_20100201-010101, Detail: 1: date_time: 2010/09/10 16:20:55, package_id: 20100907-100940, patch_id: service_20100201-010101, completed successfully.

*_checkview.log & *_opsec_output.log - contains audit message for LEA client for checkpoint log collection. These are good for checking for the initial connection has being connected properly as well as the subsequent log collections process being run properly (where it will report number of events being collected at each retrieval )

Install.log, install_sp.log, install_hf.log and  - contains installation logs for installing the full release of enVision (e.g: 4.0), the Service pack (e.g: SP2) and the cumulative hot fixes (e.g: November roll up patch or SP4 P1)

nsdatabase.log - contains audit messages when A-Srv is requesting D-Srv for logs for report generations (both dashboard, ad-hoc and schedule report. Good to troubleshoot issues when A-Srv retrieve logs from D-Srv (but not at the stage of "executing SQL")

pi_webserver.log - contains audit messages for the Web Server services. Anything related to the UI operations should look at this log as first point of troubleshooting. Things like errors you see in the UI, or not able to load particular reports, blank UI will all log some kind of messages in this log, which can lead you more information on how to tackle UI issues

etc folder - contains more low level parameters of running the enVision services as well as event sources set up and custom reports

devices - contain all the message definitions
gots - contain all the ODBC client configuration
reports - contain any custom report templates created by customer
sqltbl - contain schema of all the tables used in reporting
trapd - contains snmp trap configuration for all the devices utilise snmp for collection
windows - position of last retrieved events, as well as the message format of all the windows events we collect

pi.ini - Parameters used by NIC server and NIC alerter service
PrivateI.ini & webserver.vmi - Parameters used by NIC Web server
ver.dat - versions with patch level
v2.0_installedContentDeviceList.txt - contain list of installed devices been upgraded to content 2.0

Windows folder - contain information about installation of enVision
installations\log - Installation log
system32\drivers\etc - contains nie-oe.dat & lsconfigurationwizard.cifg, these contain parameters used for installation

database folder - contain backup of NIC configuration database

Documents and Settings - contain any application crash dump / DR watson log for crash analysis

Legacy Article IDa54422