|Applies To||RSA Product Set: RSA NetWitness Platform|
RSA Product/Service Type: Event Stream Analysis, Correlation Server, Admin server
RSA Version/Condition: 11.X
|Issue||Whenever the below is reported|
- Delay in receiving ESA Alerts.
- Respond-Server stopped generating new alerts.
- Time difference between alert time and event time (either the alert time lags the event time or even sometimes, alert time is from the future).
The below could be reasons.
- Respond-Server Mongo DB is piled up not having enough space to generate new alerts.
- Mainly that is because Respond Alert Retention is not configured thus there are old alerts that are stored in the Respond-Server not giving a chance for new ones to be generated or gets generated after some time with a lag due to slowness in Respond-Server processing.
- Any Delay or Time Difference may point out some main reasons.
- Sessions behind on the Data sources themselves (to be checked on the data sources directly ex: Concentrator)
- Sessions behind on the ESA itself due to lack of Memory/CPU so Esper Engine does not have enough chance to process real-time events against ESA Rule (Lack of memory or CPU could be due to a busy/complex ESA Rule which reserves most of ESA resources)
- Time Difference could also be due to a difference in the pointer (represented in epoch time) that ESA uses to fetch data from the data sources.
ESA-Mongo-stats script is created which provides below information to ease the troubleshooting.
It runs on the ESA host and displays the below.
- ESA Mongo Databases >> Where you can check the Respond-Server size
- Total Number of Incidents stored in the Respond-Server.
- Total Number of Alerts that are stored in the Respond-Server >>> This Can help along with above point 1 and below point 5 to check if Respond Alert Retention is needed or not.
- ESA Rules/Modules sorted in descending order with regards to the total number of alerts each is firing >>> can point out to busy ESA rules.
- Display the oldest Alert stored in Mongo DB.
- Display the most recent Alert stored in Mongo DB.
- Display the oldest Incident stored in Mongo DB.
- Display the most recent Incident stored in Mongo DB.
Then the Script connects to the Admin server to fetch.
- Position Tracking per each data source in the ESA Deployment(s) to check if the pointer in real-time with regards to ESA time or not (DOC-53338 will help you with that).
How to deploy:
- With WinSCP, move the attached ESA-Mongo-stats-v1-5.sh script be under /root in your ESA host.
- Give Executable permission to the script.
#chmod +x ESA-Mongo-stats-v1-5.sh
- Run the script.
If the output is large, you can direct it to a file for your further analysis.
|Notes||The script will need the root password when it SSH to SA (nw-node-zero).|