000032518 - Incidents page fails to load due to huge number of alerts waiting to be processed in RSA Security Analytics 10.5

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 5Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000032518
Applies ToRSA Product Set: RSA Security Analytics
RSA Version/Condition: 10.5.x
IssueIncidents page on the User Interface (UI) shows as blank.
Even though rsa-im service is running, the Administration -> Services Page shows it as down
CauseThis can happen when the Incident Management (IM) database grows to an extend that it crashes the IM service.
The database size may be huge because of the number of alerts that are coming in are too high.
ResolutionStop the Incident Management service with the following command:
# service rsa-im stop

Check the current size of the IM databse with the following command:
# mongo im -u im -p im
> show collections
system.indexes 3.48KB (uncompressed), 32.00KB (compressed)
system.users NaNundefined (uncompressed), NaNundefined (compressed)
categories 16.61KB (uncompressed), 32.00KB (compressed)
aggregation_rule 14.05KB (uncompressed), 32.00KB (compressed)
alert 1.17TB (uncompressed), 123.91GB (compressed)
incident 509.28KB (uncompressed), 1.16MB (compressed)
remediation_task 0.00B (uncompressed), 48.50KB (compressed)
tracking_id_sequence 182.00B (uncompressed), 32.00KB (compressed)
fs.files 0.00B (uncompressed), 48.50KB (compressed)
fs.chunks 0.00B (uncompressed), 48.50KB (compressed)

Remove old alerts from database with the following command by changing the date accordingly:
> db.alert.remove({receivedTime : { $gte: ISODate('2016-01-01'), $lt: ISODate('2016-02-02')}})

Once removed, re-index the database to free up space:
> db.alert.reIndex()

Once the database is cleared, check the rabbitmq queue for im.alert_queue.
Access the RabbitMQ web interface following the instructions in the article entitled How to access the RabbitMQ web interface to monitor and troubleshoot the event broker on an RSA Security Analytics Log Collector.
In the Web Interface, click on the im.alert.queue.

Check the number of queues messages.
If the number is above 100,000 , the IM service may crash again after the restart.
To purge the queue, scroll down and click on the "Purge" button.
NOTE: Any purged messages cannot be retrieved
Once the queue is purged, start the IM service on the SA head using the following command: 
# service rsa-im start
NotesThe RabbitMQ web interface should be opened from the SA server.

Attachments

    Outcomes