000029989 - How to export alerts from the ESA MongoDB for RSA Security Analytics 10.4.x and later

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support on Nov 24, 2017
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000029989
Applies ToRSA Product Set: Security Analytics, NetWitness Logs & Packets

RSA Product/Service Type: Event Stream Analysis (ESA)

RSA Version/Condition: 10.4.x and above

Platform: CentOS
IssueThis article provides supplementary information on how to bulk export alerts from the Event Stream Analysis (ESA) appliances and virtual machines’ MongoDB on Security Analytics v10.4 and later.  There is no equivalent process to bulk export for v10.3 and lower as a different database type is used.
 
ResolutionNote: To perform any of below, you must first SSH into the ESA host
To export all alerts from the ESA host

# mongoexport –d esa –c alert –u esa –p esa –out /root/alerts.json

Parameter Syntax (can be seen using `mongoexport --help`:

-d == MongoDB database instance, for this you would use 'esa'
-c == collection within MongoDB DB instance, for this you would use 'alert'
-u == username. The default for esa DB is 'esa'
-p == password. The default for the esa user of esa DB is 'esa'
-out == output filename including path.

In the above example all the alerts will be exported to alerts.json file under the /root folder.
To export alerts by module (rule) name

# mongoexport -d esa -c alert --out /root/alerts_by_rule.json -u esa -p esa --query '{"module_name": "<rule_name>"}'

All the alerts that triggered by <rule_name> will be exported to the alerts_by_rule_name.json file under the /root folder.
To export alerts by module_name and time

# mongoexport -d esa -c alert --out /root/alerts_by_rule_time.json -u esa -p esa --query '{"module_name": "<rule_name>", "time": {$gte: new Date(<time1>), $lt: new Date(<time2>)} }'

All the alerts that triggered by <rule_name> between <time1> and <time2> will be exported to the alerts_by_rule_time.json file under the /root folder.
Example:

# mongoexport -d esa -c alert --out /root/alerts_by_rule_time.json -u esa -p esa --query '{ module_name : "Suspicious Login without any activity in windows hosts", "time": { $gt: new Date(1511407954000), $lt: new Date(1511494354000)} }'

Note: The mongo date type is similar a Unix Time/Epoch time, except it is number of milliseconds rather than number of seconds since midnight 1st/January/1970
Reference: https://docs.mongodb.com/manual/core/shell-types/#mongo-shell-date-type
Current date in milliseconds from Epoch (we are adding 3 zeros to output of seconds from Unix Time)

# echo $(($(date +"%s")*1000))
1511494354000

If currently time is 1511494354000 and there are 86,400,000 milliseconds per day, then 24 hours ago would be: 1511494354000 - 86,400,000 = 1511407954000

# mongoexport -d esa -c alert --out alerts_by_rule_time.json -u esa -p -esa --query '{ module_name : "Suspicious Login without any activity in windows hosts", "time": { $gt: new Date(1511407954000), $lt: new Date(1511494354000)} }'

Other UNIX Epoch timestamp conversations:
Converting from millisecond offset from Epoch date in UTC

# date -d @$((1511407954000 / 1000))
Thu Nov 23 03:32:34 UTC 2017

Converting a particular UTC date as millisecond offset from Epoch date

# printf '23-Nov-2017 09:00:00' | xargs -I {} date +"%s" --date={}
1511427600

So date in this case would be 1511427600000 (add 3 0's to convert from seconds to milliseconds)
 
NotesThe syntax to re-import entries into mongo using mongoimport is very similar.
To re-import alerts
1) Stop ESA service from altering DB

service rsa-esa stop

2a) Import alert collection (without first dropping current contents)

mongoimport -d esa -c alert -u esa -p esa --file /root/alerts.json --stopOnError

Otherwise:
2b) Drop alert collection and import

# mongo esa -u esa -p esa
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: esa
> db.alert.drop()
true
> exit
bye
# mongoimport -d esa -c alert -u esa -p esa --file /root/alerts.json --stopOnError

3) Restart ESA service

service rsa-esa start

Attachments

    Outcomes