|Applies To||RSA Product Set: Web Threat Detection|
RSA Product/Service Type: Forensics
RSA Version/Condition: 5.1 - 6.2
|Issue||A known or unknown issue has caused the blue bars in the Forensics User Interface (FUI) to disappear, how can I determine what is the best way to resolve this. |
Example Customer issue: When our WTD server that hosts r2b2 hung and was restarted several hours later, all mitigator and all other services were working as expected. We continued to receive mitigator alerts/output during that time, so we are sure that the application was processing data, however, because the PRC server was hung, it doesn't look like those hourly files were processed, so I can't access the data in the UI. Can Customer Support help me to determine a way to rerun those hours, or confirm if the data is simply lost.
I had notes in the past instructing us to move files from /var/opt/silvertail/data/tasks/r2b2/failed to /var/opt/silvertail/data/tasks/organizer, however, I don't see the files in question in the ./failed directory. We couldn't find those files in the ./completed directory, either. Looking for next steps in trying to restore the data to the UI, if possible.
|Tasks||Please read the information in the Notes section for the background on how the directories and dataflow occurs within the /var/opt/silvertail/data directory.|
|Resolution||If something happened with the system data flow from Organizer -> Indexer -> Report builder (which create logs, indices and reports) then we do the following:|
(establish a time window of missing data. in UT)
A. Organizer creates the log files at the top of the hour and completes at the top of the hour, thereby committing the file. If that gets interrupted, one could find incomplete or nearly empty files, then these would not be able to be recovered either. In some situations wherein the reports were not processed (not showing up in FUI), but the logs directory are seen log files and it has 256 shards. If they are a good size, approximately 200Mb each, but no task files are seen, then it is possible to manually create task files. I.e., If shards in var/opt/silvertail/data/logs are present then the data is present in the system but is just not displaying in the Forensics UI.
find /var/opt/silvertail/data/tasks/indexer/failed -name *.task | sort -n (look in each of indexer, r2b2)
Or do this
find /var/opt/silvertail/data/tasks/*/failed -name *.task | sort -n
We were also looking with this command better suited to searching in the logs files
du -h /var/opt/silvertail/data/logs/2017 --max-depth=1
B. Also, the messages file should be reviewed for the cause of the issue for these components. Syslog is the guide to what was failing and which directory one should move it.
In /data there are the following directories, highlighted are the directories used to store data:.
[root@wtd data]# ll
drwxr-xr-x 2 rsawtd rsawtd 4096 Oct 20 14:00 alerts
drwxr-xr-x 2 rsawtd rsawtd 4096 Sep 21 20:26 audit
-rw-r--r-- 1 rsawtd rsawtd 603983 May 26 15:54 autotune.conf
drwxr-xr-x 5 rsawtd rsawtd 4096 Jan 8 2016 cassandra
drwxr-xr-x 2 rsawtd rsawtd 4096 Jan 8 2016 edsserver
drwxr-xr-x 2 rsawtd rsawtd 4096 Oct 20 14:19 guiduser
drwxr-xr-x 4 rsawtd rsawtd 4096 Jan 1 2017 logs >>> In logs are all the shard files
drwxr-xr-x 2 rsawtd rsawtd 4096 Sep 20 22:10 mitregisters
drwxr-xr-x 4 rsawtd rsawtd 4096 Jan 1 2017 reports >> the end of the process- multiple files for FUI display of data
drwxr-xr-x 2 rsawtd rsawtd 4096 Jun 25 2016 snapshot
drwxr-xr-x 6 rsawtd rsawtd 4096 Jan 8 2016 tasks >>task files control the hourly processing of data
Organizer also goes to Back Plex starts log files at the start of the hour and commits at the top of the hour. and creates logs that are the shards sorted by IP and also tasks (which are just empty files that are placeholders for data processing steps)
The order of processing the data as represented by the task files
ORGANIZER >> INDEXER >> R2B2(Report Builder)
tasks are seen in these directories
drwxr-xr-x 2 rsawtd rsawtd 380928 Oct 20 14:00 completed >> completed tasks...
drwxr-xr-x 5 rsawtd rsawtd 4096 Jan 8 2016 indexer/completed >> r2b2 polls for any files entering here and processes them to create reports that show up in the FUI
drwxr-xr-x 3 rsawtd rsawtd 4096 Jan 8 2016 organizer/completed >> Indexer polls for any files entering here and processes them to present to R2B2 (Report builder)
drwxr-xr-x 4 rsawtd rsawtd 4096 Jan 8 2016 r2b2 /failed and in_process
indexer puts them it's own completed folder, from where r2b2 picks it up and generates the reports/blue bars
an example - indexer looks at the organizer completed folder to pick up new tasks
r2b2 does the same, looks at indexer complete folder for new tasks
In Reports directory are the processed data used for populating the FUI more than ten types of data are seen as .json and .gz files for rules, scores, json files to populate the FUI.
If there is an issue that prevented FUI from displaying 'blue bars' for a period of time, it is because there was something that prevented the Reports files from being created.