Forensics not showing blue bars
Originally Published: 2018-10-16
Last Modified: 2023-09-22
Article Number
000041679
Applies To
RSA Product Set: Web Threat Detection
RSA Product/Service Type: Forensics
RSA Version/Condition: 6.0
Platform: null
Platform (Other): null
O/S Version: null
Product Name: null
Product Description: null
Issue
After a disk space issue, Forensics is no longer showing the blue bars. The data is on disk, but the UI is not populating. I've followed some documentation but do not see failed tasks.
Resolution
Procedure … make sure the system is processing from Silvertap to the backplex using varz grapher.
Procedure … Restore Hourly data (Blue bars) processing
  1. Determine the root cause of the outage and resolve that. (example restored or added disk space, restarted mitigator,
  2. After the resolution we want to get the system processing hourly data again.   
  3. Usually a full system component restart will get the organizer working again.
  • Restart Organizer and look at varz grapher and ps -ef for the organizer processor doing work.
  • If not, then restart the UIServer.
    1. Look for r2b2 process at the top of the hour
    2. Look in var/log/messages for the indexer running the newest hour task
Procedure … Recover data not displayed since the system outage.
Once the system functionality is restored, next we want to determine what previous data exists
  • under the circumstances that the shards were not being written we need to find the window in the data/logs/year/month/day/hour .log file directories.      
    • This is the reprocess time window.
    • Where there are no <date>.logs, this data will not be able to be reprocessed to produce hourly report data in the FUI.  
Determine if there are task files for the previous data…
  1. Look in var/opt/silvertail/data/tasks directory
The structure is ..
  1. Look in /data/tasks/failed  for the tasks within the reprocess time window.   If any are seen note it, as these will have to be reprocessed manually.
  2. Look in tasks directories for tasks that are in the reprocess time window  in
  • /data/tasks
    • /completed
    • /organizer/completed
    • /indexer/completed
  • Note while in these directories  .. Remove old task files(beyond a month of age) … clean up directories for failed tasks..  
 
  1. Note where there are no tasks seen within the reprocess time window.   These can be created manually, see below.
 
Procedure… Add tasks files to the correct directory to generate the hourly data.
 
 Procedure … Create a task file when it is missing for data that has shards available.