000031828 - Reporting engine is down due to 100% usage of /home/rsasoc in RSA Security Analytics 10.4 and higher

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000031828
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Reporting Engine
RSA Version/Condition: 10.4.1 and higher

 
IssueThe reporting engine is failing to run additional reports.  When looking at the reporting engine in the UI the following screen is displayed, with the message Add a Reporting Engine to the Reporter:
User-added image
CauseWhen the reporting engine stops in such a manner, run the following: 
tailf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log

The following error will appear;
2015-11-18 05:54:58,295 [main] ERROR com.rsa.soc.re.utils.DiskSpaceMonitorUtil - CRITICAL : Available Disk Space is critically low: 
235 MB . Shutting down reporting engine

Running df -h confirms 100% usage for /home/rsasoc directory:
User-added image
Then verify the directories where consumption is the highest in /home/rsasoc using following command that shows the formattedReports directory as consuming the excessive space:
du -xh /home/rsasoc/>filespace.txt
cat filespace.txt|grep G

User-added image
 


 
Resolution Since formattedReports directory is consuming more space, move old reports to a new location, such as /tmp/oldreports, or simply remove the old reports:
1.  SSH to the Security Analytics server using the root credentials.
2.  Stop the Reporting Engine service:
         
stop rsasoc_re
3.  Choose one of the following methods, either move or remove:
a.  To move a subset of reports to save for the month of August (
Note:  This command moves all reports from August 2015 to /tmp/oldreports.  Before running the command, ensure there is enough disk space to accommodate the moved reports):
mkdir /tmp/oldreports
cd /home/rsasoc/rsa/soc/reporting-engine/formattedReports
mv *201508* /tmp/oldreports

b.  To remove a subset of reports for the month of August 2015:
cd /home/rsasoc/rsa/soc/reporting-engine/formattedReports
rm -rf *201508*

4.  Verify /home/rsasoc has some space available using the df -h command and run start rsasoc_re.
5.  Login to the Security AnalyticsGUI and access the Reporting Engine service, which should be available once disk space is available.
IMPORTANT: To prevent this from occurring again, reduce the number of days set to retain reports.  For example, change the default value for the number of days to retain reports from 100 to a lower value, such as 30 or 50 for the parameter "Retain Report History for #Days" in the UI under Reporting Engine > Config > System Configuration.

 

Attachments

    Outcomes