000017829 - Oracle archivelog files (and other) causing 100% disk space usage on an RSA Data Protection Manager (Key Manager) appliance

Document created by RSA Customer Support Employee on Jun 15, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000017829
Applies ToRSA Product Set: DPM
RSA Product/Service Type: Key Manager Appliance
RSA Version/Condition: 1.5.x, 1.6.x, 2.5.x, 2.7.x, 3.1.2, 3.2.x, 3.5.x, 3.5.2.x
IssueDisk space usage in /opt on the appliance is getting high
100% disk usage in /opt

Error 503!  Service temporarily unavailable. The Server is temporarily unable to service your request. 
Please try again later.

 
CauseMost of disk space is taken up by Oracle archive logs.
ResolutionAll the steps mentioned in this article needs to be run as root.

      1. Identify the source of the high disk usage


Run the script /opt/rsa/cs/tomcat/find_disk_usage.sh which will show disk usage in known problematic folders as well as large files.

  • If "archdest" is full:

Identify where disk space is used. If the "archdest" folder is full, Oracle will fail to start. Move some of the newer archivelogs in a different folder, cross-check the archivelogs using RMAN, and restart Oracle.


Example, to find files created in the last 24 hours and move them out:


mkdir -p /opt/oracle/oradata/datafiles/tmp
find /opt/oracle/oradata/ONERKM/archdest -name "*.dbf" -ctime -1 -exec mv {} /opt/oracle/oradata/datafiles/tmp \;
su - oracle
rman target /
crosscheck archivelog all;
exit
exit
service oracle restart

 

Once you have resolved the disk space issues, do not forget to move those files back and re-catalog them:


mv /opt/oracle/oradata/datafiles/tmp/*.dbf /opt/oracle/oradata/ONERKM/archdest
/opt/rsa/setup/sh/cs_catalog_archivelogs.sh
su - oracle
rman target /
crosscheck archivelog all;
exit
exit

 
  • If any other partition is full

Proceed by elimination to clean up disk space. A well know issue is the Oracle "adump" folder and Oracle trace files that you can remove using those commands


find /opt/oracle/admin/ONERKM/adump -name "*.aud" -exec rm -f {} \; &
find /opt/oracle/diag/rdbms/onerkmp/ONERKM/trace/ -name "*.trc" -exec rm -f {} \; &

 

Apache's log files in /var/log/httpd are safe to delete. Stop Apache first (service httpd stop), delete the log files and restart Apache (service httpd start).
If Oracle's listener.log file is huge, you can safely delete this file by stopping Oracle first (service oracle stop), deleting the file and restarting Oracle (service oracle start). You will need to restart Access Manager (service ctrust restart) and Tomcat (service tomcat restart)
 


      2. Cleanup Oracle archivelogs


In most cases Oracle archivelog files should never be deleted using "rm -f". Follow those commands instead (see tips below as well):


su - oracle
rman target /
RMAN> catalog start with '/opt/oracle/oradata/ONERKM/archdest';
( Run this command above as long as RMAN is able to find / catalog new files )
RMAN> crosscheck archivelog all;
RMAN> delete archivelog all completed before 'sysdate-60';
(Type YES when prompted with the below message)
Do you really want to delete the above objects (enter YES or NO)?
RMAN> exit

 

Running the "delete archivelog" command may give a warning such as "archivelog belongs to a different database incarnation". If you get this warning those archivelogs can safely be deleted using "rm -f".
When re-cataloging find new files you need to restart the capture process. You can use the script cs_stop_capture.sh and cs_start_capture.sh.
In order for Oracle to mark an archivelog as being "deletable" it needs to have been applied to all remote nodes AND not be required by the capture process. Meaning if you see the First SCN of a capture being really low compared to the Current SCN of the database, or Captured SCN of the capture, it may take few hours for Oracle to re-mine all archivelogs and have the DPM Appliance internal Oracle Scheduler Job to move up the first SCN. Let it settle and do not rush to delete files.


NOTE:  Ensure that backup is configured on each appliance and backups are taken on regular basis.
 


Consider upgrading to the latest patch of DPM Appliance 3.x (3.2.4.4.2 and 3.5.2.4.6 as of writing this note).  These latest version of DPM Appliance include built-in mechanism to automatically cleanup old / un-needed archive logs on regular basis.  Contact RSA Customer Support to obtain additional documentation, migration utilities, and/or hotfixes to upgrade your appliance(s).

Notes

Tips


  1. Instead of manually running RMAN's "catalog" command you can use the script "cs_catalog_archivelogs.sh"
  2. When listing archivelogs, the filename tells you about the database incarnation (blue vs red, two different incarnation number). Doing a "ls -lrt  /opt/oracle/oradata/$ORACLE_SID/archdest/" will show you filenames sorted properly. The filename of an archivelog file is "X_SEQUENCE_INCARNATION.dbf". The newest (most recently created) archivelog filename will show you the current database incarnation number. 
-rw-r----- 1 oracle oinstall 46654976 Jun 1 21:00 1_1950_741142621.dbf
-rw-r----- 1 oracle oinstall 46665216 Jun 1 22:26 1_1951_741142621.dbf
-rw-r----- 1 oracle oinstall 46661632 Jun 2 04:15 1_1952_741142621.dbf
-rw-r----- 1 oracle oinstall 46661632 Jun 2 08:06 1_1953_741142621.dbf
-rw-r----- 1 oracle oinstall 46661632 Jun 2 22:08 1_1954_741142621.dbf
-rw-r----- 1 oracle oinstall 7754240  Jun 3 01:55 1_1_784950778.dbf
-rw-r----- 1 oracle oinstall 39315968 Jun 3 01:55 1_1955_741142621.dbf
-rw-r----- 1 oracle oinstall 17920    Jun 3 01:56 1_2_784950778.dbf
-rw-r----- 1 oracle oinstall 11589632 Jun 3 03:40 1_3_784950778.dbf
-rw-r----- 1 oracle oinstall 440832   Jun 3 03:41 1_1_784957208.dbf
-rw-r----- 1 oracle oinstall 4096     Jun 3 03:41 1_2_784957208.dbf
-rw-r----- 1 oracle oinstall 16896    Jun 3 03:41 1_3_784957208.dbf
-rw-r----- 1 oracle oinstall 2048     Jun 3 03:41 1_5_784957208.dbf
Legacy Article IDa51732

Attachments

    Outcomes