• Product: RSA Identity Governance & Lifecycle
• Version: All supported versions
• Platform: Hardware Appliance, Software Appliance
• Component: Local File System (non-ASM), Oracle Database (log files, trace files, backup files)
• Operating System: Linux — Red Hat Enterprise Linux, SUSE Linux Enterprise Server

Use this article to check local (non-ASM) file system disk space usage on an RSA Identity Governance & Lifecycle appliance and identify files that can safely be removed to recover disk space. RSA Support may direct you here as a proactive diagnostic step, or you may use it independently if you observe any of the following:

• The operating system reports:

Error: 28: No space left on device

• JBoss/WildFly server is not starting or not responding
• The Oracle instance is not starting or not responding (due to log files that cannot be written)
• Scheduled tasks are not completing (for example, collections, backups)

NOTE — Hardware vs. Software Appliance File System Scope:

If you are on a hardware appliance and suspect the Oracle ASM partition is running out of space, see Determining How Much Space Is Used in the Oracle Automatic Storage Management (ASM) Partition on an RSA Identity Governance & Lifecycle Hardware Appliance instead.

Before proceeding, confirm you have the following:

This article covers two sequential tasks. Use the table below to identify which task applies to your situation.

NOTE: Always complete Task 1 before Task 2 — you must identify which areas are consuming disk space before deciding which files to safely remove. Do not proceed to Task 2 without first reviewing the Task 1 output.

How to determine space usage:

To clear the Linux Error: 28, you must first identify which files or directories are consuming the most space. This can be done using the UNIX commands: df, du and find.  Refer to the man pages for complete details on these commands.

The df command can be used to see how much space is available on the devices mounted on your system. We recommend using the parameters -kh, for the output to be more easily understood. The output is similar to that shown below. In these examples, there is not a disk space issue. Systems that have problems would typically show 95% or higher used.

• On Red Hat:

$ df -kh
Filesystem                       Size   Used   Avail  Use%   Mounted on
/dev/mapper/VolGroup00-LogVol00   51G    13G    36G    27%     /
/dev/sda1                         99M    13M    82M    13%     /boot
tmpfs                             2.0G     0   2.0G     0%     /dev/shm

• On SUSE:

~> df -kh
Filesystem                        Size   Used   Avail   Use%   Mounted on
/dev/mapper/system-root            58G    8.2G    47G    16%      /
udev                              2.0G    100K    2.0G    1%      /dev
/dev/sda1                          61M     30M     29M   51%      /boot

To be able to see the actual space used by files and directories, the UNIX command du should be used. The output of this command depends on the current directory location, and if a directory parameter is used. If there are a large number of subdirectories, the redirect option can be used to redirect the output to a file, and then the file reviewed for details. The examples shown below are meant as guidelines for the use of this command:

$ cd /tmp
$ du -h
8.0K ./.ICE-unix
160K ./aveksa/postinstall
168K ./aveksa
12K ./.font-unix
12K ./.X11-unix
228K .

$ du -hs
228K .

$ du -h aveksa
160K aveksa/postinstall
168K aveksa

$ du -hs aveksa
168K aveksa

$ du -h aveksa > aveksa_size.txt

$ more aveksa_size.txt
160K aveksa/postinstall
168K aveksa

Another command that can be used is a combination of the find command with the -type and -size parameters. The -size option allows you to look for files over a certain size. This value can be adjusted as needed as you look for files which can be candidates for removal. In this example, the . option is used in the find command.  The command as shown means to search everything in the directory under the current directory for files larger than 20MB. The directory location could be changed to /, to indicate search everything under the root folder. Note that regardless of the search location, permission error messages may occur, depending on the current user account. It may be useful to switch to the 'root' account in this situation.

In this example, the command used is find . -type f -size +20000k -exec du -h {} \;

$ pwd
/tmp
$ find . -type f -size +20000k -exec du -h {} \;
72M ./aveksa/packages/jdk1.6.0_45_x64.tar.bz2
1.1G ./aveksa/packages/p10404530_112030_Linux-x86-64_2of7.zip
93M ./aveksa/packages/jboss-4.2.2.GA.zip
935M ./aveksa/packages/p10404530_112030_Linux-x86-64_3of7.zip
22M ./aveksa/packages/instantclient-basiclite-linux-x86-64-11.2.0.2.0.zip
1.3G ./aveksa/packages/p10404530_112030_Linux-x86-64_1of7.zip
find: ./aveksa/staging/database/lib: Permission denied
find: ./aveksa/staging/database/validation: Permission denied
find: ./aveksa/staging/database/reportdefs: Permission denied
find: ./aveksa/staging/database/Tests: Permission denied
find: ./aveksa/staging/database/custom: Permission denied
find: ./aveksa/staging/database/SampleData: Permission denied
find: ./aveksa/staging/database/DBA: Permission denied
25M ./aveksa/staging/test-verification-plugins.zip
52M ./aveksa/staging/AFX-2.8.1-Standard-Connectors.zip
719M ./aveksa/staging/ACM-WebSphere-6.8.1.tar
720M ./aveksa/staging/ACM-WebLogic-6.8.1.tar
718M ./aveksa/staging/aveksa.ear

Common locations for files eligible for removal:

Do not delete any files unless appropriate backups are created first.  That noted, the removal of any of the files listed below will not keep the RSA Identity Governance & Lifecycle  application from working.

This is not a complete list of files that can be removed, but is a start on what files can be looked at for consideration for removal.

1.  WildFly/JBoss and Application Server logs:

• The aveksaServer.log is rolled based on file size, not date created. There may be an aveksaServer.log.1, .2, .3, .4 and .5.  These logs may be deleted once they are reviewed and it is determined that they do not contain relevant or necessary diagnostic information.   

2.  Database  log and trace files:

• Both the Oracle AVDB and +ASM instance (on appliances and some software installations) create .trc files by default, even when there are no problems. These .trc files may be deleted. An example of how to identify these .trc files using a Linux command is shown here:

$ cd /u01
$ find . -name *.trc  -exec ls -l {} \;

• Both the AVDB and +ASM alert logs are not rolled automatically. They are appended to until they are manually deleted or renamed. The Oracle instance must be stopped to be able to manually rename and/or purge these files.  They should be reviewed for errors or pertinent diagnostic information before removal. These files are typically found at:
  • /u01//app/oracle/diag/rdbms/avdb/AVDB/trace/alert_AVDB.log
  • /u01/app/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log

3.  Database backup files:

The Oracle backup .dmp files (created when you export (backup) the database) may not be (located in the /home/aveksa/AveksaExportImportDir directory) may not be gzipped (compressed), which would consume more space than when they are gzipped. The number of .dmp files may accumulate if they are not periodically moved off the system. The impact of several .dmp files will vary depending on how often they are created and their size.

4.  Application installation files:

The directory /tmp/aveksa/packages can be removed, as long as these installation files are backed up, and as long as it is understood that if the application needs to be re-installed, they might need to be restored from a backup or downloaded again from RSALink.  It is not recommended to delete the /tmp/aveksa/staging directory, unless absolutely necessary and the same caveat as with the packages directory applies.

5.  Potential Operating System system core dump files:

These files are a dump of memory and can be quite large. They should be moved off the system for future analysis to determine why a core file was created. They are typically named core or core.pid (where pid is the process ID of the process which created the core file).  They can be found by searching for core* from the / partition as the root user.

6.  Application java profiling files:

The RSA Identity Governance and Lifecycle application has the capability to be started with Java profiling enabled, for diagnostic purposes. Java profile snapshots can then be created. These files can be quite large and should be moved off the system as soon as possible, and profiling disabled when no longer required. After moving these files, they can then be deleted.  These profiles are created with a datestamp in the name in the /home/oracle/snapshots directory, and can be found with a command similar to this:

$ ls -l /home/oracle/Snapshots/JBoss*.snapshot

7.  Archived redo logs:

Although we do not support the use of RMAN, archiving of redo log files might be enabled, and if there is no remote archive location set up, the files may end up in this default location with a similar name:

/u01/app/oracle/product/12.1.0/db_1/dbs/arch1_499_918293958.dbf

In the database, check for the following:

sqlplus / as sysdba
SQL> ARCHIVE LOG LIST

Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u01/app/oracle/product/12.1.0/db_1/dbs
Oldest online log sequence     11160
Next log sequence to archive   11163
Current log sequence           11163

Check if archiving is enabled, and if the files go to the local file system.