000027944 - KB-1479 How to check local disk space usage for the RSA VIA L&G application

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000027944
Applies ToAffected Versions: All Versions
Platform: Hardware and Software Appliance
Issue

There are times when the local file system uses up all available space and new files cannot be created.  Various errors can occur as a result. A typical one is the OS error: Linux Error: 28: No space left on device


Behavior can include the Jboss server not starting or responding, the Oracle instance not starting or responding, (this is due to log files not being able to be written to..) collections not completing, backups not completing, etc.


Note that when the Oracle ASM partition has no space available for use, this use case is only possible on a hardware appliance because a Software Appliance does not use/implement the +ASM partition.   Refer to Kbase: KB-1241 How to free up Oracle (ASM) disk space for information on how to resolve ASM space issues.


Hardware appliances have two file systems:
- Local filesystem - this is visible to the OS using commands such as df -kh.  For example, Application files, Jboss and/or Wildfly files, Oracle software, log files, temporary data collection files and database backups exist here
- +ASM partition: this is only visible at the device level, using the fdisk -l command, or using Oracle asmcmd commands  This is the Oracle ASM partition, where the physical database files are located.   Refer to KB 
Software appliances have just one local filesystem where all files, including the physical database, are located.   A Software Appliance does not use the +ASM partition.
 
Tasks1) Determine if this is a hardware or software appliance - this tells you whether or not there might be physical Oracle database files to consider.
2) Review existing file system and identify which files could be moved or deleted to free up disk space.
Resolution

How to determine space usage:


To clear the Linux Error: 28, you must first identify which files or directories are consuming the most space. This can be done using the UNIX commands: df, du and find.. (refer to the man pages for complete details on these commands).


The 'df' command can be used to see how much space is available on the devices mounted on your system. We recommend using the parameters -kh, for the output to be more easily understood. The output is similar to that shown below.  In these examples, there is not a disk space issue.  Systems that have problems would typically show 95% or higher used.


On RedHat:


$ df -kh
Filesystem                       Size   Used   Avail  Use%   Mounted on
/dev/mapper/VolGroup00-LogVol00   51G    13G    36G    27%     /
/dev/sda1                         99M    13M    82M    13%     /boot
tmpfs                             2.0G     0   2.0G     0%     /dev/shm


 


On SuSE:


~> df -kh
Filesystem                        Size   Used   Avail   Use%   Mounted on
/dev/mapper/system-root            58G    8.2G    47G    16%      /
udev                              2.0G    100K    2.0G    1%      /dev
/dev/sda1                          61M     30M     29M   51%      /boot


 


 


To be able to see the actual space used by files and directories, the UNIX command 'du' should be used. The output of this command depends on the current directory location, and if a directory parameter is used. If there are a large number of sub-directories, the redirect option can be used to redirect the output to a file, and then the file reviewed for details. The examples shown below are meant as guidelines for the use of this command:


$ cd /tmp
$ du -h

8.0K ./.ICE-unix
160K ./aveksa/postinstall
168K ./aveksa
12K ./.font-unix
12K ./.X11-unix
228K .
$ du -hs
228K .
$ du -h aveksa
160K aveksa/postinstall
168K aveksa
$ du -hs aveksa
168K aveksa
$ du -h aveksa > aveksa_size.txt
$ more aveksa_size.txt
160K aveksa/postinstall
168K aveksa


 


Another command that can be used is a combination of the find command with the -type and -size parameters. The -size option allows you to look for files over a certain size. This value can be adjust as needed as you look for files which can be candidates for removal. In this example, the '.' option is used in the find command.  The command as shown  means to search everything in the directory under the current directory for files larger then 20m. The directory location could be changed to '/', to indicate search everything under the root folder. Note that regardless of the search location,  permission error messages may occur, depending on the current user account. It may be useful to switch the root account in this situation.


 


In this example, the command used is: find . -type f -size +20000k -exec du -h {} \;


$ pwd


/tmp


$ find . -type f -size +20000k -exec du -h {} \;


72M ./aveksa/packages/jdk1.6.0_45_x64.tar.bz2
1.1G ./aveksa/packages/p10404530_112030_Linux-x86-64_2of7.zip
93M ./aveksa/packages/jboss-4.2.2.GA.zip
935M ./aveksa/packages/p10404530_112030_Linux-x86-64_3of7.zip
22M ./aveksa/packages/instantclient-basiclite-linux-x86-64-11.2.0.2.0.zip
1.3G ./aveksa/packages/p10404530_112030_Linux-x86-64_1of7.zip
find: ./aveksa/staging/database/lib: Permission denied
find: ./aveksa/staging/database/validation: Permission denied
find: ./aveksa/staging/database/reportdefs: Permission denied
find: ./aveksa/staging/database/Tests: Permission denied
find: ./aveksa/staging/database/custom: Permission denied
find: ./aveksa/staging/database/SampleData: Permission denied
find: ./aveksa/staging/database/DBA: Permission denied
25M ./aveksa/staging/test-verification-plugins.zip
52M ./aveksa/staging/AFX-2.8.1-Standard-Connectors.zip
719M ./aveksa/staging/ACM-WebSphere-6.8.1.tar
720M ./aveksa/staging/ACM-WebLogic-6.8.1.tar
718M ./aveksa/staging/aveksa.ear

 


Common locations for files eligible for removal


(reference KB-1425 - Files of Interest for locations of some of these files)


 


Note: No files should be deleted unless appropriate backups are created first. That noted, the removal of any of the files listed below will not keep the RSA Via L&G application from working.


 


This is not a complete list of files that can be removed, but is a start on what files can be looked at for consideration for removal.


 


1) Jboss and Application Server logs:


In 4.1+, the Jboss server log is rolled on a daily basis, but not purged.  Log files from previous days are candidates for removal.


In 3.6, the Tomcat catalina.out file is NOT rolled, and must be manually renamed when the server is not running.
The aveksaServer.log is 'rolled' based on filesize, not date created.  There may be an aveksaServer.log.1, .2, .3, .4 and .5.  These logs can be deleted once they are reviewed and it's determined that they do not contain relevant or necessary diagnostic information.   


 


2) Database  log and trace files:


Both the Oracle AVDB and +ASM instance (on appliances and some software installations) create .trc files by default, even when there are no problems.
These .trc files can be deleted.    An example of how to identify these .trc files using the Linux find command is shown here:


$ cd /u01
$ find . -name *.trc  -exec ls -l {} \;
 


 


Both the AVDB and +ASM alert logs are not rolled automatically.  They are appended to until they are manually deleted or renamed. The Oracle instance must be stopped to be able to manually rename and/or purge these files.  They should be reviewed for errors or pertinent diagnostic information before removal.   These files are typically found at:


/u01//app/oracle/diag/rdbms/avdb/AVDB/trace/alert_AVDB.log
/u01/app/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log


 



3) Database backup files:


The Oracle backup .dmp file (located in the /home/aveksa/AveksaExportImportDir) may not be gzipped (compressed), which would consume more space than when they are gzipped.  The number of .dmp files may accumulate if they are not periodically moved off system. The impact of several .dmp files will vary depending on how often they are created and their size.


 


4) Application installation files:
The directory /tmp/aveksa/packages can be removed, as long as these installation files are backed up, and as long as it's understood that if the application needs to be re-installed, they might need to be restored from backup.    It is not recommended to delete the /tmp/aveksa/staging directory, unless absolutely necessary and the same caveat as with the packages directory applies.

5) Potential Os system core dump files:
These files are a dump of memory and can be quite large. They should be moved off system for future analysis to determine why a core file was created.   They are typically named  core or core.pid (where pid is the process id of the process which created the core file.  They can be searched for by searching for core* from the / partition as the root user.
6) Application java profiling files:
The RSA Via L&G application has the capability to be started with java profiling enabled, for diagnostic purposes.  Java profile snapshots can then be created.  These files can be quite large and should be moved off system as soon as possible, and profiling disabled when no longer required. After moving these files, then can then be deleted.  These profiles are created with a datestamp in the name in the /home/oracle/snapshots directory, and can be found with a command similar to this::


$ ls -l /home/oracle/Snapshots/JBoss*.snapshot



 

Attachments

    Outcomes