000028143 - KB-1368 ASM volume filled with REDO logs; Oracle fails to start

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000028143
Applies ToAffected Versions: All Versions

System Adminstrators may see the following error in the application server logs on the appliance:



[JDBCExceptionReporter] [WARN] SQL Error: 257, SQLState: 64000


[JDBCExceptionReporter] [ERROR] ORA-00257: archiver error. Connect internal only, until freed.

org.hibernate.exception.GenericJDBCException: Cannot open connection


[SystemSettingsCustomTable] [ERROR] retrieving system settings
com.aveksa.server.db.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection



Also, in the Oracle startup.log, the following errors may be observed:


Copyright (c) 1982, 2005, Oracle. All Rights Reserved.

SQL> Connected to an idle instance.
SQL> ORACLE instance started.

Database mounted.
ORA-16038: log 2 sequence# 6525 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DG01/avdb/onlinelog/group_2.263.696692269'
ORA-00312: online log 2 thread 1: '+DG01/avdb/onlinelog/group_2.278.696692275'
SQL> Disconnected from Oracle Database 10g Release - Production



When restarting the services, you may see that the system appears to be stuck on a 'Updating System Values... " message:
[oracle~]$ sudo service dbora start
[oracle~]$ sudo service aveksa_server start

Starting Aveksa Compliance Manager Server: Waiting for DB to to finish starting...
Updating System Values...

CauseBased on these symptoms, it could be that the ASM volume is filled with old 'redo', causing this issue.

To check the ASM diskspace, run the following commands as the oracle user:

$ export ORACLE_SID=+ASM
$ asmcmd lsdg


The output will look something like:


State Type Rebal Unbal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED EXTERN N N 512 4096 1048576 33596 21828 0 21828 0 DG01/


This output will show the total space (Total_MB) and free space (Free_MB). If the disk space is > 90% full, it's recommended that you delete the old re-do logs using the ASM_Cleanup script, located in /home/oracle/database/DBA/AVDB/scripts/(ASM_Cleanup.sh)


Oracle needs to be running in order for this script to execute properly.


Once the script has been run, you must restart the Oracle service, or restart the appliance.



Also refer to KB-1302, Disabling Oracle Archiving, https://na3.salesforce.com/50150000000SAId?srPos=0&srKp=501, for information on disabling archiving (and thus stopping the creation of redo .dbf files all together). More information on this can be found in the V4.0.1 Aveksa Compliance Manager Release Notes