Updated to reflect that this is for ACM 3.6 and 4.x using Oracle 10g.. Oracle 11g uses different processes.
This process should only be used to recover from a catastrophic database failure and only when Aveksa support has validated the problem.
This process will destroy all data in the target database.
There are several things that need to be in place for the database to come up properly:
Any one of these things can be going wrong. This page will describe how to check each one and to ensure that it is working.
Oracle CSSD is Oracle's clustering services. If this process is not running, the system will not work correctly.
Do this section as root.
Check if it is running properly with the following:
[root@vm-sandbox-dzehme-01 ~]# ps -ef|grep cssd
If you don't see ocssd.bin running, you need to make that happen first.
If it is not running, first check /etc/init.d/init.cssd. We have seen cases where this gets truncated to 0 bytes:
[root@vm-sandbox~]# ls -l /etc/init.d/init.cssd
If this file is 0 bytes, get the file from another machine.
Make sure the start up link is there:
[root@vm-sandbox~]# ls -l /etc/rc3.d/*cssd*
If this link is missing, recreate it:
[root@vm-sandbox~]# ln -l /etc/init.d/init.cssd /etc/rc3.d/S96init.cssd
Make sure that cssd is started by inittab:
It must appear before the rc 3 line and not appear at the end of the file. If it is missing or in the wrong place, add or move it to the line before rc 3.
If CSSD is not running, it is best to reboot the machine after correcting these items. Make sure the Aveksa does NOT start on reboot with:
chkconfig --levels 345 aveksa_server off
The listener is how some things are able to talk to the database.
Do this section as oracle.
Check the status:
[oracle@vm-sandbox]$ lsnrctl status
In particular, make sure (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=vm-sandbox.aveksa.local)(PORT=1555))) appear with your hostname. This is the main listener port that things need to communicate on.
If this does not look right, first check listener.ora
[oracle@vm-sandbox]$ cd $ORACLE_HOME/network/admin
We have also seen this file truncated to 0 bytes, so if this file is not correct, restore it with the contents here (adjusting the host name).
The listener can be stopped with:
[oracle@vm-sandbox]$ lsnrctl stop
The listener can be started with:
[oracle@vm-sandbox]$ lsnrctl start
You may find after starting, that the listener will not list services for some time (seen this take 1-2 minutes) before it will show database instances. If you are having ASM or AVDB problems, you probably will not see the database instances, but you need the listener working before continuing.
First see if you really need to rebuild by running the two tests below:
From the “active” node – make sure the database Oracle processes are running Then run this command
A. See if the Oracle volume is present by running:
sudo service oracleasm listdisks
It should return “VOL1”
B. If you run the command sequence:
$ export ORACLE_SID=+ASM
Does it return DG01/?
Typically neither one of the above works - which means the DB needs to be recreated If it returns the correct information - contact Aveksa Support for additional troubleshooting.
This section will cause data loss! Unless you have a dump to recover from, do not proceed. It is at this point Oracle support should be consulted for further support.
1. Make sure Oracle CSSD is running (see above)
2. Make sure Oracle Listener is running
3. Find the oracle partition
the partition for Oracle should be the largest
4. Clean/format the partition:
dd if=/dev/zero of=/dev/<partition> bs=8192 count=12800
5. Start Oracle (if not auto-started by the reboot)
sudo /etc/init.d/dbora start
6. Create the Oracle Volume:
sudo service oracleasm createdisk VOL1 /dev/<partition>
A. (Pre 3.6) If not done get a release distribution and untar it then deploy the upgrade tools
B. DO THE BELOW ONLY AS THE ORACLE USER!!!
C. In certain situations you may have to perform a step several times Try rebooting and then when the DB comes up perform the failed step
7. Run the script to create +ASM partition by running (pre 3.6):
Note: you may see an error stating that DG01 can not be deleted. This is OK if the test above shows that there was no DG01
8. Run the script to create the AVDB database by running (pre 3.6):
9. Create the Aveksa schema in the database (pre 3.6):
10. Load backup if needed
11. Ensure the /etc/oratab is set to start the AVDB and +ASM instance. It should look like this:
The attached document has the details of this process