000028168 - How to rebuild Oracle Automatic Storage Management (ASM) for RSA Identity Governance & Lifecycle

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support on Jan 8, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000028168
Applies ToRSA Product Set: Identity Governance & Lifecycle
Resolution

This article has been updated to reflect that it is for ACM 3.6 and 4.x using Oracle 10g.  Oracle 11g uses different processes.



This process should only be used to recover from a catastrophic database failure and only when Aveksa support has validated the problem.

This process will destroy all data in the target database.



There are several things that need to be in place for the database to come up properly:



  1. Oracle CSSD needs to be running,
  2. Oracle Listener needs to be running,
  3. Oracle ASM needs to be running,
  4. Oracle ASM diskgroup needs to be properly configured, and
  5. Oracle AVDB needs to be running

Any one of these things can fail, causing the database to not come up. This page will describe how to check each one and to ensure that it is working.




Oracle CSSD



Oracle CSSD is Oracle's clustering services. If this process is not running, the system will not work correctly.



Perform the steps in the section as root.



  1. Check to see if the Oracle CSSD is running properly with the following command:


[root@vm-sandbox-dzehme-01 ~]# ps -ef|grep cssd
root      5224  5157  0 14:11 pts/3    00:00:00 grep cssd
root     25513     1  0 Sep04 ?        00:00:00 /bin/su -l oracle -c sh -c 'cd /u01/app/oracle/product/10.2.0/db_1/log/vm-sandbox-dzehme-01/cssd;  ulimit -c unlimited; exec /u01/app/oracle/product/10.2.0/db_1/bin/ocssd '
oracle   25619 25513  0 Sep04 ?        00:00:45 /u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin


  1. If the occssd.bin is not running, first check /etc/init.d/init.cssd. We have seen cases where this gets truncated to 0 bytes:


[root@vm-sandbox~]# ls -l /etc/init.d/init.cssd
-rwxr-xr-x  1 root root 39611 Sep  4 11:06 /etc/init.d/init.cssd


If this file is 0 bytes, get the file from another machine.


  1. Make sure the start up link is there:


[root@vm-sandbox~]# ls -l /etc/rc3.d/*cssd*
lrwxrwxrwx  1 root root 21 Sep  4 11:06 /etc/rc3.d/S96init.cssd -> /etc/init.d/init.cssd


  1. If this link is missing, recreate it:


[root@vm-sandbox~]# ln -l /etc/init.d/init.cssd /etc/rc3.d/S96init.cssd


  1. Make sure that CSSD is started by inittab:


...
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
...


It must appear before the rc 3 line and not appear at the end of the file. If it is missing or in the wrong place, add or move it to the line before rc 3.



  1. If CSSD is not running, it is best to reboot the machine after correcting these items. Make sure the Aveksa server does NOT start on reboot with:


chkconfig --levels 345 aveksa_server off
chkconfig --levels 345 aveksa_agent off


  1. Then reboot:


reboot

 

Oracle Listener



The listener is how some things are able to talk to the database.



Perform the steps in the section as the oracle user.




    1. Check the status:


    [oracle@vm-sandbox]$ lsnrctl status

    LSNRCTL for Linux: Version 10.2.0.2.0 - Production on 12-SEP-2008 14:18:05

    Copyright (c) 1991, 2005, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
    STATUS of the LISTENER
    ------------------------
    Alias                     LISTENER
    Version                   TNSLSNR for Linux: Version 10.2.0.2.0 - Production
    Start Date                05-SEP-2008 11:19:11
    Uptime                    7 days 2 hr. 58 min. 53 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
    Listener Log File         /u01/app/oracle/product/10.2.0/db_1/network/log/listener.log
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
    Services Summary...
    Service "AVDB" has 1 instance(s).
      Instance "AVDB", status READY, has 1 handler(s) for this service...
    Service "AVDBXDB" has 1 instance(s).
      Instance "AVDB", status READY, has 1 handler(s) for this service...
    Service "AVDB_XPT" has 1 instance(s).
      Instance "AVDB", status READY, has 1 handler(s) for this service...
    The command completed successfully


    In particular, make sure (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=vm-sandbox.aveksa.local)(PORT=1555))) appears with your hostname. This is the main listener port that things need to communicate on.

     

    1. If this does not look right, first check listener.ora:


    [oracle@vm-sandbox]$ cd $ORACLE_HOME/network/admin
    [oracle@vm-sandbox]$ cat listener.ora
    # listener.ora Network Configuration File: /u01/app/oracle/db_1//network/admin/listener.ora
    # Generated by Oracle configuration tools.

    LISTENER =
      (DESCRIPTION_LIST =
        (DESCRIPTION =
          (ADDRESS = (PROTOCOL = TCP)(HOST = vm-sandbox-dzehme-01.aveksa.local)(PORT = 1555))
          (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
        )
      )


    We have also seen this file truncated to 0 bytes, so if this file is not correct, restore it with the contents here (adjusting the host name).


    1. The listener can be stopped with: the following command:


    [oracle@vm-sandbox]$ lsnrctl stop
    LSNRCTL for Linux: Version 10.2.0.2.0 - Production on 12-SEP-2008 14:23:33

    Copyright (c) 1991, 2005, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
    The command completed successfully


    1. The listener can be started with the following command:


    [oracle@vm-sandbox]$ lsnrctl start

    LSNRCTL for Linux: Version 10.2.0.2.0 - Production on 12-SEP-2008 14:28:18

    Copyright (c) 1991, 2005, Oracle.  All rights reserved.

    Starting /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

    TNSLSNR for Linux: Version 10.2.0.2.0 - Production
    System parameter file is /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
    Log messages written to /u01/app/oracle/product/10.2.0/db_1/network/log/listener.log
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
    STATUS of the LISTENER
    ------------------------
    Alias                     LISTENER
    Version                   TNSLSNR for Linux: Version 10.2.0.2.0 - Production
    Start Date                12-SEP-2008 14:28:18
    Uptime                    0 days 0 hr. 0 min. 0 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
    Listener Log File         /u01/app/oracle/product/10.2.0/db_1/network/log/listener.log
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=vm-sandbox-dzehme-01.aveksa.local)(PORT=1555)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
    The listener supports no services
    The command completed successfully


    1. You may find after starting, that the listener will not list services for some time (we have seen this take one to two minutes) before it will show database instances. If you are having ASM or AVDB problems, you probably will not see the database instances, but you need the listener working before continuing.


    Oracle ASM



    1. First see if you really need to rebuild by running the two tests shown below.  From the “active” node, make sure the database Oracle processes are running  then run this command
      1. See if the Oracle volume is present by running:


    sudo service oracleasm listdisks


    It should return VOL1


    1. If you run the command sequence:


    $ export ORACLE_SID=+ASM
    $ export ORACLE_HOME=/u01/app/12.1.0/grid
    $ asmcmd
    ASMCMD> ls


    Does it return DG01/?


    Typically neither one of the above commands work, which means the database needs to be recreated.  If it returns the correct information, contact RSA Customer Support for additional troubleshooting.



    This section will cause data loss! Unless you have a dump to recover from, do not proceed. It is at this point Oracle support should be consulted for further troubleshooting.



    1. Make sure the Oracle CSSD is running (see steps above).
    2. Make sure the Oracle Listener is running.
    3. Find the Oracle partition.  The partition for Oracle should be the largest:


    fdisk -l


    1. Clean/format the partition:


    dd if=/dev/zero of=/dev/<partition> bs=8192 count=12800


    1. Start Oracle (if it did not auto-start from the reboot):


    sudo /etc/init.d/dbora start


    1. Create the Oracle volume:


    sudo service oracleasm createdisk VOL1 /dev/<partition>


    For example:




    sudo service oracleasm createdisk VOL1 /dev/sda3


    Pre 3.6



    1. If not done, get a release distribution and untar it, then deploy the upgrade tools.

    DO THE STEPS BELOW ONLY AS THE ORACLE USER!



    1. In certain situations you may have to perform a step several times Try rebooting and then when the DB comes up perform the failed step.

     


    1. Run the script to create +ASM partition by running the following commands:

    Pre 3.6





    cd /home/oracle/deploy/create_asm
    export ORACLE_SID=+ASM
    ./Create_ASM_Instance.sh




    Post 3.6




    cd /home/oracle/deploy/create_asm
    export ORACLE_SID=+ASM
    ./Create_ASM_Instance.sh

     

    Note: you may see an error stating that DG01 can not be deleted. This is OK if the test above shows that there was no DG01 



    1. Run the script to create the AVDB database by running one set of the following commands:

    Pre 3.6



    cd /tmp/postinstall/create_avdb
    export ORACLE_SID=AVDB
    ./Create_AVDB_Instance.sh


    Post 3.6



    cd /home/oracle/deploy/create_avdb
    export ORACLE_SID=AVDB
    ./Create_AVDB_Instance.sh

     

    1. Create the Aveksa schema in the database

    Pre 3.6



    cd /home/oracle/database
    ./createSchema_V3.5.sh


    Post 3.6




    cd /home/oracle/database
    ./createSchema.sh

     

    1. Load backup if needed
    2. Ensure the /etc/oratab is set to start the AVDB and +ASM instance. It should look like this:


    +ASM:/u01/app/oracle/product/10.2.0/db_1:Y
    AVDB:/u01/app/oracle/product/10.2.0/db_1:Y
    Notes 
     

    Attachments

      Outcomes