000027868 - KB-1169 -ACM / database startup fails after reboot

Document created by RSA Customer Support Employee on Jun 15, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000027868
Applies ToRSA Identity Management & Governance
All Versions
UI database startup failure GUI
IssueOccasionally after an upgrade, installation or even sometimes a system reboot,  the system fails to start properly, here is the scenario: 



During the initialization and boot-up process the cssd  (Oracle Cluster Synchronization Services daemon) fails to start. takes too long to start or starts, fails and attempts a restart. This failure causes the Oracle database to hang. During this time init correctly starts the aveksa_server. The aveksa_server does not validate the database status and begins its initialization and attempts to connect to the database.  Unfortunately, the database is not up because cssd has not started and the aveksa_server commands wait. Since there are database commands pending the cssd is unable to start and consequently Oracle doesn't start. So we are stuck in a loop. 

After a reboot the UI is not available.
KB-1169 -ACM / database startup fails after reboot
Resolution

1. Verify that you are experiencing this problem 


 


 A. Run "ps-ef" and check for the aveksa_server and possibly the agent are in a startup state, the database is not running and the init.cssd says either ?startup? or ?run? as shown below:


 


root   2999     1  0 16:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd run


root   3000     1  0 16:56 ? 00:00:00 /bin/bash /etc/rc.d/rc 5


root   9650  3000  0 17:08 ? 00:00:00 initlog -q -c /etc/rc5.d/S98aveksa_server


root   9651  9650  0 17:08 ? 00:00:00 /bin/bash /etc/rc5.d/S98aveksa_server


root  14404  9651  0 17:12 ? 00:00:00 /bin/bash /usr/bin/addMACAddress


root  14410 14404  0 17:12 ? 00:00:00 su - oracle -c sqlplus avuser/secret


oracle 14411 14410  0 17:12 ?  00:00:00 sqlplus    @/tmp/.....


 


B. If this is the case proceed to step #2  Otherwise check  KB-1438 and KB-1414


 


2. The workaround for this bug is to manually kill (as root) the processes listed below, use the kill -9 <pid> command for the following processes:


 


  •  initlog -q -c /etc/rc5.d/S98aveksa_server
  • /bin/bash /etc/rc5.d/S98aveksa_server
  • /bin/bash /usr/bin/addMACAddress
  •  Any sqlplus processes

3. The aveksa_server (and aveksa_agent if running 3.X) processes will need to be stopped twice.


 


4. This should allow the init.cssd process to complete the startup routine and then the database will start. You can tell the init.cssd process has started by running a ps-ef and looking for this line:


 


ps -ef | grep css


oracle 5245 1 0 Jan11 00:00:21/u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin


 


5. If you don't see the above process in 5 minutes do the following as root:


 


  1. Run the command /etc/init.cssd stop
  2. Do a ps -ef and then a kill -9 on any init.cssd process
  3. Run the command  /etc/init.d/init.cssd start
  4. Monitor for the output in  #4
  5. Once the cssd starts continue ? otherwise contact Aveksa
6. Start Oracle (as root) using:
 
service dbora start

OR


/etc/init.d/dbora start


 


7. Start the Aveksa services


 


service aveksa_server start


service aveksa_agent start (version 3.X only)


 


ACM / database startup fails after reboot

 
Legacy Article IDa66414

Attachments

    Outcomes