|Applies To||RSA Identity Management & Governance|
UI database startup failure GUI
|Issue||Occasionally after an upgrade, installation or even sometimes a system reboot, the system fails to start properly, here is the scenario: |
During the initialization and boot-up process the cssd (Oracle Cluster Synchronization Services daemon) fails to start. takes too long to start or starts, fails and attempts a restart. This failure causes the Oracle database to hang. During this time init correctly starts the aveksa_server. The aveksa_server does not validate the database status and begins its initialization and attempts to connect to the database. Unfortunately, the database is not up because cssd has not started and the aveksa_server commands wait. Since there are database commands pending the cssd is unable to start and consequently Oracle doesn't start. So we are stuck in a loop.
After a reboot the UI is not available.
KB-1169 -ACM / database startup fails after reboot
1. Verify that you are experiencing this problem
A. Run "ps-ef" and check for the aveksa_server and possibly the agent are in a startup state, the database is not running and the init.cssd says either ?startup? or ?run? as shown below:
root 2999 1 0 16:56 ? 00:00:00 /bin/sh /etc/init.d/init.cssd run
root 3000 1 0 16:56 ? 00:00:00 /bin/bash /etc/rc.d/rc 5
root 9650 3000 0 17:08 ? 00:00:00 initlog -q -c /etc/rc5.d/S98aveksa_server
root 9651 9650 0 17:08 ? 00:00:00 /bin/bash /etc/rc5.d/S98aveksa_server
root 14404 9651 0 17:12 ? 00:00:00 /bin/bash /usr/bin/addMACAddress
root 14410 14404 0 17:12 ? 00:00:00 su - oracle -c sqlplus avuser/secret
oracle 14411 14410 0 17:12 ? 00:00:00 sqlplus @/tmp/.....
2. The workaround for this bug is to manually kill (as root) the processes listed below, use the kill -9 <pid> command for the following processes:
3. The aveksa_server (and aveksa_agent if running 3.X) processes will need to be stopped twice.
4. This should allow the init.cssd process to complete the startup routine and then the database will start. You can tell the init.cssd process has started by running a ps-ef and looking for this line:
ps -ef | grep css
oracle 5245 1 0 Jan11 00:00:21/u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin
5. If you don't see the above process in 5 minutes do the following as root:
service dbora start
7. Start the Aveksa services
service aveksa_server start
service aveksa_agent start (version 3.X only)
ACM / database startup fails after reboot
|Legacy Article ID||a66414|