000011578 - Applying patch 2.7.1.5 on secondary RKM Appliance hangs with message 'Waiting for database to open in read write mode before applying post patch process...'

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000011578
Applies ToRSA Key Manager Appliance 2.7.1.5
IssueApplying patch 2.7.1.5 on secondary RKM Appliance hangs with message "Waiting for database to open in read write mode before applying post patch process..."
After successfully applying RKM Appliance patch 2.7.1.5 on the primary appliance, the patch failed to completely install on the secondary appliance.  The last part of the script output logged on the secondary appliance shows the following messages (with the last two lines below looping infinitely):

<...snip...>
OPatch completed with warnings.
~
Applied the patch CPU p11725015_10204_Linux-x86.zip
/opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5 /opt/hotfix/hotfix-2.7.1.5
/opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5/oracle /opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5 /opt/hotfix/hotfix-2.7.1.5
/opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5/oracle /opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5/oracle /opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5 /opt/hotfix/hotfix-2.7.1.5
/opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5/oracle /opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5 /opt/hotfix/hotfix-2.7.1.5
/opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5 /opt/hotfix/hotfix-2.7.1.5
Starting up the database

SQL*Plus: Release 10.2.0.4.0 - Production on Wed Aug 31 14:42:46 2011

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

ORACLE instance started.

Total System Global Area 1577058304 bytes
Fixed Size      1267716 bytes
Variable Size    570427388 bytes
Database Buffers   989855744 bytes
Redo Buffers     15507456 bytes
Database mounted.
ORA-16649: database will open after Data Guard broker has evaluated Fast-Start
Failover status


Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining Scoring Engine
and Real Application Testing options
Executed the SQL /opt/hotfix/hotfix-2.7.1.5/hotfix-2.7.1.5/lib/startup_database.sql
Sleep for 30 seconds ...
Waiting for database to open in read write mode before applying post patch process...
Waiting for database to open in read write mode before applying post patch process...
Waiting for database to open in read write mode before applying post patch process...
Waiting for database to open in read write mode before applying post patch process...
Waiting for database to open in read write mode before applying post patch process...


LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 31-AUG-2011 14:43:53

Copyright (c) 1991, 2007, Oracle.  All rights reserved.

Starting /opt/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /opt/oracle/product/10.2.0/db_1/network/admin/listener.ora
Log messages written to /opt/oracle/product/10.2.0/db_1/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rkmapp-2.test.rsa.net)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date                31-AUG-2011 14:43:54
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listener Log File         /opt/oracle/product/10.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rkmapp-2.test.rsa.net)(PORT=1521)))
Services Summary...
Service "ABCDRKMS" has 1 instance(s).
  Instance "ABCDRKM", status UNKNOWN, has 1 handler(s) for this service...
Service "ABCDRKMS_dgb" has 1 instance(s).
  Instance "ABCDRKM", status UNKNOWN, has 1 handler(s) for this service...
Service "ABCDRKMS_dgmgrl" has 1 instance(s).
  Instance "ABCDRKM", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
Waiting for database to open in read write mode before applying post patch process...
Sleep for 30 seconds ...
Waiting for database to open in read write mode before applying post patch process...
Sleep for 30 seconds ...
Waiting for database to open in read write mode before applying post patch process...
Sleep for 30 seconds ...


Oracle's Observer process not running on either of the appliances.  The command "ps -ef | grep observer" does not show observer/dgmgrl process running on either appliance; when Observer process is running, this commond would show a process "dgmgrl -silent start observer".

The observer cronjob is running (based on output of "crontab -l") and cron job logs (under /opt/rsa/cluster/logs) show that observer is being started, but there are no errors in observer cron logs or drc logs.

DataGuard status on /status/db pages for both appliances in the cluster show the following:

Current status for "abcdrkm": Warning: ORA-16607: one or more databases have failed
Current status for "abcdrkmp": Error: ORA-16820: Fast-Start Failover observer is no longer observing this database
Current status for "abcdrkms": Error: ORA-16820: Fast-Start Failover observer is no longer observing this database
/status/db pages shows that the primary appliance rkmapp-1 database has PHYSICAL STANDBY role with open-mode as MOUNTED and rkmapp-2 database has PRIMARY role and open_mode is READ-WRITE.  This is a normal state of database roles when applying the patch.
Lockbox protected file /opt/rsa/setup/sh/System.properties can not be opened in system mode on the primary appliance rkmapp-1, SSV threshold exceeded, unable to open lockbox file
CauseAfter successfully patching the primary appliance, the 2.7.1.5 installation script switches the database role so that the database on secondary appliance takes Primary role and the database on primary appliance takes Physical Standby role. As part of the process, Oracle DataGuard's Observer process should startup on the primary appliance (where db is currently running in Physical Standby role).

In this particular scenario, Observer process did not startup on the primary appliance which resulted in an error condition causing post patch process to fail on the secondary appliance.  The Observer process did not start on the primary appliance due to a lockbox protected file System.properties being in invalid state (see Fix statement for more details).
ResolutionThe Observer process did not startup on the primary appliance where database role was switched to PHYSICAL STANDBY, due to lockbox protected file /opt/rsa/setup/sh/System.properties being in invalid state.  Update SSV's on the lockbox file System.properties to correct the problem, then update 2.7.1.5 patch installation script to resume applying of the patch where it hung.

If you are running into a similar situation, contact RSA Customer Support for assistance.  RSA Customer Support will provide necessary steps to
complete the installation of 2.7.1.5 hotfix on the secondary RKM Appliance; these steps may include the following:

A) Confirm that System.properties can be opened in system mode on secondary appliance:

[root@rkmapp-1 sh]# /usr/lib/clb -l /opt/rsa/setup/sh/System.properties -r ORA_PASSWORD

If System.properties can be opened in system mode, you should see the following output (the password shown below is just an example):

[root@rkmapp-1 sh]# /usr/lib/clb -l /opt/rsa/setup/sh/System.properties -r ORA_PASSWORD
Lockbox file: /opt/rsa/setup/sh/System.properties opened.
This lockbox is running in System mode
Retrieved value "Mypasswd01" for name "ORA_PASSWORD".
Done!

This has worked correctly. On a system where the hardware has been chnaged you will get an error saying the fingerprint has changed. Modify using the "Security Admin Pass Phrase" (in this example it is Abc12301.):

[root@rkmapp-1 sh]# /usr/lib/clb -l /opt/rsa/setup/sh/System.properties -p Abc12301.


B) Carefully update 2.7.1.5 patch scripts exactly as guided by RSA Customer Support.

C) Re-run the install script as documented in the 2.7.1.5 release notes to re-apply 2.7.1.5 hotfix on the secondary appliance.  The scripts updated in step B above should result in carrying out only the remaining tasks in applying the hotfix.
WorkaroundApplied RKM Appliance patch 2.71.5 on a dual-node single cluster RKM Appliance 2.7.1.4
NotesSee solution "Things to do before upgrading RKM Appliances to 2.7 SP1" for other known issues and steps that can be taken to avoid problems in applying 2.7.1.x patch.
Legacy Article IDa56058

Attachments

    Outcomes