Unable to switchover database roles back to the default after a failover (Error: ORA-16608: one or more databases have warnings)
3 years ago
Originally Published: 2010-06-14
Article Number
000045932
Applies To
RSA Key Manager Appliance 2.5.0.3
Issue
Unable to switchover database roles back to the default after a failover (Error: ORA-16608: one or more databases have warnings)
Unable to switchover database roles back to the default after a failover (Error: ORA-16608: one or more databases have warnings)
An attempt (through rkmawa => Operations => Cluster => Make Primary button) to switchover roles back to the default configuration failed with the following error:

Cluster Settings
Message:
Make primary performed on cluster XXXX; output:
Switching over to xxxxrkmp
DGMGRL for Linux: Version 10.2.0.4.0 - Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected.
Performing switchover NOW, please wait...
Error: ORA-16608: one or more databases have warnings
Failed.
Unable to switchover, primary database is still "xxxxrkms"
Resolution
Take the following steps to find out the actual error condition(s) and then take necessary steps to resolve those errors before database role can be switched back to the default:

1. Check detailed status of Oracle DataGuard and Fast Start FailOver using either the status pages tool ( How to check status or health of RKM Appliance for troubleshooting purposes? ) or the following commands (shown in red while replacing xxxxrkm, xxxxrkmp, and xxxxrkms with your installation specific names):

Log in to shell prompt as root

[root@rkmapp1 ~]# su - oracle

-bash-3.00$ dgmgrl
DGMGRL for Linux: Version 10.2.0.4.0 - Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys
Password:            <== provide Oracle password
Connected.

DGMGRL> show configuration verbose
Configuration
  Name:                xxxxrkm
  Enabled:             YES
  Protection Mode:     MaxAvailability
  Fast-Start Failover: ENABLED
  Databases:
    xxxxrkmp - Primary database
    xxxxrkms - Physical standby database
           - Fast-Start Failover target
Fast-Start Failover
  Threshold:           30 seconds
  Observer:            rkmapp2.domain.com
  Shutdown Primary:    FALSE
Current status for "xxxxrkm":
SUCCESS

DGMGRL> show database xxxrkmp
Database
  Name:            xxxxrkmp
  Role:            PRIMARY
  Enabled:         YES
  Intended State:  ONLINE
  Instance(s):
    XXXXRKM
Current status for "xxxxrkmp":
SUCCESS

DGMGRL> show database xxxrkms
Database
  Name:            xxxxrkms
  Role:            PHYSICAL STANDBY
  Enabled:         YES
  Intended State:  ONLINE
  Instance(s):
    XXXXRKM
Current status for "xxxxrkms":
SUCCESS

2. Review the output of above commands/tool.  The example shown above is normal/correct output.  In case of an error or failure, specific Oracle error code will show in the output.  Depending upon the error condition(s), take corrective steps to resolve those error conditions.

3. If you see a combination of the following errors, take steps provided in solution "Oracle DataGuard and FSFO status check shows ORA-16607 ORA-12541 and ORA-16825":
    ORA-16607: one or more databases have failed
    ORA-12541: TNS:no listener
    ORA-16825: Fast-Start Failover and other errors or warnings detected for the database

4. If you see a combination of the following errors, take steps provided in solution "ORA-16766: Redo Apply unexpectedly offline":
    ORA-16607: one or more databases have failed
    ORA-16766: Redo Apply unexpectedly offline

5. Once the actual error conditions have been rectified, and Oracle DataGuard/FSFO output shows normal like shown in step #1 above, an attempt to switchover database roles back to the default should work fine.
Workaround
Both appliances in the cluster were powered up (after a planned shutdown for maintenance) at the same time which likely resulted in database role switchover so that the original standby took primary role and the original primary took standby role.

Related Articles

Trending Articles

Don't see what you're looking for?