RSA Identity Governance and Lifecycle 7.0.x upgrade or installation fails with [FATAL] [INS-13013] and Oracle Installer Exit Code: 253 fatal errors

[#1]

Article Number

000031687

Applies To

RSA Product Set: Identity Governance & Lifecycle 
RSA Product/Service Type: Appliance
RSA Version/Condition: 7.0.x

 

Issue

When installing RSA Identity Governance and Lifecycle 7.0.x, the installation script terminates with the errors:  
 

[FATAL] [INS-13013]

and 

Oracle Installer Exit Code: 253

The full error noted in the /tmp/aveksa-install.log is shown here:
 

[FATAL] [INS-13013] Target environment does not meet some mandatory requirements. CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /tmp/OraInstall2015-11-06_10-42-51AM/installActions2015-11-06_10-42-51AM.log ACTION: Identify the list of failed prerequisite checks from the log: /tmp/OraInstall2015-11-06_10-42-51AM/installActions2015-11-06_10-42-51AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually. A log of this session is currently saved as: /tmp/OraInstall2015-11-06_10-42-51AM/installActions2015-11-06_10-42-51AM.log. Oracle recommends that if you want to keep this log, you should move it from the temporary location. Oracle Installer Exit Code: 253 Duplicated /tmp/OraInstall2015-11-06_10-42-51AM/installActions2015-11-06_10-42-51AM.log in /tmp/aveksa/oracle.log

Cause

Subsequent review of the /tmp/aveksa/oracle.log shows this critical operation failure and potential causes. Note that in the below examples from /tmp/aveksa/oracle.log the hostname and nameserver IP addresses have been modified but the errors are real.  
 

Scenario 1:  Connection Timeout

The Oracle installation fails due to a connection timeout while trying to resolve the nameserver IP addresses noted in /etc/resolv.conf .

'<CV_CMD>/usr/bin/nslookup -querytype=A<hostname> 111.22.333.123 </CV_CMD><CV_VAL>;; connection timed out; trying next origin;; connection timed out; no servers could be reached </CV_VAL><CV_VRES>0</CV_VRES><CV_LOG>Exectask: runexe was successful</CV_LOG><CV_CMDLOG><CV_INITCMD>/tmp/CVU_12.1.0.2.0_oracle/exectask -runexe /usr/bin/nslookup -querytype=A <hostname> 111.22.333.123 </CV_INITCMD><CV_CMD>/usr/bin/nslookup -querytype=A <hostname> 111.22.333.123 </CV_CMD><CV_CMDOUT> ;; connection timed out; trying next origin</CV_CMDOUT><CV_CMDSTAT>0</CV_CMDSTAT></CV_CMDLOG><CV_ERES>0</CV_ERES>' INFO: FINE: [Task.perform:594] sTaskResolvConfIntegrity:Task resolv.conf Integrity[STASKRESOLVCONFINTEGRITY]:TASK_SUMMARY:FAILED:CRITICAL:OPERATION_FAILED INFO: ERROR: [Result.addErrorDescription:624] Check for integrity of file "/etc/resolv.conf" failed

  

Scenario 2:  nslookup Failure

The installation fails due to an nslookup issue during the Grid installation, with this error:
 

INFO: FINE: [Task.perform:493] TaskResolvConfIntegrity:Task resolv.conf Integrity[RESOLV_CONF]:TASK_START INFO: FINE: [Task.perform:514] m_nodeList='<hostname-alias>' INFO: FINE: [Task.perform:493] sTaskResolvConfIntegrity:Task resolv.conf Integrity[STASKRESOLVCONFINTEGRITY]:TASK_START INFO: FINE: [Task.perform:514] m_nodeList='<hostname-alias>' INFO: FINE: [VerificationCommand.execute:297] Output: '<CV_CMD>/usr/bin/nslookup unknown-not-reachable-node </CV_CMD><CV_VAL>Server: 111.222.33.44 Address: 111.222.33.44#53 ** server can't find unknown-not-reachable-node: NXDOMAIN

 

Scenario 3:  ntpq Time Lookup Failure

The installation fails due to an ntpq failure during the Grid installation, with this error:

INFO: FINE: [Task.perform:493]   TaskNTP:Network Time Protocol (NTP)[TASKNTP]:TASK_START INFO: FINE: [Task.perform:514]   m_nodeList='acm-702' INFO: FINE: [VerificationCommand.execute:297]   Output: '<CV_VRES>0</CV_VRES><CV_LOG>Exectask: file check successful</CV_LOG><CV_CMDLOG><CV_INITCMD>/tmp/CVU_12.1.0.2.0_oracle/exectask -chkfile /etc/ntp.conf </CV_INITCMD><CV_CMD>access() /etc/ntp.conf F_OK</CV_CMD><CV_CMDOUT></CV_CMDOUT><CV_CMDSTAT>2</CV_CMDSTAT></CV_CMDLOG><CV_ERES>0</CV_ERES>' INFO: FINE: [VerificationCommand.execute:297]   Output: '<CV_CMD>/bin/grep "^ntp" /etc/services </CV_CMD><CV_VAL>ntp                123/tcp      # Network Time Protocol  [Dave_Mills] [RFC5905] ntp                123/udp      # Network Time Protocol  [Dave_Mills] [RFC5905] </CV_VAL><CV_VRES>0</CV_VRES><CV_LOG>Exectask: RunGenCmd successful</CV_LOG><CV_CMDLOG><CV_INITCMD>/tmp/CVU_12.1.0.2.0_oracle/exectask -rungencmd /bin/grep "^ntp" /etc/services </CV_INITCMD><CV_CMD>/bin/grep "^ntp" /etc/services </CV_CMD><CV_CMDOUT> ntp                123/tcp      # Network Time Protocol  [Dave_Mills] [RFC5905]</CV_CMDOUT><CV_CMDSTAT>0</CV_CMDSTAT></CV_CMDLOG><CV_ERES>0</CV_ERES>' INFO: FINE: [VerificationCommand.execute:297]   Output: '<CV_CMD>/usr/sbin/ntpq -p </CV_CMD><CV_VAL>     remote           refid      st t when poll reach   delay   offset  jitter ============================================================================== *mickey.lss.emc. 10.254.225.252   2 u   39   64   37    4.662  334643. 106.368 </CV_VAL><CV_VRES>0</CV_VRES><CV_LOG>Exectask: RunGenCmd successful</CV_LOG><CV_CMDLOG><CV_INITCMD>/tmp/CVU_12.1.0.2.0_oracle/exectask -rungencmd /usr/sbin/ntpq -p </CV_INITCMD><CV_CMD>/usr/sbin/ntpq -p </CV_CMD><CV_CMDOUT>      remote           refid      st t when poll reach   delay   offset  jitter</CV_CMDOUT><CV_CMDSTAT>0</CV_CMDSTAT></CV_CMDLOG><CV_ERES>0</CV_ERES>' INFO: ERROR: [ResultSet.addErrorDescription:1078]  PRVG-1015 : Time server "10.254.225.252" has time offsets that are not within permissible limit "1000.0" on nodes "acm-702".  INFO: ERROR: [Result.addErrorDescription:624]  PRVF-5413 : Node "acm-702" has a time offset of 334643.0 that is beyond permissible limit of 1000.0 from NTP Time Server "10.254.225.252"  INFO: ERROR: [ResultSet.addErrorDescription:1078]  PRVF-5424 : Clock time offset check failed INFO: FINE: [Task.perform:594]   TaskNTP:Network Time Protocol (NTP)[TASKNTP]:TASK_SUMMARY:FAILED:CRITICAL:VERIFICATION_FAILED           ERRORMSG(GLOBAL): PRVG-1015 : Time server "10.254.225.252" has time offsets that are not within permissible limit "1000.0" on nodes "acm-702".            ERRORMSG(GLOBAL): PRVF-5424 : Clock time offset check failed           ERRORMSG(acm-702): PRVF-5413 : Node "acm-702" has a time offset of 334643.0 that is beyond permissible limit of 1000.0 from NTP Time Server "10.254.225.252"   

Resolution

The resolutions for each scenario are below:

Scenario 1:  Connection Timeout

The /etc/resolv.conf file contained this information:

search <domain>.com
nameserver 111.22.333.123
nameserver 111.22.333.444

The problem may be resolved by removing the nameserver entry for the IP address that could not be resolved (nameserver 111.22.333.123) and changing etc/resolv.conf to:

search <domain>.com
nameserver 111.22.333.444

Because the installation failed previously, the correct next sequence of steps is to uninstall the failed product then reinstall, as follows

1. As 'root', execute /tmp/aveksa/staging/deploy/uninstall.sh
2. Make sure that all Oracle processes are gone before resuming the installation process.  
3. As 'root', execute /tmp/aveksa/staging/install.sh

Scenario 2:  nslookup Failure

The Oracle 12 Grid installation does a more rigid confirmation of the hostname using the Linux command nslookup.  In this scenario, the network configuration was not able to be confirmed. To verify, run the linux command nslookup <hostname_alias>, outside the installation and note the same NXDOMAIN failure command that occurred during the installation.

To resolve this problem, contact your internal network team to resolve the nslookup issue. 

WARNING: Do NOT temporarily rename /etc/resolv.conf to another name and reinstall as this will bypass important verification steps and there will be future problems.
 

Because the installation failed previously, the correct next sequence of steps is to uninstall the failed product then reinstall, as follows

1. As 'root', execute /tmp/aveksa/staging/deploy/uninstall.sh
2. Make sure that all Oracle processes are gone before resuming the installation process.  
3. As 'root', execute /tmp/aveksa/staging/install.sh

Scenario 3:  ntpq Time Lookup Failure

The Oracle 12 Grid installation uses the ntpq -p command to ensure the time difference between the machine and the NTP time server is less than one second. In this scenario, the time of the machine was more than one second off from the time that was reported by the NTP server configured on the machine.

1. Confirm that an NTP server is configured and that the offset time is less than 1000 milliseconds by running the ntpq command with the -p parameter.

ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*mickey.lss.emc. 10.254.225.252   2 u   53   64  377    2.221    5.507   0.092

2. If the offset is greater than 1000, then as the 'root' user run the ntpd -q command to reset the system time from the timeserver and test again.

ntpd -q

         

NOTE: If you are running SuSE Linux, please use the below command.  [The below output shows the result of the command run in SuSE 11 SP3].

                      

# rcntp ntptimeset
 Output: Time synchronized with mickey.lss.emc.com

Because the installation failed previously, the correct next sequence of steps is to uninstall the failed product then reinstall, as follows

1. As 'root', execute /tmp/aveksa/staging/deploy/uninstall.sh
2. Make sure that all Oracle processes are gone before resuming the installation process.  
3. As 'root', execute /tmp/aveksa/staging/install.sh