How to troubleshoot an RSA Identity Router that is in a Distressed state

[#1]

Article Number

000036759

Applies To

RSA Product Set: SecurID Access
RSA Product/Service Type: Identity Router

Issue

When an RSA Identity Router (IDR) is distressed, you will see the following in the Cloud Administration Console:

• The System Status - Identity Routers section of the Dashboard will show the number of Identity Routers that are distressed in red.
• The Platform > Identity Routers page will show the Identity Router as Distressed.

Cause

The Cloud Administration Console shows an IDR as distressed for one or more of these reasons:
 

A networking problem

• The Cloud cannot connect to the Identity Router.
• The Identity Router cannot connect to the Cloud.

A service problem

• One or more services that should be running on the IDR, are not running.

A hypervisor problem

• Due to a problem with the IDR's hypervisor or VM, the IDR is is not running, or is not performing as required.

An expected outage

• A deliberate change has been done which has the known and expected side-effect of the IDR being temporarily in Distressed state.

Resolution

To determine if the IDR is distressed due to a networking, service or hypervisor problem, or an expected outage, check the following.  Use the links provided to learn more about each item:

• Has a deliberate action has been taken that is expected to cause the IDR to be temporarily Distressed: 
  • A newly added IDR,
  • Publishing Changes to the Identity Router and Cloud Authentication Service,
  • IDR software update,
  • IDR reboot,
  • IDR restart or
  • IDR VM stop/start? 

If so, this is an expected outage.

• Is there a response if you Test the IDR?  If not, this is likely a networking or hypervisor problem.
  • Check  your hypervisor server (VMWare or Hyper-V).  Is the hypervisor itself or the IDR's VM, in a stopped or stopping state, or running out of resources (CPU, memory, etc) or in any other undesirable state?  If so, this is a hypervisor problem.  
  • If all looks well on the hypervisor, this is likely a networking problem.
• Is there a response when you Test the IDR, but not all services are in running state?   All services should be running, except possibly the two below:
  • ssoLifecycleService:  this service should be in paused state if SSO Agent is disabled on the cluster.
  • radiusService:  this service should be in paused state if RADIUS is disabled on the cluster.

If either of the above services are not in running state when they should be, or if any other services are not in running state, there is a services problem.

Now, go to the appropriate section below for suggested troubleshooting tasks, based on your conclusions from the above questions.

Networking Problem

• Check if any recent network changes or Cloud Authentication Service configuration changes could have introduced a problem. Examine any such changes for errors. Is more than one IDR affected? Cloud and IDR connectivity requirements are documented in the following RSA Link locations:
  • The chapter entitled "Step 1. Plan" in each Quick Setup Guide available on the Cloud Authentication Service Planning and Configuration page.
  • Configure Company Information and Certificates
  • Security Levels and Identity Router Connection Ciphers (RSA Link login required) 
    • If your RSA SecurID Access deployment includes a component that terminates the SSL connection between the identity router and the Cloud Authentication Service, you must ensure that this component is configured to use at least one cipher suite supported by the identity router when reestablishing the SSL connection.
  • Identity Router DNS Requirements
  • Identity Router Default Ports and Interfaces.
  • Identity Router Network Interfaces.
• Read Identity router status changed to distressed after reboot in RSA SecurID Access
• Read the Cloud Authentication Service's Service Notifications page on RSA Link for notifications of maintenance or outage that may be impacting connectivity between the Cloud and IDRs (RSA Link login required).
• Are there currently any network outages in your organization's network, or that of your organization's ISP?
• Can you access the IDR's setup.jsp pages?  If so:
  • Generate and download an Identity Router log bundle from the IDR's setup.jsp pages.
    • Review the contents of identity router log bundle, and especially the symplified.log in the bundle, for connectivity error events.
  • Check the IDR's Network Diagnostics page for errors.
• If SSH was previously enabled for the IDR, can you access the IDR by SSH? 
  • Note that if SSH was not previously enabled, you will not now be able to enable it until the network problem is fixed.
• If the IDR is completely unreachable by setup.jsp and SSH:
  • Can you access the network settings page in the Identity Router VM console?  If so, check if the settings there are correct and if not, adjust accordingly.
  • If there is no means to access the IDR, or if adjusting network settings in the IDR VM console doesn't help, you can stop and restart the IDR's VM (from the hypervisor) to see if that fixes the problem.

Services Problem

Follow these steps to gather data and pass it to RSA Support:

1. Set the Identity Router Logging Level to Debug, then wait 5 minutes for internal IDR logging to capture activity.
2. If you need to resolve the issue as quickly as possible (rather than referring it to RSA Customer Support first), you can try one or more of the following to see if they fix the problem:
   • Restart services on the IDR
   • Reboot the IDR.
   • From the hypervisor, shutdown then restart the IDR's VM (recommended only as a last resort)

Make a note of which of the above were done, the date and time they were done (with timezone) and the outcome (fixed or did not fix the problem).
Note that if you decide to later refer the matter to RSA Support, these actions will make it less likely that RSA will be able to determine root cause of the issue.

3. Set the Identity Router Logging Level to Standard.

• ​If the problem is not fixed, continue with the remaining steps below.

4. Generate and Download an Identity Router Log Bundle.
5. Contact RSA Customer Support to log a support case.  Also provide the actions you tried at step 3 above, and the date/time of each (with timezone).
6. Upload the Identity Router Log Bundle to RSA Customer Support for analysis.

Hypervisor Problem

• Ensure Identity Router Virtual Appliance Hardware and Software Requirements are all being met.
• Refer the matter to the administrator of your hypervisor server (VMWare or Hyper-V).

Expected Outage

• Wait until the cause of the outage is completed.  Then, monitor the IDR to ensure it returns to an Active state.
• If the cause is an IDR software update, see:  What to expect during an RSA SecurID Access Identity Router (IDR)/Cluster software update
• If you are adding a new IDR and after waiting for 20 minutes or so the IDR remains in Distressed state, confirm that all of the necessary requirements have been met and configurations done appropriately for your deployment.  Check that:
  • Configure Company Information and Certificates has been done.
  • Security Levels and Identity Router Connection Ciphers (RSA Link login required) requirements have been met.
    • If your RSA SecurID Access deployment includes a component that terminates the SSL connection between the identity router and the Cloud Authentication Service, you must ensure that this component is configured to use at least one cipher suite supported by the identity router when reestablishing the SSL connection.
  • Identity Router DNS Requirements have been met.
  • Identity Router Virtual Appliance Hardware and Software Requirements have been met.
  • Identity Router Default Ports and Interfaces are accessible.
  • Identity Router Network Interfaces requirements are met.
  • All necessary steps for Deploying an Identity Router have been done correctly.
    • The IDR will not change to Active status until step 7 in Deploying an Identity Router has been completed and all other required configuration items are correctly done.

Notes

If further assistance is required, contact RSA Customer Support.