How to upgrade an RSA SecurID Access IDR
2 years ago
Originally Published: 2016-10-12
Article Number
000065118
Applies To
RSA Product Set: SecurID Access
RSA Product/Service Type: Identity Router
RSA Version/Condition: October 2016
 
Issue
Starting from the October 2016 release, you can either apply the latest update to all identity routers in each cluster immediately, or schedule a date and time for each cluster in your deployment to install the latest software version. When you schedule an update, the identity routers in that cluster automatically install the update in rapid succession, starting on the date and time you specify.
RSA will announce new RSA SecurID Access releases on RSA Link, and set an update schedule. The Dashboard page in the Administration Console lists schedules for clusters that contain out-of-date identity routers. On the clusters page you can select a cluster to view or modify its update schedule. 
Tasks

How will you know that an update is available?

RSA will post a Release Notification on RSA Link when the update is available.  Customers who have followed the RSA SecurID Suite Service Notifications page on RSA Link will receive an email.
When an admin logs into the SecurID Access Administration Console, they will see a notice on the dashboard if both of the following are true:
  • The minimum configuration has been completed (Protected Domain Name configured, SSL certificates uploaded, IDR registered, Identity Source configured, and at least one Application configured)
  • There are 1 or more IDRs that are OUT_OF_DATE (not running the latest available release)
The dashboard message will appear in a yellow box, and contain text like this:

Each of the following clusters contains at least one identity router that is not running the latest software version. Automatic updates are scheduled for the dates and times shown.
  • MyClusterName:  12/12/16 3:45am MDT
To modify this update schedule, go to Platform > Clusters.

Note:  If an IDR is both OUT_OF_DATE and in DEBUG logging mode, the dashboard and IDR list page status will be inconsistent.  If you refresh, the dashboard message may appear or disappear, and the IDR list page will alternate between showing OUT_OF_DATE or DEBUG.

When will you be able to update you IDRs?  When should you update?

As soon as the update is available, you can update your IDRs or schedule the update for a date and time suitable to you.  However, you should follow or regularly check the RSA SecurID Suite Service Notifications page to avoid updating your IDRs during RSA SecurID Access maintenance.  You should change your scheduled update date/time if RSA SecurID Access maintenance is announced for a time that clashes with it.

Default Schedule

At some point after the Release Notification, RSA will provide you with a default schedule if no schedule has yet been set by you.  The default schedule for Development environments will typically be earlier than for Production environments.  Update not scheduled will appear in the dashboard notification instead of a date/time if  you have not scheduled the update and if a default date/time has not yet been set by RSA.

Must-update-by

RSA will also communicate a must-update-by date.  All customers must update their IDRs by the must-update-by date.  Any IDRs still running the older release as of the must update-by date will be updated automatically by RSA.  Note that if a default schedule is set by RSA, it may be earlier than the must-update-by date. RSA will likely post a reminder to the Service Notifications page on RSA Link, approximately 2 weeks prior to the must-update-by date.

Multiple Clusters

If you have multiple clusters within the same environment (like a DR environment), you may opt to update the clusters at two different times.
You have the option of updating one cluster, testing there, and directing traffic to the other cluster to minimize end-user impact.  
However, note that if synchronization is configured between clusters, you may or may not be able to synchronize while different clusters are running different IDR versions. The ability to synchronize will be release-specific.  For the upgrade to the October 2016 release, we expect this to work.

Resolution

Expectations - Before/During/After updates

What can an admin do with an out-of-date IDR?

Admins will still be able to Publish configurations to an OUT_OF_DATE IDR, but they will need to update to the latest version in order to take advantage of new features.  Note that this is discouraged as it may be confusing.  The UI will not clearly indicate which features are only available on new IDRs. 
Some connectors may require the latest version IDR (if the connector expects to use a newer adapter API, or relies on latest-release IDR side features).

What should an admin do before updating?

Publish the current config and verify that things are working as expected.  All pending changes will be published as part of the update, so before scheduling an update, ensure that there aren't any half-configured settings.
Confirm that the update is scheduled for an off-peak time, as there may be some downtime depending on the cluster configuration settings.  Follow best practices for high availability to minimize downtime.

What to expect during an update?

Each IDR will stop services, install the new software, and reboot before reconnecting to the hosted service.  When updating an entire cluster, only one IDR will be updated at a time.  The order of IDR updates cannot be specified when updating an entire cluster.
Internally, the IDRs have a job that will check for new software packages each day, and pre-download the new packages so that the installation process/downtime doesn't have to wait for each of the downloads (provided the IDRs have had an opportunity to pre-fetch the packages!)
The Admin UI will provide some indication about what is happening with the update, but does not currently provide many details.  Here are some things you can look for, however there's no need to sit at the admin UI clicking refresh to watch these updates, as the messages are infrequent:
  • Once a scheduled cluster update begins, the update schedule is cleared.  There is no "Update In Progress" indicator on the dashboard.
  • If there is a cluster of 1 IDR, and the update has started, the Dashboard will not show the yellow update message (the upgrading IDR is considered DISTRESSED rather than OUT_OF_DATE at this point)
  • If there is a cluster of 3 IDRs, and 1 IDR update has started, and the other 2 are pending, the Dashboard will show the yellow update message, and include in the list:  <cluster name>: Not Scheduled
  • If there is a cluster of 3 IDRs, and 2 IDR updates have completed, and 1 is in progress, the Dashboard will not show the update message (the upgrading IDR is considered DISTRESSED rather than OUT_OF_DATE at this point
  • Once services have stopped, the IDR will change from OUT_OF_DATE to DISTRESSED (visible on the Platform > Identity Routers page)
  • Once the update has completed and the IDR has successfully reconnected, it will appear as ACTIVE (visible on the Platform > Identity Routers page)
Updates can take several minutes – typically between 10 to 30 minutes per IDR, but on slower systems they may take longer.  If an update has not completed within 40 minutes, the cluster update logic will consider that IDR update to have failed.  If it was the first IDR in the cluster that failed, subsequent IDR updates will not be attempted.  If the first IDR update succeeded, but another failed, then any subsequent IDR updates will still be attempted.  It is possible that the IDR may recover on its own. See Support & Troubleshooting section, below.
You SHOULD NOT REBOOT the IDR during the update process.  Rebooting during the middle of an update could result in an IDR that does not come back online, and it might have to be re-deployed.  
While IDR services are stopped, admins will be unable to Test or View Log from the IDR in the Access Console, and you will not be able to access the IDR Setup Console (a.k.a. setup.jsp) to obtain a log bundle.

End-user experience during an update

If you have a high availability cluster with properly configured load balancer and session replication, users may see little or no impact.  Even with HA, however, the load balancer needs to detect a pool member being down for some period of time before removing it from rotation, so some users may still be directed to a "down" (updating) IDR during that time, and see an error message.
If you have a single IDR environment, then all services will be down during the update.  End-users will need to log in again after the update is complete.

Support & Troubleshooting

If engaging RSA Support for assistance with an upgrade, please provide the following information:
  • When did you start the IDR update (date and time, including timezone)?
  • How did you initiate the update(s)?  It will be one of:
    • Platform > Clusters > Update > Later (scheduled)
    • Platform > Clusters > Update > Update now
    • Platform > Identity Routers > Update (individual IDR update)
  • IDR Logs
    • symplified.log
    • update.screen (will show upgrade-specific logs)

Messages

Examples of event messages that may be logged during upgrade:
  • Starting cluster upgrade:
    • Starting update for each appliance in cluster [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06]
  • Starting an IDR upgrade:
    • Starting update for IDR [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06, IDR: defiant-yeti-ngx-qe06 IDR2]
  • IDR upgrade  finished - successfully
    • IDR update complete and reporting system status OK after 96 seconds [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06, IDR: defiant-yeti-ngx-qe06 IDR2]
  • 1st IDR upgrade did not finish – timed out (after 40min) – not upgrading any other IDRs in the cluster
    • First IDR in cluster failed to complete update within 2700 seconds.   Will not proceed with any remaining IDR updates in this cluster. [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06, IDR: defiant-yeti-ngx-qe06 IDR2]
  • Another (not 1st) IDR upgrade failed – but we will continue with other IDR upgrades
    • IDR update failed to complete within 2700 seconds. [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06, IDR: defiant-yeti-ngx-qe06 IDR2]
  • Cluster upgrade finished - successfully
    • Cluster update completed successfully [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06]
  • Cluster upgrade finished – but some IDRs may not have been successful
    • Cluster update finished with some errors.  Check IDR status.  [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06]  
  • IDR update skipped because not publishable or is Inactive (if it’s the first in the cluster, we’ll still proceed with the next – it essentially won’t count)
    • Skipping update for IDR because it is inactive or distressed. [customer: defiant-yeti-ngx-qe06, cluster: DefaultApplianceCluster_defiant-yeti-ngx-qe06, IDR: defiant-yeti-ngx-qe06 IDR2]

 

Related Articles

Trending Articles

Don't see what you're looking for?