000034352 - All token authentications in RSA Authentication Manager 8.2 Patch 1 fail with "passcode reuse or previous token code detected"

Document created by RSA Customer Support Employee on Mar 22, 2017
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000034352
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2.0.1
IssueAfter applying Authentication Manager 8.2 patch 1, all token authentication attempts to the primary and all replicas will fail with the following error:
Passcode reuse or previous token code detected
 

Fixed passcodes and emergency tokencodes work successfully.
Other symptoms include:
  • Time shifts in the system logs or System Activity report from the Security Console
10/26/2016 2:15    WARN    26041    Clock Setback Detected    Detected clock setback, current:“2016-10-26 06:15 UTC” expected:“2016-10-26 12:15 UTC”    
Warning    Adjudicator detected clock set back    SYSTEM    CLOCK_SET_BACK_DETECTED <FQDN>    <IP_Addr>    
thmgr.internal.adjudicator.am.ClockSetBackDetector

  • The system log also shows time jumping backwards:
    10/26/2016 3:03    INFO    16256    Update configuration    
    Administrator “SYSTEM” updated configuration parameter “ims.weblogic.domain.secret” 
    10/26/2016 3:03    WARN    26070    Started UDP Server    
    “Authentication service” started on port “5500”    
    10/25/2016 23:56    WARN    16006    Check license for feature    
    System attempted to check the license for feature
    10/25/2016 23:59    WARN    16006    Check license for feature    
    System attempted to check the license for feature
    10/26/2016 0:10    INFO    16032    Delete batch job    
    Administrator “SYSTEM” attempted to delete batch job “CleanupExpiredSMSAuthenticatorsJob”    Success

    • The /opt/rsa/am/server/logs/biztier.log shows time jumps forward and time setback 
    STATUS | wrapper | main | 2017/05/06 04:01:23 | <-- Wrapper Stopped
    STATUS | wrapper | main | 2017/05/06 10:10:31 | --> Wrapper Started as Daemon

    2016/10/26 08:13:32 | <Oct 26, 2016 8:13:32 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.> STATUS | wrapper  | main
    2016/10/26 04:01:16 | TERM trapped.  Shutting down. INFO   | jvm 1    | main
    2016/10/26 04:01:17 | <Oct 26, 2016 4:01:17 AM EDT> <Notice> <WebLogicServer> <BEA-000388> <JVM called the WebLogic Server shutdown hook. 
    The server will force shutdown now.> INFO
    2016/10/26 04:01:17 | <Oct 26, 2016 4:01:17 AM EDT> <Notice> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by 
    <WLS Kernel>.> INFO ...<BEA-000365> <Server state changed to FORCE_SUSPENDING.>...<Notice> <Server> <BEA-002607>...<IP>:7002, was shut down.> INFO

    • In the authentication activity log (Reporting > Reports or Reporting > Real Time Activity Monitors > Real Time Authentication Activity), the following entries are seen:
    Authentication attempted
    Passcode reuse or previous token code detected for user “<UserID>” in security domain “SystemDomain” from “Internal Database” identity source. Request originated from agent “<FQDN_of_agent>” with IP address “<IP_address_of_agent>” in security domain “SystemDomain” with protocol version “Internal Database”. Authentication method: “SecurID_Native”, Authentication policy exp: “”, Activation Group: “”, Token serial number: “<Token_serial_number>”, Alias: “”
    AUTHN_METHOD_FAILED
    Authentication method failed
    Bad tokencode, but good PIN detected for token serial number “<Token_serial_number>” assigned to user “<UserID>” in security domain “SystemDomain”
    CauseThe hardware clock is set ahead of the system/NTP time; meaning the hardware clock is in the future relative to) the system time.
    Access Authentication Manager through an SSH session and compare the date system time (NTP) with the hardware clock.  They should be the same; but if not, that is what is causing the problem.
    1. From an SSH session or when directly connected to the Authentication Manager primary, run the following:
    login as: rsaadmin
    Using keyboard-interactive authentication.
    Password: <enter operating system password>
    Last login: Mon Nov 21 11:18:49 2016 from jumphost.vcloud.local
    RSA Authentication Manager Installation Directory: /opt/rsa/am
    rsaadmin@am81p:~> sudo su -
    rsaadmin's password: <enter operating system password>
    am81p:~ # date
    Tue Nov 22 16:24:18 EST 2016
    am81p:~ # hwclock -r
    Tue Nov 22 16:24:25 2016  -1.057803 seconds
    am81p:~ #

    1. Be sure to note the time zone returned when running date and the time when running hwclock -r.
    2. The values for date and hwclock -r should be the same.
    Versions prior to Authentication Manager 8.2 patch 1 were not affected by this problem. If there was a discrepancy in the times, the incorrect time on the hardware clock was ignored.  Starting in Authentication Manager 8.2 patch 1, a reboot triggered new code that used the hardware time as a starting point when starting the Authentication Manager services, before checking the system/NTP time, and whether the hardware clock time was in the future as compared to the correct time.
    Starting the Authentication Manager services set the token High Water Mark (HWM) to that future time.  The HWM is kind of a shortcut used by the Authentication Manager Adjudicator (replication service) to quickly determine if any tokencode had been used previously.  In short, any tokencode or passcode from a time (relatively) earlier than the HWM automatically is considered as previously used.  In this situation where the HWM is mistakenly set to the future by reading the incorrect time on the hardware clock, followed by NTP fixing the time to correct time, which causes the problem.
    All tokencodes from all tokens will not work until system time catches up to the future HWM time.  Fixed passcodes and emergency tokencodes or passcodes will still work, since they are not time-based and cannot have a High Water Mark.
    ResolutionIf the hardware clock is different from the system clock, but system time is correct, then you can manually update the system time to the hardware clock using the command hwclock -w to write the system time to hardware clock.  To complete this, follow the steps below:
    1. SSH to the Authentication Manager server.
    2. Stop the Authentication Manager services.
    3. Change to the root user.
    4. Write the hardware clock value.
    5. Exit out of root.
    6. Restart the Authentication Manager services.
    login as: rsaadmin
    Using keyboard-interactive authentication.
    Password: <enter operating system password>
    Last login: Mon Nov 21 11:18:49 2016 from jumphost.vcloud.local
    RSA Authentication Manager Installation Directory: /opt/rsa/am
    rsaadmin@am81p:~> cd /opt/an/server
    rsaadmin@am81p:/opt/rsa/am/server> ./rsaserv stop all
    Stopping RSA RADIUS Server: **
    RSA RADIUS Server                                          [SHUTDOWN]
    Stopping RSA Runtime Server: ***
    RSA Runtime Server                                         [SHUTDOWN]
    Stopping RSA Console Server: **
    RSA Console Server                                         [SHUTDOWN]
    Stopping RSA Replication (Primary): **
    RSA Replication (Primary)                                  [SHUTDOWN]
    Stopping RSA Database Server: *
    RSA Database Server                                        [SHUTDOWN]
    Stopping RSA RADIUS Server Operations Console: **
    RSA RADIUS Server Operations Console                       [SHUTDOWN]
    Stopping RSA Administration Server with Operations Console: **
    RSA Administration Server with Operations Console          [SHUTDOWN]
    rsaadmin@am81p:/opt/rsa/am/server> sudo su -
    rsaadmin's password: <enter operating system password>
    am81p:~ # hwclock -w
    am81p:~ # exit
    logout
    rsaadmin@am81p:/opt/rsa/am/server> ./rsaserv start all
    Starting RSA Administration Server with Operations Console:
    Starting RSA Database Server: *************
    RSA Administration Server with Operations Console          [RUNNING]
    Starting RSA RADIUS Server Operations Console: \ RSA Database Server                                        [RUNNING]                   **************
    RSA RADIUS Server Operations Console                       [RUNNING]
    Starting RSA Runtime Server: ***************************
    RSA Runtime Server                                         [RUNNING]
    Starting RSA RADIUS Server: **
    RSA RADIUS Server                                          [RUNNING]
    Starting RSA Console Server: *
    Starting RSA Replication (Primary): ***
    RSA Replication (Primary)                                  [RUNNING]*****************
    RSA Console Server                                         [RUNNING]
    rsaadmin@am81p:/opt/rsa/am/server>

    Engineering will provide a patch (or hot fix) in the near future to avoid this situation  
    WorkaroundAs a work around, wait for the system time to reach the time that the hardware clock was at when the reboot happened; basically wait to pass the value of the High Water Mark.
     
    NotesJira bug AM-30552 - CE Assist: RCA on AM 8.2 upgrade failure - Primary had all "Passcode reuse or previous token code detected"
     “previously used passcode” 
    High Water Mark, HWM
     

    Attachments

      Outcomes