000012389 - Passcode reuse or previous token code detected for user in Authentication Activity report for RSA Authentication Manager 7.1

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000012389
Applies ToRSA Product Set:  SecurID
RSA Product/Service Type:  Authentication Manager
RSA Version/Condition:  7.1
 
Issue
  1. Many instances of the following message are seen in the Authentication Activity report:

Passcode reuse or previous token code detected for user


  1. There is nothing in the Administration Log report to suggest administrative changes before the error message is logged.  
  2. The system log shows the following messages:

ADJUDICATOR_FAILOVER.message
ADJUDICATOR_OFFLINE
ADJUDICATOR_FAILOVER


IMPORTANT Prerequisites: There are several possible common issues that need to be checked, and any problems ruled out first: 
  1. The tokencode really is being reused either by the user, being intercepted and replayed by another system, or being resent by a misconfigued agent.
  2. The clocks on all servers must be correct and synchronized within a few seconds of each other (compensate for different timezones).
  3. Replication seems healthy, no issues should be found
  4. Network problems that can prevent an agent or RADIUS device from operating properly.
  5. An agent retry interval is set too low.
  6. A RADIUS client has a retry or timeout set too low.
  7. A RADIUS client is improperly re-sending a Access-Request, regardless of retry and timeout settings.
  8. If using Authentication Manager 7.1 SP2 or SP3 on Windows, a buildup of .sql files in the system temp directory.
  9. No sudden clock changes, (including correcting an incorrect clock, have happened.
  10. No agents, or systems using the Agent API, are version 4.x or older. These use a legacy authentication method known as Acting Master/Acting Slave, which is not supported with Authentication Manager 7.1 or higher.
CauseThe adjudicator is responsible for keeping the token high water mark in memory on each server. This value is being set into the future, which invalidates some tokencodes. 
 
ResolutionAuthentication Manager 7.1 SP4 P14 lists this as fixed (see AM-21692 Tokencodes failed frequently during authentication, which caused the following error message to display in the authentication activity monitor:  "Passcode reuse or previous token code detected").
DO NOT  RESTART SERVICES UNTIL READING THE SECTION BELOW
  • If ALL systems are already at Authentication Manager 7.1 SP4 patch 14 or higher, follow these steps to collect Additional Adjudicator diagnostic information.
  • Authentication Manager 7.1 SP4 P2 and higher have additional Adjudicator tracing available. 
  1. Start the real-time authentication Monitor.
  2. Attempt to resynchronize a token through the Security Console. If it succeeds, it is probably not the issue in JIRA AM-21692.   If it fails with similar symptoms, continue. 
  3. The RSA TSE will contact CE to inform them that there will be incoming data, as they may also want to do some real-time checks.
  4. Make sure all servers or appliances are getting their time from a known-good NTP server, they need to be synchronized. 
  5. Log onto the Security Console of the primary and go to Setup > Instances
  6. Select the primary instance and choose Logging .  Change trace Log to Fatal.
  7. Repeat step 5 for EVERY instance in the deployment.  You change the settings for each replica from the Security Console on the primary.
  8. Open a command prompt or SSH session to the primary.
  9. Navigate to C:\Program Files\RSA Security\RSA Authentication Manager\utils or /usr/local/RSASecurity/RSAAuthenticationManager/utils.
  10. Enable Verbose tracing at the component level on the primary with the following command.  Note the command uses a lower case L before VERBOSE).  
rsautil set-trace -u <name of superadmin user who is in the internal database> -p <password for this superadmin user> -c trace.com.rsa.authmgr.internal.adjudicator -l VERBOSE

  1. Repeat step 8 - 10 for EVERY instance in the deployment.  This needs to be done directly on each server.
  2. Do test authentications with the affected user and token.  Note the time this is done.
  3. Do a test token synchronization through the Security Console with the affected user and token. Note the time this is done, and the results.
  4. Trace information is written to RSA_HOME/server/logs/imsTrace.log and should be provided to the engineer working the issue. Collect this file form EVERY instance in the deployment.  Rename the file slightly so we can identify the server from which it came.
  5. Generate a report for authentication activity, administration activity, and system log from the primary, setting the start date from before the issue started happening. 
    1. Collect the report outputs as .csv files.  
    2. Send these to the RSA engineer.
  6. The RSA engineer will attach the data to AM-21692.
  7. The RSA engineer will inform CE, and send them the data.  If possible, wait for them to have some time to analyze the data, as they may want to do some additional testing with the customer on the line.
  8. After collecting the trace, turn off verbose tracing for the component on the primary with the following command:
rsautil set-trace -u admin -p <password> -c trace.com.rsa.authmgr.internal.adjudicator -l NONE

  1. Repeat the steps above on every instance in the deployment.
  2. Stop all RSA services on the primary and reboot.  Wait for all services to come online, and verify replication is running.
  3. Repeat with each replica, one at a time.
If ANY systems  are at a patch level lower than Authentication Manager 7.1 SP4 patch14 , the additional diagnostic information steps above are not relevant, and the systems will need to be updated to prevent the problem in the future: 
  1. Run a backup from the primary's Operations Console (Maintenance > Backup > Backup Now).
  2. Stop all RSA Services on the primary, and REBOOT.  Wait for all RSA services to restart, and verify you can logon to both the Security Console and Operations Console.
  3. Stop all RSA Services on a replica, and REBOOT. Wait for all RSA Services to restart, and verify you can logon to both the Security Console and Operations Console.
  4. Repeat step 3 for all  other replicas, but only do one at a time.
  5. Upgrade the primary to Service Pack 4, if it is not already at SP4.  
  6. Upgrade a replica to Service Pack 4 if it is not already at SP4. 
  7. Repeat step 6 for any remaining replicas, one at a time.
  8. If the systems are the RSA SecurID Appliance 3.0 and any were upgraded to SP4 as part of steps 5, 6,and 7, or the Operations Console shows a version lower than 3.0.4.9, apply the appropriate SP4 Factory-Reset Patch to each appliance. Do not actually factory-reset any appliances after applying the factory reset patch, unless directed by RSA Support. 
  9. Apply the most recent post SP4 patch to the primary.
  10. Apply the most recent post SP4 patch to a replica.
  11. Repeat step 10 for any remaining replicas, one at a time.
NotesA fix is expected in Authentication Manager 7.1 SP4 patch 14  (RSA SecurID Appliance 3.0.4.24 ) , but it is only relevant for this specific issue, and is not relevant if any of the other issues listed in the Causes are present. 
 
Legacy Article IDa53940

Attachments

    Outcomes