I currently have an Authentication Manager Appliance (1U) from RSA/EMC that had AM8.2 p3 on it previously. Once I installed the p4 patch, all my users were not able to authenticate in and needed to have their tokens resyc'd. Is there a reason and/or fix for this? Thank you.
Okay, hence the "hazard a guess"
basically there's only a few things that would cause this type of issue.
The first and most common is time....whenever there's a time change then tokens will need to be resync'ed.
The best way to identify that is with the setsync utility on the RSA server. You would run it when you have the problem to get an initial list and then try resyncing a couple tokens manually and run it again to see what the offset updated to. that would give you the time discrepancy.
The other reason is replication....If for some reason there was a replication problem and users were hitting a replica that wasn't talking to Primary then after patch it synched back up then that could cause token to be out of sync, though this one is less likely.
The third option would be only if it was all software tokens and you also would not be able to resync and would have to redistribute. It is possible to bulk redistribute all software tokens and if this was done would cause all currently installed software tokens to stop working and would need to resend them to all users.....from what you described this doesn't really seem applicable.
Hi Brian and thanks for responding. Based on what you have mentioned I'd put my money on the time being the issue. The reason I say that is because in this particular environment that I had this issue in, there's only one appliance so replication wouldn't be the issue. And lastly we aren't using software tokens which leads me to your initial thought of the time server being an issue. I'm going to work with the person who manages the NTP server to see if they see anything on their side that may have caused this.
Mr. Twomey, you in fact were correct about the time being the issue! After running the commands Erica suggested to pull the report I can see the offset of time between each hardware token. Not to mention the difference in time when running the commands Jay had suggested as well. Thank you!
Did you resynchronize all of the tokens or just the ones for users who had trouble authenticating after the patch process?
You could open an SSH session to your primary and run a command to create a report showing token offsets in the database. For anyone who had a working token that was not resynchronized today you may see an offset that would prove out Brian Twomey's point.
1. To generate the report, SSH to your primary and do the following:
login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter the operating system password>
Last login: Thu Feb 2 15:29:21 2017 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am81p:~> cd /opt/rsa/am/utils
rsaadmin@am81p:/opt/rsa/am/utils> ./rsautil sync-tokens -IAuthenticator Bulk Synchronization Utility 8.1.1.13.0 (1384830)
Copyright (C) 1994 - 2014 EMC Corporation. All Rights Reserved.Enter the absolute path for the output report file : /tmp/token_report.txt
Enter the base security domain name for recursive search [(none)]: none
Enter the type of token selection [ (all) | file ]: all
Choose a token filter [ assigned | unassigned | (both) ]: both
What action do you wish to perform? [ (list) | modify ]: list
Enter administrator user ID : <enter the user name of a superadmin>
Enter administrative password : <enter password for the super admin>Authenticator Bulk Synchronization Utility 8.1.1.13.0 (1384830)
Copyright (C) 1994 - 2014 EMC Corporation. All Rights Reserved.Started job on Fri Feb 03 17:08:53 EST 2017 with ID = ims.0509d9fd1e02a8c01949f3de6f56f9f2
2. When done, use WinSCP or FileZilla to copy the file from the server to your workstation for review.
3. Open the file and check the column labeled Clock Offset. The offset values with be something like 0, 1, 2, 3, etc. and can be either positive or negative values, indicating if the offset is correcting a time that is fast or slow.
An offset of 0 means the token is authenticating in what we call the perfect minute. Other values mean the server is looking at tokencode values in the past or present from the perfect minute to compare the submitted passcode to what is expected.
Typically, Authentication Manager can authenticate users with a token offset +/-3. Any window more than that would require the token to be resynchronized.
Regards,
Erica
Hi Erica and thanks for responding. I had to re-sync all the tokens that were affiliated with this appliance. I'm going to test what you have layed out for me to prove exactly what Brian Twomey's point. I'm going to get back once I've completed the steps you've provided to me to confirm.
Erica thanks so much for those instructions! They were very helpful and clear to understand. Essentially I ran and obtained the report to see the "Clock Offset" and the only perfect token time was mine, probably because I manage the appliance and make sure my token is constantly working. For those users that rarely token in, they have huge offsets and more than likely need to be re-sync'd with the appliance.
Did the authentication failures have any specific errors or just passcode incorrect? If it just said incorrect and resync fixed it, you could also check the hardware clock time in addition to what Erica said about the time offsets. Compare HWclock to the date to make sure they are the same.
login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter operating system password>
Last login: Mon Nov 21 11:18:49 2016 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am81p:~> sudo su -rsaadmin's password: <enter operating system password>
am81p:~ # date
Tue Nov 22 16:24:18 EST 2016
am81p:~ # hwclock -r
Tue Nov 22 16:24:25 2016 -1.057803 seconds
am81p:~ #
Hi Jay, thank you as well for assisting. The only message we got when trying to authenticate using the hardware tokens was "user not found or passcode incorrect". The first thing I tried once I had encountered this issue was to re-sync the token and that seemed to have resolved the issue. My worry is that when patching another environment, if this were to happen again I'd have a ton of users I would potentially have to re-sync tokens. Not to mention the getting famous in the environment which is not something I'd like to do. I'm working on testing the steps Erica and you have suggested and will respond back once completed. Thanks again!
Much appreciated Jay, so you as well were correct there was about a minute difference in time between the two:
rsaappliancenamehere:/ # date
Mon Feb 6 16:37:59 EST 2017
rsaappliancenamehere:/ # hwclock -r
Mon 06 Feb 2017 04:38:20 PM EST -0.688269 seconds
Hi Michael,
We appreciate the update and the feedback. Looks like we were all honing in on the time. Hopefully this helps going forward.
Yes and thank you all for your awesome feedback!
Hi Michael,
If I had to hazard a guess I'd say that the server was probably not on the correct time prior to the patch and as part of the patch it does need to restart some RSA services which could then force it to go out and sync to NTP server thus changing the time. If the server had been out of sync for any length of time the tokens would have built an offset which would then be a problem when the time was updated.