How to troubleshoot On-Demand Authentication (ODA) login failures in RSA Authentication Manager 8.1

[#1]

Article Number

000029925

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.1, 8.1 SP1

 

Issue

A specified user is getting an authentication failure;  however, the server does not display an access denied message. The problem is intermittent or sporadic. Sometimes after entering the On-Demand Tokencode (ODT), the user ends up back at the login or logon prompt instead of being allowed access, yet no authentication failure appears in the Authentication Manager logs or in the real time authentication activity monitor.

Task

To troubleshoot this or similar ODA authentication failures, you need network packet captures running on the primary and on any replica servers in the deployment.  These packet captures should be filtered on port 5500 UDP for authentication and port 25 TCP for SMTP mail.

Resolution

Start the TCP dump on the primary and all replicas in separate SSH sessions to capture both SMTP and authentication traffic.  Make sure the tcpdump command saved the output to a file. When the next intermittent ODA login failure happens, you will capture the packets.  Once the data is logged, stop the captures and send the network packet capture files to RSA support. 

The capture files will show if both the primary and any replicas are sending an email, if the replica is sending two emails, or if the primary and any replicas are not sending any emails.

1. Launch two SSH sessions to the Authentication Manager primary with the operating system account rsaadmin.
2. In Session 1,
   1. Sudo to the root user.
   2. Navigate to /usr/sbin.
   3. Start the tcpdump to capture the SMTP traffic.

login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter OS user password>
Last login: Mon Sep 12 15:13:53 2016 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am81p:~> sudo su -
rsaadmin's password: <enter OS user password>
am81p:~ # cd /usr/sbin
am81p:/usr/sbin # ./tcpdump -i eth0 -s 1514 -Z root port 25 -w /tmp/smtp_<server_name>.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes

3. In Session 2,
   1. Sudo to the root user.
   2. Navigate to /usr/sbin.
   3. Start the tcpdump to capture the authentication traffic.

login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter OS user password>
Last login: Tue Sep 13 14:40:59 2016 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am81p:~> sudo su -
rsaadmin's password: <enter OS user password>
am81p:~ # cd /usr/sbin
am81p:/usr/sbin # ./tcpdump -i eth0 -s 1514 -Z root port 5500 -w /tmp/auth_<server_name>.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes

4. Concurrently with steps 1 through 3 on the primary, repeat steps 1 through 3 on all replica servers
5. Let the captures run until the failures occur.  When the logon failure occurs, stop the TCP dump in each session by typing Ctrl+C.
6. On each server, navigate to /tmp to change permissions on the files.

am81p:/usr/sbin # cd /tmp
am81p:/tmp # ls -al *.pcap
-rw-r--r-- 1 root root 24 Sep 13 14:51 auth_<server_name>.pcap
-rw-r--r-- 1 root root 24 Sep 13 14:51 smtp_<server_name>.pcap
am81p:/tmp # chmod 777 *.pcap
am81p:/tmp # ls -al *.pcap
-rwxrwxrwx 1 root root 24 Sep 13 14:51 auth_<server_name>.pcap
-rwxrwxrwx 1 root root 24 Sep 13 14:51 smtp_<server_name>.pcap
am81p:/tmp #

6. Copy the smtp._<server_name>pcap and the auth_<server_name>.pcap off the Authentication Manager primary and replicas using  WinSCP or a similar program.
7. In addition to providing the .pcap files to RSA Support, please login to the Operations Console for each server and download the troubleshooting logs (Administration > Download troubleshooting Files) and send those along as well.  Note that a password is required when creating this file and that password needs to be provided to RSA support.
8. Also send a copy of the authentication activity report covering the time frame of the ODA failures (Reporting > Reports > Add New > Authentication Activity).

Notes

Big Picture of an On-Demand Authentication

For an ODA/ODT login success,

1. The user enters their PIN.  
2. The PIN triggers an email or an SMS delivery of the On-Demand Tokencode.  
3. The use enters the ODT to complete the authentication request.
4. The second login screen often says to enter the next tokencode, even for an ODA.

From the Authentication Agent

1	Primary/replica	<	PIN received for user ID
2		>	Email sent to user with On-Demand Tokencode (ODT)
3	Primary/replica	<	User enters ODA code
4		>	Successful authentication

Note:  If the authentication in Step 3 takes more than 60 seconds after the PIN is entered in Step 1 some agents timeout and do not send the Step 3 authentication request and nothing shows in the authentication activity logs because the authentication request never arrived.  Moreover, the user has to enter their PIN again to trigger a second email/SMS ODT.