Troubleshooting NTP errors on an RSA Authentication Manager 8.x appliance

[#1]

Article Number

000029291

Applies To

RSA Product Set:  SecurID
RSA Product/Service Type:  Authentication Manager
RSA Version/Condition:  8.x

Issue

RSA Authentication Manager 8.1 Patch 2 included a fix to prevent startup problems when there was an issue with NTP. As part of this patch, additional alerts were added to the system to alert when a NTP error occurs:

Attention! The following critical system event occurred: Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.

System Time Synchronization Configuration Check,"Checking configuration for System Time Synchronization

Warning,All NTP Servers are unavailble - potential for significant system time drift,SYSTEM,,,,,ALL_NTP_SERVERS_UNVAILABLE

Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.,,,,,,,,

Both the Authentication Manager and the SuSE Linux Operating System will attempt to do an NTP Synchronization several times an hour. The SecurID Appliance sends both NTP Version 3 and NTP Version 4 requests.

• NTP Version 3 requests are from the Authentication Manager, approximately every 300 seconds.  
• NTP Version 4 requests are from the SuSE Linux Operating System , approximately every 1024 seconds.

 If the alert is only seen occasionally, especially if only a single NTP server has been configured, it can usually be ignored. It is useful to also configure a second NTP server, to reduce the frequency of these alerts.  If it is still seen frequently after configuring a second NTP Server, then additional investigation may be required.

 

Task

If this is not sufficient to indicate the issue, there are various files that can be used to check for NTP events and these include:

• /var/log/messages
• /var/log/ntp  (this is written with UTC timestamps)
• Authentication Manager System Log Report

It is also possible to collect simultaneous tcpdump packet captures on the SecurID Appliance and the NTP Server(s). These can then be analyzed for when the NTP events happen, and determine if the issue is with the SecurID Appliance, the NTP Server, or the network.

The things to check:

1. Did the SecurID Appliance generate a NTP Request, and is it NTP Version 3 or Version 4 ?
2. Did the NTP Request get to the NTP Server ?    
3. Did the NTP Server send a valid response to the NTP request?    
4. Did the response get back to the SecurID Appliance?  

Instructions for making a packet capture on the SecurID Appliance are below. Please contact your NTP server vendor for instructions on doing a packet capture on your NTP Server. 

The SecurID Appliance 8.x includes the tcpdump utility in the /usr/sbin directory, and you need to be root to use it.  Typically it will be used by SSH, but you can also use the local console.  

If SSH is not enabled, log onto the Operations Console, go to Administration > Operating System Access, put a check in Enable SSH, Save. 

Login with rsaadmin  and the Operating System password.
sudo su       (it will ask for a password again, supply the operating system password again)
cd  /usr/sbin
 
When you are ready to run the Packet capture, some examples of running tcpdump are below (note the -Z is capitalized)

To capture all traffic to a NTP server at 192.168.1.10, and save it to a file  in /tmp named cap1.cap:  
./tcpdump -i eth0 -s 1514 -Z root host 192.168.1.10 -w /tmp/cap1.cap

To capture all traffic on the NTP port 123 and save it to a file:  
./tcpdump -i eth0 -s 1514 -Z root port 123 -w /tmp/cap1.cap  

Once the error happens, stop the capture using control-C . Copy other related files to assist troubleshoot NTP
cp  /var/log/messages  /tmp
cp  /var/log/ntp   /tmp

Open the files' permissions to allow access with the command chmod  777  /tmp/* and get the capture and logs using any convenient method, such as WinSCP .  

Notes

Examples of NTP-related events

System Log Report

2014-08-26 20:36:43,WARN,16350,Critical System Event Notification,System encountered a critical event.,Warning,Unknown Warning,SYSTEM,,,,,ATTEMPT_WARN,hostname.company.com,,10.20.30.40,cation.impl.CriticalNotificationAdministrationImpl,Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.,,,,,,,,

Also, look for other issues around the time frame of the NTP failure.  

Messages file

Aug 26 20:36:28 rsa2 sudo: rsaadmin : TTY=unknown ; PWD=/opt/rsa/am/server ; USER=root ; COMMAND=/opt/rsa/am/utils/bin/appliance/queryTimeSettings.sh  

/var/log/ntp    (timestamps are in UTC) 

27 Aug 06:37:14 ntpd[7130]: no servers reachable
27 Aug 07:45:28 ntpd[7130]: synchronized to 192.168.1.10, stratum 3