How to troubleshoot NTP error Critical Event Notification - Not able to sync time error with RSA Authentication Manager 8.0 and 8.1

[#1]

Article Number

000029292

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.0, 8.1

Issue

Authentication Manager 8.1 Patch 2 included a fix to prevent startup problems when there was an issue with NTP. As part of this patch, additional alerts were added to the system, to alert when a NTP error occurs:
 

Attention! The following critical system event occurred: Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.

System Time Synchronization Configuration Check,"Checking configuration for System Time Synchronization.
Warning,All NTP Servers are unavailble - potential for significant system time drift,SYSTEM,,,,,ALL_NTP_SERVERS_UNVAILABLE

Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.,,,,,,,,
 

Both the Authentication Manager 8.1 application and the SUSE Linux OS try to do a NTP synchronization several times an hour. The appliance sends both NTP Version 3 and NTP Version 4 requests.   

NTP V3 requests are from the Authentication Manager application, approximately every 300 seconds.  NTP V4 requests are from the SUSE Linux OS, approximately every 1024 seconds.

If the alert is only seen occasionally, especially if only a single NTP server has been configured, it can usually be ignored. It is useful to also configure a second NTP server, to reduce the frequency of these alerts.  If it is still seen frequently after configuring a second NTP Server, then additional investigation may be required.

Task

If this is not sufficient to indicate the issue, there are various files that can be used to check for NTP events. These include:

• /var/log/messages
• /var/log/ntp  (this is written with UTC timestamps)
• Authentication Manager System Log Report

It is also possible to collect simultaneous tcpdump packet captures on the Authentication Manager server and on the NTP server(s). These can then be analyzed  for when the NTP events happen and determine if the issue is with the Authentication Manager server, the NTP server or the network. The things to check:

1. Did the Authentication Manager server generate a NTP Request, and is it NTP V3 or V4 ?
2. Did the NTP request get to the NTP Server ?    
3. Did the NTP server send a valid response to the NTP request?    
4. Did the response get back to the Authentication Manager server?  


Instructions for making a packet capture on the RSA Appliance are below, please contact your NTP server vendor for instructions on doing a packet capture on your NTP Server.  The RSA Appliance 8.x includes the tcpdump utility in the /usr/sbin directory , and you need to be root to use it.  Typically it will be used by SSH, but you can also use the Linux console.  

If SSH is not enabled, log onto the Operations Console, go to Administration > Operating System Access, put a check in Enable SSH, Save.   and enable SSH.
Login with rsaadmin  and the Operating System Password.
sudo su       (it will ask for a password again, supply the operating system password again)
cd  /usr/sbin
 
When you are ready to run the Packet capture, some examples of running tcpdump are below (note the -Z is capitalized)

To capture all traffic to a NTP server at 192.168.1.10, and save it to a file  in /tmp named cap1.cap:  
./tcpdump -i eth0 -s 1514 -Z root host 192.168.1.10 -w /tmp/cap1.cap

To capture all traffic on the NTP port 123 and save it to a file:  
./tcpdump -i eth0 -s 1514 -Z root port 123 -w /tmp/cap1.cap  

Once the error happens, stop the capture using control-C . Get the other files to troubleshoot NTP
cp  /var/log/messages  /tmp
cp  /var/log/ntp   /tmp

Open the files' permissions to allow access
chmod  777  /tmp/*  
and get the capture and logs using any convenient method, such as WinSCP .  

Contact the NTP Server's vendor for instructions on getting the NTP Server's capture.

Resolution

Logs to check during the event:

• System Log Report

2014-08-26 20:36:43,WARN,16350,Critical System Event Notification,System encountered a critical event.,Warning,Unknown Warning,SYSTEM,,,,,ATTEMPT_WARN,hostname.company.com,,10.20.30.40,cation.impl.CriticalNotificationAdministrationImpl,Not able to sync time. Either the NTP service is not running or unable to sync time from the NTP server.,,,,,,,,

Also look for other issues around the time of the NTP failure  

• Messages file

Aug 26 20:36:28 rsa2 sudo: rsaadmin : TTY=unknown ; PWD=/opt/rsa/am/server ; USER=root ; COMMAND=/opt/rsa/am/utils/bin/appliance/queryTimeSettings.sh  

• /var/log/ntp 

Timestamps are in UTC  

27 Aug 06:37:14 ntpd[7130]: no servers reachable
27 Aug 07:45:28 ntpd[7130]: synchronized to 192.168.1.1, stratum 3