000030432 - Security Analytics and Netwitness: Important information regarding the upcoming Leap Second on June 30th 2015

Document created by RSA Customer Support Employee on Jun 14, 2016
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000030432
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server, Event Stream Analysis, Archiver, Broker, Concentrator, Decoder (Packet/Log), Hybrid (Packet/Log), All-In-One (Packet/Log), Malware/Spectrum, Remote Log Collector, Data Warehouse 
RSA Version/Condition: 9.X and 10.X (all supported versions)
Platform: CentOS
O/S Version: EL5, EL6

The next leap second clock adjustment is scheduled for June 30th, 2015 at 23:59:60 UTC. In order to insure RSA Security Analytics Appliances are ready for the leap second event, RSA recommends customers review this article in its entirety for further clarification.

CauseLeap seconds are periodic one-second adjustments to Coordinated Universal Time (UTC) which maintains a system's time of day as close as possible to actual solar time. Due to the Earths rotation, atomic clocks can drift from actual solar time in response to climatic and geological events. Therefore, a leap second is periodically added to accommodate the Earths rotation. 
Resolution1.  NTP:
Insure that NTP is configured and functioning properly on all of your Security Analytics appliances.  To verify that ntp is running, check the system time on each Security Analytics appliance by running the "date" command.  Insure that the date and time reported is accurate.  Also insure that ntpd is running on all of your appliances using the services ntp status command, example:


# service ntpd status
ntpd (pid  2218) is running...


NTP servers are specified in /etc/ntp.conf via the "server" directive, examples:
server myntpserver.mydomain.com
Up to 3 ntp servers may be specified on separate server lines, noting that both hostnames or IP addresses are valid in the server directive.  However when specifying hostnames, the names must be resolvable (typically via DNS). For customer experiencing problems with NTP or hostname resolution, please contact RSA Customer Support for assistance.

2. TZData update:
A) RSA has tested Security Analytics 10.X and 9.X functionality with and without installing tzdata update.  While no significant issues were observed without the update, RedHat has recommended tzdata-2015 be updated based on the following article: https://access.redhat.com/articles/15145.  To check the version of TZdata on your host, use the rpm command as demonstrated in the example below (tzdata version in red, centos version in blue, and this sample from a host running Cent/OS 6:

[root@saserver doc]# rpm -qa | grep tzdata-

B) The tzdata updates may be picked up here:

For Cent/OS 5: use tzdata-2015a-1.el5, labled "EL5 Time Zone package".

For Cent/OS 6: use tzdata-2015a-1.el6, labled "EL6 Time Zone Package".

C) To install the TZdata update:  Download the appropriate tzdata update from SCOL (EL5 or EL6), then apply the update using one of the 2 options described below:

Option1: From the SA UI (10.4.X ONLY)

1. Login to the Security Analytics UI with an administrative account

2. Navigate to Administration -> System -> Updates -> Manual Updates

3. Upload the new tzdata rpm, then click the "Apply" key.

*4. From Administration -> Appliances page, select each component, perform "Check Update".

*5. The Updates column should display that the update is now available for the appliance.

*6. Confirm that the update corresponds to tzdata, then apply the update
*NOTE: Once the tzdata rpm is added to the repository, steps 4, 5 & 6 can be replaced with the yum method mention below to update the single tzdata package:

yum update tzdata

Option 2 – From the command line (This method must be used on appliances running SA 10.3x and lower, on Netwitness 9.X, and may also be used on 10.4.X)

1. Copy the tzdata rpm onto each appliance /root/ dir

2. SSH to each appliance and navigate to /root/

3. Run the command: 

rpm –Fvh  <tzdata update file>.rpm

4. Reboot the system after applying the tzdata update using init 6

D) Prior to 10.4, collectd was not in use on Security Analytics systems.  At the time of leap second insertion without patching tzdata, collectd errors may appear in the /var/log/messages  file on the Security Analytics server similar to the example below.

Apr 19 23:59:50 SA-Server collectd[2472]: rrdtool plugin: rrd_update_r (/var/lib/netwitness/collectd/rrd/8a926cb7-943e-4c68-bff8-330098ba0eb6/appliance_df-var_netwitness/percent-used.rrd) failed: /var/lib/netwitness/collectd/rrd/8a926cb7-943e-4c68-bff8-330098ba0eb6/appliance_df-var_netwitness/percent-used.rrd: illegal attempt to update using time 1429487990 when last update time is 1429487990 (minimum one second step).

While these errors are typically non intrusive and subside over time, it is technically possible that the Reporting Engine CPU utilization may increase after the leap second insertion depending on individual system configuration.  While this was not observed during RSA leap second testing, for customers that are using ntp (see bullet #1)  but unable to patch tzdata, the following may alternately be considered:

          1. Record and monitor cpu utilization from the unix “top” command prior to

              and after the leap second event.  

          2. Should an abnormal cpu spike be observed after the event, a system reboot will

               clear the problem. 

          3. To reboot the system, execute init 6 from the command line of the unit as root.