Skip navigation
All Places > Products > RSA NetWitness Platform > Blog > 2016 > September > 26

Entropy is a term I am sure most of us are familiar with. In layman’s terms, it refers to randomness and uncertainty of data; it is in this randomness that we can detect potential malicious traffic.


A gentleman named George Zipf lead the way in the study of character frequency in the early 1930’s, his work was further expanded upon by Claude Shannon to examine the entropy of language. These two forms of analysis have become engraved in the computer security domain and often used for cryptography – but what if we used their ideas to help detect malicious traffic?


Some malicious actors utilise domain generation algorithms (DGA) to produce pseudo random domain names they will utilise for their C2 communications. If we apply Shannon’s Entropy to these domains, we can calculate a score of their entropy and possibly identify these maliciously formed domains from the norm:-



Using the RSA Event Stream Analysis (ESA) component and a customised Java based Shannon Calculator, we can generate these entropy scores on the fly for any given metadata, and should they exceed a score greater than X, create an alert.


NOTE: Java plugins can be added to the ESA component as described by Nikolay Klender in his post - Extending ESA rules with custom function.


Once the Java plugin is implemented, we can then create our ESA correlation rule to utilise the new plugin available and calculate the entropy. In this example, we will use the plugin to calculate entropy for DNS domains using the following EPL:-



SELECT * FROM Event(service = 53 AND calcEntropy(alias_host)>4);


The entropy value for this is set to anything greater than ‘4’ but can be edited dependent upon what results are observed.


I have attached the java used for calculating Shannon's Entropy should anyone be interested.


DISCLAIMER: This is by no means a full proof detection method for malicious traffic. The information is here to show the capabilities of the product and avenues of exploration to help thwart the adversary. This content is provided as-is with no RSA direct support, use it at your own risk. Additionally, you should always confirm architecture state before running content that could impact the performance of your SA architecture. 

Michael Sconzo

Content Update

Posted by Michael Sconzo Employee Sep 26, 2016

We've got some nice new additions to Live as well as a high-impact update with our HTTP_lua parser.


First off we've expanded out detection capability and the following App Rules are now available via Live.


In addition the team has pushed an update to the HTTP_lua parser. If you're running this in your environment you want to make sure you get updated to this version. The high points of this release are:

  • 95% rewritten to better accommodate updates, performance and analysis.
  • Language extraction from Accept-Language request headers
  • Additional bug fixes 


Fun Fact: Did you know you could use a CSV file as a whitelist or blacklist in ESA?


Stay tuned for more!

In my previous post, Trend Analysis with the Netwitness Suite, I've presented an approach to develop a baseline and perform a trend analysis with ESA. As mentioned many times, every threat is different and detection techniques not only can, but must vary to effectively protect the businesses of our organizations.


There are situations in which threat patterns can be identified by simply reporting on new values of a given meta key, without the need of performing complicated statistical analysis. For example if a new browser or a new TLD never seen before shows up in our environment.


The Netwitness reporting engine has a very handy function called show_whats_new() which is doing the job for you. However, if you want to leverage the power of ESA to achieve the same, it will be more challenging since there is the need to work with large timeframes which must be handle with care within ESA.


By using the same approach detailed in my previous post, the attached EPL can safely look at the last 30 days of every meta key you want to monitor and alert once there is a new value. Events are aggregated every minute, hour and day so to limit the impacts on ESA performance and store in memory only the information required for achieving the use case.


Multiple meta keys can be monitored by replicating and customizing the last statement.


From an implementation standpoint, the model creates a history of meta key - value pairs which is checked on a daily basis to alert for each new value found. In order to setup a learning phase, the model internally stores also the current date so to prevent alerting until the warm up period is over.


Please note this is not RSA official/supported content so use it at your own risk!

Filter Blog

By date: By tag: