Lee Kirkpatrick

Shannon. Have you seen my Entropy?

Blog Post created by Lee Kirkpatrick Employee on Sep 26, 2016

Entropy is a term I am sure most of us are familiar with. In layman’s terms, it refers to randomness and uncertainty of data; it is in this randomness that we can detect potential malicious traffic.


A gentleman named George Zipf lead the way in the study of character frequency in the early 1930’s, his work was further expanded upon by Claude Shannon to examine the entropy of language. These two forms of analysis have become engraved in the computer security domain and often used for cryptography – but what if we used their ideas to help detect malicious traffic?


Some malicious actors utilise domain generation algorithms (DGA) to produce pseudo random domain names they will utilise for their C2 communications. If we apply Shannon’s Entropy to these domains, we can calculate a score of their entropy and possibly identify these maliciously formed domains from the norm:-



Using the RSA Event Stream Analysis (ESA) component and a customised Java based Shannon Calculator, we can generate these entropy scores on the fly for any given metadata, and should they exceed a score greater than X, create an alert.


NOTE: Java plugins can be added to the ESA component as described by Nikolay Klender in his post - Extending ESA rules with custom function.


Once the Java plugin is implemented, we can then create our ESA correlation rule to utilise the new plugin available and calculate the entropy. In this example, we will use the plugin to calculate entropy for DNS domains using the following EPL:-



SELECT * FROM Event(service = 53 AND calcEntropy(alias_host)>4);


The entropy value for this is set to anything greater than ‘4’ but can be edited dependent upon what results are observed.


I have attached the java used for calculating Shannon's Entropy should anyone be interested.


DISCLAIMER: This is by no means a full proof detection method for malicious traffic. The information is here to show the capabilities of the product and avenues of exploration to help thwart the adversary. This content is provided as-is with no RSA direct support, use it at your own risk. Additionally, you should always confirm architecture state before running content that could impact the performance of your SA architecture.