Decoder: Enable and Configure the Entropy Parser

Document created by RSA Information Design and Development on Sep 13, 2017Last modified by RSA Information Design and Development on Oct 11, 2017
Version 6Show Document
  • View in full screen mode
 

Beginning with NetWitness Suite 11.0, the administrator can configure a Decoder to use a NetWitness native parser, known as the Entropy parser. When the Entropy parser is enabled, analysts have visibility into channels that are trying to blend in with other traffic, but do not follow normal protocol behavior. This helps to identify channels that do not conform to the normal environment traffic baseline, and may, therefore, be worthy of investigation.

The parser creates meta keys, based on statistics collected by the native NetWitness Suite parser, that help to identify behavior of any channel that is getting lots of network traffic. When the parser is first enabled, the analyst needs to become familiar with overall behavior for the different channels seen in a captured session to understand the frequency of bytes and the normal client and server payload. Once the normal behavior is known, analysts can use the meta keys to find behavior that does not match the expected.

By default, the Entropy parser generates 10 additional meta keys that do not add significantly to the load on a Decoder, and are useful for this specialized case. The parser is disabled by default.

Enable indexing if you have interest in exploring interesting sessions based on payload byte analysis of the packets. By default, to make indexing easier, the normal Float32 value for entropy.req and entropy.res is multiplied by 10k and stored in a UInt16 (thus giving four digits of precision, 0 to 10,000).

However, if you define the entropy.* fields in the Decoder language to be Float32, the Decoder will store it as a float with a range of 0.0 to 1.0. Take care to change the language everywhere if you decide to keep it as a Float32.

RSA does not recommend indexing as a Float32 because of the high unique counts due to minute changes in precision.

These are the 10 new meta keys generated by the Entropy parser by default:

  • entropy.req and entropy.res: These meta keys capture entropy using the Shannon entropy equation, which has a floating point value as a result. The floating point value of 0 to 1.000 is multiplied by 10000 and written in NetWitness Suite Suite as UInt 16, an unsigned integer of 0 through 10000. .
  • mcb.req and mcb.res: The most common byte is simply which byte for each side (0 thru 255) was seen the most.
  • mcbc.req and mcbc.res: The most common byte count is the number of times the most common byte (above) was seen in the session streams.
  • ubc.req and ubc.res: - Unique byte count is the number of unique bytes seen in each stream. 256 would mean all byte values of 0 thru 255 were seen at least once.

  • payload.req and payload.res: The payload size metrics are the payload sizes of each session side at the time of parsing. However, in order to keep indexing from having high unique counts (bad for performance), the two payload size metas below are calculated this way:
    • Less than 1000 is the exact number of payload bytes.

    • 1000 or greater is bucketed in increments of 1000. So a size of 5826 would be stored as 5000.

To enable and configure the Entropy parser on a Decoder

  1. Log in to RSA NetWitness, and select ADMIN > Services in the NetWitness Suite menu.
  2. In the Services view, select the Decoder that you want to configure, and then View > Config.
    The Services Config view for the selected Decoder is displayed.
  3. The Entropy parser is disabled by default. Click the drop-down list under Config Value and select Enabled. If you want to disable some of the meta keys, click the drop-down list and select Disabled next to the meta key.

  4. Click Apply
    The Entropy parser is enabled and begins creating the new meta keys as configured in the Concentrator custom index file.
  5. Navigate to the Explore View for the Decoder, and select the decoder > parsers > config node. In the parsers.options, you can set the Entropy parser payload. The default value shown in the screen capture is Entropy = payload = 1000. When defining the value, the syntax is Entropy = payload = "1000" The quotes are required if there is white space in the value, and it is a good practice to always use them to avoid white space issues. If you want to see the exact payload, set this parameter to "1".

    The default Entropy payload is 1000, which means that if the payload count is less than 1000, the exact value is provided. If the payload count is greater than 1000, the value is rounded down to the nearest 1000. For example, a count of 3798 is rounded down to 3000.
  6. If you want to change the default Entropy payload rounding factor, edit the value. This change takes effect when the parser is reloaded

  7. In the Service Config view select the Concentrator that is aggregating traffic from this Decoder. Select View > Files and open the Custom Index file for the Concentrator. Look for the Entropy parser meta keys to see if they are included and uncommented.
    By default the keys are commented out and therefore not enabled. To enable that part of the language the administrator needs to copy that part of index file into the index-concentrator-custom.xml and uncomment the key description line for each meta key. An example of the custom index file with the Entropy parser keys and instructions is shown below.
  8. With the Entropy meta keys enabled, they are available to analysts in Investigate, but hidden by default. To make the meta keys visible in the Investigate Values view, edit the default meta keys in the Default Meta Keys dialog so that they are open instead of hidden. You can manage these meta key the same way you manage other meta keys.
    This is an example of the Manage Default Meta Keys dialog.

 

Entropy Parser Configuration in the Concentrator Custom Index File

The following is an excerpt of the Concentrator Index file lines that the administrator must copy to the custom index file. The comments provide guidance on configuring the parser.

<!-- This section is commented out because it's only used by the Entropy parser which is disabled by default. To enable this part of the language, copy to index-concentrator-custom.xml and uncomment the keys. HOWEVER, take note that depending on how the Entropy parser is configured, the entropy.req and entropy.res format might be a Float32 instead of a UInt16. So make sure to change to the correct type if necessary.-->

<!-- Entropy parser meta - enable indexing if you have interest in exploring this for interesting sessions based on payload byte analysis of the packets. By default, to make indexing easier, the normal Float32 value for entropy.res and entropy.req is multiplied by 10k and stored in a UInt16 (thus giving 4 digits of precision, 0 to 10,000). However, if you define the *.entropy fields in the Decoder language to be Float32, it will store it as a float with a range of 0.0 to 1.0. Take care to change the language everywhere if you decide to keep it as a Float32. We also don't recommend indexing as a Float32.-->

<!--

<key description="Client Entropy" format="UInt16" level="IndexNone" name="entropy.req" valueMax="10001"/>

<key description="Server Entropy" format="UInt16" level="IndexNone" name="entropy.res" valueMax="10001"/>

-->

<!-- The most common byte is simply which byte for each side (0 thru 255) was seen the most -->

<!--

<key description="Client Most Common Byte" format="UInt8" level="IndexNone" name="mcb.req"/>

<key description="Server Most Common Byte" format="UInt8" level="IndexNone" name="mcb.res"/>

-->

<!-- The most frequest byte count is the number of times the most common byte was seen in the session streams -->

<!--

<key description="Client Most Frequent Byte Count" format="UInt32" level="IndexNone" name="mcbc.req" valueMax="500000"/>

<key description="Server Most Frequent Byte Count" format="UInt32" level="IndexNone" name="mcbc.res" valueMax="500000"/>

-->

<!-- Unique byte count is the number of unique bytes seen in each stream. 256 would mean all byte values of 0 thru 255 were seen at least once -->

<!--

<key description="Client Unique Bytes" format="UInt16" level="IndexNone" name="ubc.req"/>

<key description="Server Unique Bytes" format="UInt16" level="IndexNone" name="ubc.res"/>

-->

<!-- The payload size metrics are the payload sizes of each session side at the time of parsing. However, in order to keep

indexing from having high unique counts (bad for performance), the two payload size meta values below are calculated like so:

Less than 1000 is the exact number of payload bytes

1000 or greater is bucketed in increments of 1000. So a size of 5826 would be stored as 5000. -->

<!--

<key description="Client Payload Size" format="UInt32" level="IndexNone" name="payload.req" valueMax="500000"/>

<key description="Server Payload Size" format="UInt32" level="IndexNone" name="payload.res" valueMax="500000"/>

-->

 
Previous Topic:Use Custom Parsers
You are here
Table of Contents > Configure Feeds and Parsers > Enable and Configure the Entropy Parser

Attachments

    Outcomes