000031300 - How to monitor if a meta index key is full in the RSA NetWitness Platform

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support on Jun 15, 2018
Version 7Show Document
  • View in full screen mode

Article Content

Article Number000031300
Applies ToRSA Product Set: Security Analytics, NetWitness Logs & Network
RSA Product/Service Type: Concentrator, Log Decoder
Platform: CentOS
IssueEach meta key that is indexed in the RSA NetWitness Platform has a valueMax value associated with it. This is the maximum number of unique values that can be stored in the index for this meta key.

This is defined in the index-concentrator.xml and index-concentrator-custom.xml values on a Security Analytics concentrator.

For example:

<key description="Hostname Aliases" level="IndexValues" name="alias.host" format="Text" valueMax="2500000" />

This shows that the alias.host meta key can contain up to 2500000 unique values in an index slice.

If this value is exceeded, then the additional unique values will not be able to be seen when investigating, although it will still be recorded in the session information.

It is therefore important to be alerted if the index for the meta key is full.

In some cases the number of unique values for a key may exceed any setting of ValueMax that is used. For example, if you were to index URLs seen by the system then the index for this meta key would become quickly full due to the large number of unique possible values. For source ports and destination ports for a TCP session then there is a maximum of 65536 possible values so the valueMax is set to this value.

When a meta key is full the following will be seen in /var/log/messages on the concentrator:

Sep 18 16:40:20 logconc nw[11922]: [Index] [warning] Index key alias.host has reached max capacity of 2500000 values and will ignore new values for this slice.
TasksWe wish to be alerted if a metakey becomes full so that we can plan accordingly. To do this

1. Install the RSA Security Analytics Parsers from RSA Link here:  https://community.rsa.com/docs/DOC-28866
2. Add the following message to the RSA Security Analytics parser above the </DEVICEMESSAGES> tag.

content="&lt;@event_description:Index Key Full&gt;[warning] Index key &lt;metakey&gt; has reached max capacity of &lt;fld2&gt; values and will ignore new values for this slice."/>

3. Add the following meta key to your concentrators in the  /etc/netwitness/ng/index-concentrator-custom.xml

<key description="Meta Keys" level="IndexValues" name="metakey" format="Text" valueMax="1000" />

4. Add the following to your Log Decoder at /etc/netwitness/ng/envision/etc/table-map-custom.xml

<mapping envisionName="metakey" nwName="metakey" flags="None" format="Text"/>

5. When the Index Key becomes full the event description "Index Key Full" will become populated.
User-added image
ResolutionIf an index key does become full then there are several options. 
  • Create more index slices
  • Increase the valueMax value for the IndexKey
  • Switch off indexing for the key
  • Do nothing - it may not be practical to capture all unique values for the key or it may be that all possible unique values have been captured.
The best option will depend on your environment and what you wish to achieve, so please contact RSA Customer Support if further advice is required.

A script is attached that can be run as a cronjob to monitor the index keys across your environment. The script was written by Davide Veneziano and modified by Maxim Siyazov.

The improved version 1.2 is attached to this article.
Usage: ./index-profile.pl concentrator_ip [out_file_name]
  • The concentrator IP can be passed as a command line argument.
  • Added threshold parameter (variable) to show up only keys reached the threshold.
  • Added the option to output in CSV format to take snapshots over the time using cron. This can help to work out an optimal ValueMax. Can be used to take snapshots of multiple concentrators from a single server. This assumes that the admin password is the same across all concentrators.
For example, to take an index profile snapshot every 30 min create a cron job for every concentrator as follows:

# Concentrators 1
0,30 * * * * /root/index-profile-1.2.pl <Concentrator1_IP> index-snap-1.csv >> index-snap-1.csv
# Concentrator 2
0,30 * * * * /root/index-profile-1.2.pl <Concentrator2_IP> index-snap-2.csv >> index-snap-2.csv

 This will generate a CSV file which then can be easily analysed, so the optimal values of ValueMax, save.session.count or the index save scheduler can be worked out.   

The sample output:

NotesAll screenshots and hostnames are from an internal lab system.