This topic describes how to use the custom index file to customize the index. Each Security Analytics Core service is installed with a default index configuration that is intended to cover the index needs for most users of the product. However, it is possible to index new meta keys in order to use the index with custom content that generated custom meta.
Index Configuration File Locations
The index customization is accomplished by making changes to the custom index file. The location of this file is /etc/netwitness/ng/index-<servicename>-custom.xml, where <servicename> corresponds to the name of the product that you are customizing. For example, the Concentrator custom index file is /etc/netwitness/ng/index-concentrator-custom.xml.
Concentrator products also include a file that describes the default index configuration: /etc/netwitness/ng/index-concentrator.xml. This file is useful as a template to show how the custom index file is formatted.
If you make customizations to the index in the custom index file, those customizations override any conflict with the default index configuration.
You can make changes to the custom index file while the service is running. When the service receives an index save command, the changes to the custom index file are read and applied to the index.
Note: Changes to the index can only be applied to new incoming data. Data cannot be retroactively reindexed with a new custom index configuration, except with a very time consuming reindex procedure.
Index configuration entries
The custom index file is an XML document. The root element of this document is the language element, and inside there is an elements per meta key to describe each custom index. Each element of the custom index configuration looks like this:
<key name="did" description="Decoder Source" level="IndexValues" format="Text" valueMax="100" />
Definitions for each attribute in this element: Attribute | Description -|- name | The name of the meta key that will be indexed description | A human-readable description for the meta type level | The type of index that will be created for this meta key valueMax | The maximum unique values that will be stored for this key per slice format | The format of the data held by all meta items with this meta key name.
The next few sections examine these parameters in greater detail.
The meta name used by the index refers to the meta key name present within every meta item in the meta database. These meta names are generated by the Decoders when parsing. Parsers can choose to generate meta with any meta key name. Therefore, the custom index allows you to choose which of the meta items generated by the Decoder are indexed.
Meta key names can be 16 characters long, and contain only letters or the '.' character.
When the Decoder generates meta items, it assigns a data type. Each parser can choose the data type of the meta it generates. However, there are recommended and standard data types for each of the default meta keys. For example, ip.src and ip.dst are stored as the IPv4 meta type, and alias.host is stored as the Text meta type. Each parser must agree on the data format for each meta key generated by the Decoder.
When adding a custom index to the Concentrator, the data type of the custom index must match the format of the data generated by the Decoder. If the types do not match, the Concentrator attempts conversions of the meta generated into the type specified for the custom index. However, these conversions sometimes fail, and the resulting index can produce undefined results.
Likewise, when many Decoders and Concentrators work together as part of a Security Analytics installation, they must all agree on the types for each meta key. Conflicts of meta types between Security Analytics Core services can lead to undefined behavior.
The following table shows the metadata types supported by the Security Analytics Core services.
|Type||Size in bytes||Description|
|Int8||1||Signed 8-bit integer|
|UInt8||1||Unsigned 8-bit integer|
|Int16||2||Signed 16-bit integer|
|UInt16||2||Unsigned 16-bit integer|
|Int32||4||Signed 32-bit integer|
|UInt32||4||Unsigned 32-bit integer|
|Int64||8||Signed 64-bit integer|
|UInt64||8||Unsigned 64-bit integer|
|UInt128||16||Unsigned 128-bit integer|
|Float32||4||32-bit floating point number, single precision|
|Float64||8||64-bit floating point number, double precision|
|TimeT||8||Unix epoc timestamp|
|Binary||1-255||Arbitrary binary data|
|Text||1-255||UTF-8 Encoded text data|
|IPv4||4||IPv4 address bytes|
|IPv6||16||IPv6 address bytes|
|MAC||6||MAC Address bytes|
When defining a custom index, it is important to use the best data type for the meta. For example, never store IP addresses as Text, since the Text representation takes more bytes than the IPv4 representation.
There are three levels, or types, of indexing: IndexNone, IndexKeys, and IndexValues.
This type of custom index is not really an index at all. Custom index entries with the IndexNone level exist only to define and document the meta key. IndexNone entries can be used in custom Decoder indices to enforce a specific data type for a meta key across all the parsers on a Decoder.
This type of custom index indicates that the index only keeps track of sessions that contain meta items with this meta key name. However, it does not index any unique values in the meta database for the meta key.
Key-level indices take much less storage space, memory, and CPU time to manage, but they require a lot more work from the query engine when you perform query or values operations using them.
If used in a where clause, a meta key indexed at the key level can only be used to resolve operations such as exists or !exists.
This type of custom index keeps sessions that contain each individual unique value for the meta key. This type of index is also known as a "full index".
This type of index is needed for efficient processing of most where clauses, and for use of this meta key as the fieldName parameter of a values call.
Value max is a parameter that can have a very significant impact on the accuracy and performance of a Value-level index.
As a Decoder parses packets or logs, it is allowed to create meta of any type with any value. Usually, these meta items are created from data copied directly out of the packet or log. Therefore, anyone can create unique meta values in response to nearly any event.
The performance of the index is directly dependent on the number of unique values it has found for each meta key. As the number of unique values increases, the rate at which new meta is indexed can decrease, and the speed with which queries are completed decreases. Since any person can influence the creation of unique meta values, it is possible for any person to affect the performance of the index.
The value max parameter limits the number of unique values that can enter the index. Therefore, a malicious user cannot flood the system with a large number of unique values in an attempt to make the Security Analytics system not work.
It is important to set a value max on any meta key that may have its value influenced directly by incoming packets or logs.
The value max applies only to values added since the last index save operation.
The limit for how high value max can be set varies from version to version and on the amount of RAM available to the Security Analytics Core service. As of 10.3, the recommended ceiling for value max is 5,000,000 for any meta key. If there are a lot of custom indices, then the value max may have to be lower.
The max length parameter is used exclusively on the word meta type. It must match the corresponding setting for /decoder/parsers/config/token.max.length on the Log Decoder service that is generating word token metas. The index uses the maxLength to properly interpret search terms fed into the msearch SDK function.