Decoder: Create Custom Meta Keys Using Custom Feed

This topic provides information on how to add custom meta keys, using custom feed in the Log Decoder.

You can create custom meta keys to retrieve data, to investigate and analyze the logs and packets. Custom meta keys enable you to add an enrichment context for the log and packet data. This document highlights the configuration changes to reflect the custom meta keys in the Concentrator, ESA, Archiver, Warehouse Connector, and Reporting Engine schema.

Here is a example of creating the custom meta key in the Log Decoder. In this scenario, an organization wants to track the location of an asset such as a printer. So, a custom meta key source location is introduced which indicates the location of the asset, for example the Printer1, which is located in the 'Fifth Floor A wing'. 

Note: Custom meta keys can be created in Decoder as well. Make sure to select the index.decoder.xml file when you create a custom meta in the Decoder.

Procedure

Add custom meta key in Log Decoder

To add custom meta keys using custom feed:

1. In the Security Analytics menu, select Administration > Services > Log Decoder.
2. Select a service and click > View > Config > Files tab > index-logdecoder-custom.xml.

<Language>
 <?xml version="1.0" encoding="utf-8"?>
 <Language level="IndexNone" defaultAction="Auto">
 <!-- Reserved Meta key for Feed -->
 <Key description="Source Location" level="IndexNone" name="location.src" format="Text"/>
</Language>

3. Restart the Log Decoder service. In the Services view, click  > Restart.

Deploy feed in Live

To deploy the feed in the live environment:

1. In the Security Analytics menu, select Live > Feed.
2. In the toolbar, click .
   The Setup Feed dialog is displayed.

To select the feed type, click Custom Feed and Next.
The Configure a Custom Feed wizard is displayed, with the Define Feed form open.
Enter the name and upload the Feed CSV file.

Note: For a STIX feed you must upload the .xml file.


3. Click Next.
4. Select the Log Decoder service, where the feed needs to be uploaded.
5. In the Define Index section, select the index type, index column, and callback key. In the Define Values section, enter the custom meta key.
   The contents of the .csv file are displayed in the feed wizard. In this case, the first column displays the asset hostname and the second column indicates the asset location.

Note: The Source IP should be indexed by selecting the type as 'IP' as the ip.src. and ip.dst are in IPv4 format. 

In this scenario, a custom meta key location.src (location source) is added by indexing the hostname (alias.host). In this example, the printer hostname are populated in meta key 'alias.host'. So, select 'alias.host' as callback key, and index type as 'Non IP' in the Feed Wizard as shown below. In the Define Values section, select the custom meta key from the drop down menu.

7. Click Next.
8. Click Done.

For more information on the feed wizard, see Create and Deploy Custom Feed Using Wizard.

Add the custom meta entry in Concentrator index file

To add the custom meta entry in the concentrator index file:

1. In the Security Analytics menu, select Administration > Services > Concentrator.
2. Click  > View > Config > Files tab > index-concentrator-custom.xml.
3. Add the custom meta entry in the Concentrator index file.

 <Language>
  <?xml version="1.0" encoding="utf-8"?>
  <Language level="IndexNone" defaultAction="Auto">
  <!-- Reserved Meta key for Feed -->
  <Key description="Source Location"  level="IndexValues" name="location.src" format="Text"                 valueMax="10000" defaultAction="Open"/>
 </Language>

4. Restart the Concentrator services. In the Services view, click  > Restart.

Note: In case of the Broker, the Broker derives its index from the Concentrator from which it aggregates. So you do not need to create custom meta in the broker. If you have not indexed the meta key in the concentrator, the broker will not display in the investigation.

Investigate 

Note: Make sure that you logout and login from the Security Analytics User Interface, before you can view the custom meta key in Investigation.

To investigate on the custom meta key: 

1. In the Security Analytics menu, select Investigation > Navigate.
2. Select a Concentrator service.
3. Click Navigate. 
   
   

Here is an example of a report executed on the concentrator.

Additional Procedures

The following procedures must be executed if you have Warehouse Connector, Archiver, Reporting Engine and ESA configured.

Update the Schema in ESA 

Before you update the schema in ESA, the custom meta key should be indexed in the concentrator.

To update the schema ESA rules and to be able to use the new custom meta keys:

1. In the Security Analytics menu, select Administration > Services > ESA- Event Stream Analysis > View > Config.
2. Edit the Concentrator Datasource.
3. Click Test Connection.

4. Click Save after the connection is successful.
5. Click Apply.
6. Navigate to Alerts > Configure > Settings.

7. Click the Search tab and search for the name of the custom meta key.
   The custom meta key name and type is displayed.

Update the Schema in Archiver

If you want to configure the Security Analytics Archiver, using the new custom meta keys, you need to update the Archiver schema in the Reporting Engine.

To update the Archiver schema in Reporting Engine:

1. In the Security Analytics menu, select Administration > Services > Archiver.
2. Click on > View > Config > Files > index-archiver-custom.xml.
3. Add the custom meta entry in the Archiver index file.

<Language>
 <?xml version="1.0" encoding="utf-8"?> 
 <Language level="IndexNone" defaultAction="Auto">
 <!-- Reserved Meta key for Feed -->
 <Key description="Source Location" level="IndexValues" name="location.src" format="Text"
 valueMax="10000" defaultAction="Open"/>
</Language>

4. Restart the Archiver service. Click on  > Restart.
   The Archiver schema gets updated with the custom meta key.

Update the Schema in Warehouse Connector

If you want to configure the Security Analytics Warehouse with custom meta and use it in warehouse report then you need to update the Warehouse schema in the Reporting Engine.

If the Log Decoder or Decoder, where the custom meta key is added, is one of the sources in the Warehouse Connector stream, you need to update the schema in the Warehouse Connector.

To update the Warehouse schema in the Reporting Engine:

1. In the Security Analytics menu, select Administration > Services > Warehouse Connector.
2. Click on  > View > Config > Files tab > index-logdecoder-custom.xml.
3. Select the stream and click Reload.
   The warehouse connector pulls the schema from the downstream devices (log decoder/decoder).

For more information on streams, see the Configure Streams topic in the Warehouse Connector Configuration Guide.

Update the Schema in Reporting Engine

To update the schema in Reporting Engine:

1. In the Security Analytics menu, select Administration> Services > Reporting Engine.
2. Click on > Restart.

Note: Restart the Reporting Engine or wait for thirty minutes for the schema to be updated.

To view the custom meta key:

1. Navigate to Reports > Rules.
2. In the toolbar, click .
3. Select Warehouse DB.
4. In the Build Rule page, search for the custom meta from the right panel of the page.
   The custom meta key is displayed.