Alerting: Configure In-Memory Table as Enrichment Source

Document created by RSA Information Design and Development on Sep 12, 2017Last modified by RSA Information Design and Development on Oct 10, 2017
Version 5Show Document
  • View in full screen mode
 

This topic provides instructions on how to configure an in-memory table. When you configure an in-memory table, you upload a .CSV file as an input to the table. You can associate this table with a rule as an enrichment source. When the associated rule generates an alert, ESA will enrich the alert with relevant information from the in-memory table.

For example, a rule could be configured to detect when a user tries to download freeware and to identify the person by user ID in the alert. The alert could be enriched with additional information from an in-memory table that contains details such as full name, title, office location and employee number.

An in-memory table is ideal for handling lightweight data. It is easy to set up and requires less maintenance than a database. For example, the AllTech Company is a small organization so the system administrator can maintain employee information in a .CSV file. If AllTech grows into a very large company, the administrator would have to configure an external database reference as an enrichment and associate the database with a rule.

Prerequisites

The column name in the .CSV file cannot have whitespace characters.

For example Last_Name is correct, and Last Name is incorrect.

The .CSV file must begin with a header line that defines fields and types.
For example, address string would define the header field as address, and the type as string.

The following shows a valid .CSV file represented as a .CSV and as a table.

Valid .CSV file

Procedures

Configure an Ad hoc In-Memory Table

  1. Go to CONFIGURE > ESA Rules.
    The Configure view is displayed with the ESA Rules tab open.
  2. Click the Settings tab.
  3. In the options panel, select Enrichment Sources.
    Enrichment Sources
  4. In the Enrichment Sources section, click Add List icon  > In-Memory Table.
    Adhoc In-Memory Table
  5. Describe the in-memory table:
    1. Select Ad hoc.
    2. By default, Enable is selected. When you add the in-memory table to a rule, alerts will be enriched with data from it.
      If you add an in-memory table to a rule but do not want alerts to be enriched, deselect the checkbox.
    3. In the User-Defined Table Name field, type a name, such as Student Information, for the in-memory table configuration.
    4. If you want to explain what the enrichment adds to an alert, type a Description such as:
      When an alert is grouped by Rollno, this enrichment adds student information, such as name and marks. 
  6. In the Import Data field, select the .CSV file that will feed data to the in-memory table.
  7. If you want to write an EPL query to define an advanced in-memory table configuration, select Expert Mode.
    The Table Columns are replaced by a Query field.
  8. In the Table Columns section, click Add icon to add columns to the in-memory table. 
  9. If a valid file is selected in the Import Data field, the columns populate automatically.

Note: If you selected Expert mode, a Query field is displayed instead of Table Columns.

  1. In the Key drop-down menu, select the field to use as the default key to join incoming events with the in-memory table when using a CSV-based in-memory table as an enrichment. By default, the first column is selected.  You can also later modify the key when you open the in-memory table in enrichment sources.
  2. In Max Rows drop-down menu, select the number of maximum number of rows that can reside in the in-memory table at a particular instance.
  3. Select Persist to preserve the in-memory table on disk when the ESA service stops and to re-populate the table when the service restarts.
  4. In Stored File Format field, do one of the following:
    • Select Object, if you want to store the file in a binary format.
    • Select JSON, if you want to store the file in a text format.
      By default, Object is selected.
  5. Click Save.
    The adhoc in-memory table is configured. You can add it to rule as an enrichment or part of the rule condition. See Add an Enrichment to a Rule.

When you add an in-memory table, you can add it to a rule as an enrichment or as a part of the rule condition. For example, the following rule uses an in-memory table as a part of the rule condition to create a whitelist, and it also uses an in-memory table of details in the user_dst file to enrich the alert that is displayed. 

The rule shows the in-memory table as a whitelist rule condition:

In-rule enrichment

Next, the alert is enriched with the User_list in-memory table:

Post-alert Enrichment

Therefore, the user_dst in-memory table is used to create a whitelist, and it is also used to enrich the data in the alert if the alert is triggered. 

Add a Recurring in-Memory Table

  1. Go to CONFIGURE > ESA Rules.
    The Configure view is displayed with the ESA Rules tab open.
  2. Click the Settings tab.
  3. In the options panel, select Enrichment Sources.
  4. Click Add List icon  > In-Memory Table.
  5. Describe the in-memory table:
    1. Click Recurring.
    2. By default, Enable is selected. When you add the in-memory table to a rule, alerts will be enriched with data from it.
      If you add an in-memory table to a rule but do not want alerts to be enriched, deselect the checkbox.
    3. In the User-Defined Table Name field, type a name, such as Student Information, for the in-memory table configuration.
    4. If you want to explain what the enrichment adds to an alert, type a Description such as:
      When an alert is grouped by Rollno, this enrichment adds student information, such as name and marks.
  6. Type the URL of the .CSV file that will feed data to the in-memory table. Click Verify to validate the link and populate the columns in the .CSV file.  You can add or remove columns using the plus or minus button. 
  7. If the server is configured behind another server, select Use Proxy.
  8. If the server requires logon credentials, select Authenticated
  9. For Recur Every, indicate how frequently ESA must check for the most recent .CSV:
    1. Select Minute(s), Hour(s), Day(s), or Week.
    2. If you select Week, select a day of the week. 
    3. Click Date Range to select a Start Date and End Date for the recurring schedule.
      Date Range
  10. In the Key drop-down menu, select the field to use as the default key to join incoming events with the in-memory table when using a CSV-based in-memory table as an enrichment. By default, the first column is selected.  You can also later modify the key when you open the in-memory table in enrichment sources.
  11. In Max Rows drop-down menu, select the number of rows that can reside in the in-memory table at a particular instance.
  12. Select Persist to preserve the in-memory table on disk when the ESA service stops and to re-populate the table when the service restarts.
  13. In Stored File Format field, do one of the following:
    • Select Object, if you want to store the file in a binary format.
    • Select JSON, if you want to store the file in a text format.
      By default, Object is selected.
  14. Click Save.
    The recurring in-memory table is configured. You can add it to rule as an enrichment or part of the rule condition. See Add an Enrichment to a Rule.

Configuring an Esper Query as an Enrichment Source

When you use "expert mode," you can create an enrichment source or named window based on an Esper query. This allows you to have more control over the content and create more dynamic content. When you do this, an EPL query constructs the named window to capture interesting state from event stream.

Workflow

The following shows the workflow for creating a query using a named window:

  1. The event is sent to the Esper Engine.
  2. An EPL query is generated.
  3. An alert is triggered.
  4. The query checks to see if there is a connection between the event and the Named Window.
  5. If there is a connection, the query that populates the Named Window is run and populated.
  6. The content from the Named Window is added to the alert content and sent or displayed (depending on your settings).

Esper Query workflow

Prerequisites

  • The meta used in the EPL statement must exist in the data.
  • You must create well-formed EPL statements.

Procedure

Configure an In-Memory Table Using an EPL Query

  1. Go to CONFIGURE > ESA Rules.
    The Configure view is displayed with the Rules tab open.
  2. Click the Settings tab.
  3. In the options panel, select Enrichment Sources.
  4. In the Enrichment Sources section, click Add List icon  > In-Memory Table.
    In-Memory Table - Enrich Advanced Query
  5. Select Adhoc.
    By default, Enable is selected. When you add the in-memory table to a rule, alerts will be enriched with data from it.
  6. In the User-Defined Table Name field, type a descriptive name to describe the in-memory table.
  7. If you want to explain what the enrichment adds to an alert, enter information in the Description field.
    This description displays when you view the list of enrichments from the Enrichment Sources view, so it's a good idea to enter a thorough description as a best practice. Doing this allows other users to understand the content of the enrichment without opening it to examine its contents.
  8. Select Expert Mode to define an advanced in-memory table configuration by writing an EPL query.
    The Table Columns are replaced by a Query field.
  9. Select Persist to preserve the in-memory table on disk when the ESA service stops and to re-populate the table when the service restarts.
  10. Enter the EPL query in the Query field. The query should be well-formed, and it's a good idea to test it before entering it in the field.
  11. Click Save.

Example

For example, you created a rule that searches for five failed attempted logins followed by a successful login. When that rule is triggered, you may want the notification to contain information about the last user logged into the system when this successful login occurred. To add this enrichment to the notification, you might choose to create a stream-based in-memory lookup table that is populated from incoming events to maintain a mapping of IP addresses to the last user logged in from that address. To do this, you create an enrichment using a query as your source.

Step 1: Create Your Rule

First, you need to create your correlation rule. In this case, you create failure and success rule conditions, and group by the ip_src.

                               
Rule ConditionDescription
FailuresThis condition searches for five failed logins with a "followed by" connector, meaning that the condition (Failures) must be followed by the next condition (Success).
SuccessThis condition searches for one successful login. 
GroupBy: ip_src, device classThe GroupBy field ensures that all the previous conditions are grouped by the ip_srcand device class. This is important to the construction of the rule because the rule attempts to find a case where a user has attempted to log into the same destination account multiple times, and finally logged in successfully. Grouping by device class ensures that the user logged in from the same machine attempted to log into an account multiple times. The rule may give unexpected results if you do not group the results.
Occurs within 5 minutesThe time window for the events to occur is five minutes. If the events occur outside of this time window, the rule does not trigger. 
Event Sequence: StrictThe event sequence is configured for a strict pattern match. This means that the pattern must match exactly as it is specifiedwith no intervening events. 

For the rule conditions, you create the following statements:

  • The "Failures" statement searches for failed login attempts:
    Failures Statement
  • The "Success" statement searches for one successful login:
    Success Statement
  • Combined, you have the following correlation rule:
    Login Failure Followed by Success

Step 2: Create the Enrichment

Now that you have created your rule, you need to create the enrichment to add to the notification output. Follow the steps above to create the enrichment, name it Last_Logon, and add the following query:

create window LastLogon.std:unique(ip_src) as (ip_src string, user_dst string);

insert into LastLogon select ip_src, user_dst from CoreEvent

where ec_activity='Logon' and ec_outcome='Success';

The enrichment should look like the following:

In-Memory Table dialog showing advanced query

 

Step 3: Add the Enrichment to the Rule

Now that you have created your basic rule and your enrichment, you'll need to add the enrichment to the rule and join (or connect) the enrichment to the meta in the rule.

Open the Login_Failure_Followed_by_Success rule for editing.

Enrichment added to rule

                                 
FieldEnterDescription
Output

In-Memory Table

The In Memory Table option creates a Named Window, which can be populated with the EPL query data.
Enrichment SourceLast_Logon (the enrichment you created above).This is the stream-based in-memory lookup table that is populated from incoming events to maintain a mapping of IP addresses to the last user logged in from that address.

ESA Event Stream Meta

ip_src

This is an event stream meta that you can join to the enrichment data you are populating. Essentially, ip_src is the join condition .

Enrichment Source Column Nameip_srcThis is the meta from the enrichment that you can join to the event stream data. It must be the same as join condition from the Event Stream Meta field.

Once you have added the enrichment, you can save the rule.

When the rule is triggered, the ESA runs the query in the enrichment and populates the Named Window with the data. If the data in the Named Window matches the join condition, the data is added to the output you can view in Email, SNMP, Syslog or Script, depending on how you configured notifications.

You are here
Table of Contents > Add a Data Enrichment Source > Enrichment Sources > Configure In-Memory Table as Enrichment Source

Attachments

    Outcomes