Alerting: Add an Advanced EPL Rule

Document created by RSA Information Design and Development Employee on Sep 12, 2017Last modified by Shree Kulkarni on Oct 12, 2020
Version 19Show Document
  • View in full screen mode

This topic provides instructions to define rule criteria by writing an EPL query. EPL is a declarative language for handling high-frequency time-based event data. It is used to express filtering, aggregation, and joins over possibly sliding windows of multiple event streams. EPL also includes pattern semantics to express complex temporal causality among events.

Write an advanced EPL rule when rule criteria is more complex than what you can specify in Rule Builder.

It is outside the scope of this guide to explain EPL syntax. 

For best practices on writing advanced EPL rules, see ESA Rule Writing Best Practices.

Prerequisites

The following are prerequisites for adding an advanced rule:

  • You must know Event Processing Language (EPL).
  • You must understand ESA Annotations to mark which EPL statements are linked to generating alerts.

Add an Advanced EPL Rule

  1. Go to (Configure) > ESA Rules.
  2. In the Rule Library, select Add List icon> Advanced EPL.
    New Advanced EPL Rule Tab

  3. Type a unique, descriptive name in the Rule Name field.

    This name will appear in the Rule Library so be specific enough to distinguish the rule from others.

  4. In the Description field, explain which events the rule detects.

    The beginning of this description will appear in the Rule Library

  5. Select Trial Rule to automatically disable the rule if all trial rules collectively exceed the memory threshold.

    Use trial rule mode as a safeguard to see if a rule runs efficiently and to prevent downtime caused by running out of memory. For more information, see Working with Trial Rules.

  6. (This option applies to 11.5 and later.) Enter a Memory Threshold for a rule that uses memory, such as a rule that contains windows or pattern matching. If the configured memory threshold is exceeded, the rule gets disabled individually and an error is displayed for that rule on the (Configure) > ESA Rules > Services tab. The Memory Threshold option works for trial rules and non-trial rules. New rules default to a 100 MB memory threshold. Rules that existed before version 11.5 do not have a default value and a memory threshold is not set.
  7. (This option applies to 11.3 and later.) Select Alert to send an alert to Respond. Clear the checkbox if you do not want to send an alert to Respond. To turn alerts on or off for ALL rules, see the ESA Configuration Guide.
  8. For Severity, classify the rule as Low, Medium, High or Critical.
  9. To define rule criteria, write a Query in EPL.

    Note: For all meta key names, use an underscore not a period. For example, ec_outcome is correct but ec.outcome is not.

    Meta entities are not currently supported, such as:

    fullname.all

    eth.all

    ip.all

    ipv6.all

    port.src.all

    port.dst.all

    dir.path.all

    org.all

    geoip.all

    port.all

    domain.all

    email.all

    filename.all

    directory.all

    checksum.all

    param.all

    context.all

    attack.all

    analysis.all

    compromise.all

    inv.all

    outcome.all

    ec.all

    user.all

    host.all

    client.all

    Caution: If you add meta entities to your rule, they cannot get data from the data sources, so they do not trigger alerts.

  10. For dynamic statement name generation in ESA, you must enclose the meta keys in curly brackets and include this annotation in the syntax:

    @Name("RIG {ip_src} {alias_host} {ec_activity}")

    where,

    • RIG is the static part of the statement name
    • {ip_src}, {alias_host}, {ec_activity} is the dynamic part of the statement name

    Note: If any of the metas in the dynamic part of the statement name has a null value, it is displayed as a static text.

    If a rule should generate an alert, include this ESA annotation in the syntax:

    @RSAAlert

    For more information on ESA Annotations, see ESA Annotations.

Validate an Advanced EPL Rule

You can confirm that an ESA rule generates the expected alerts by testing the rule logic using JSON input data. You can view the alerts in the output, but this test does not send any alert notifications. If you want to view all of the debug information, include an @Audit(‘stream’) annotation to your rule query and view the Debug Log in the test output. To enable auditing you require to add @Audit to the rule and set the logging level for the Esper audit package to INFO. This can be done by creating correlation-server.yml file under /etc/netwitness/correlation-server with this content and restarting the correlation-server service:

logging:

     level:

         com.espertech.esper.audit: INFO


The following basic example query contains the @Audit(‘stream’) annotation and queries for events that do not have a source IP of 1.1.1.1 or 2.2.2.2.

Example basic query that contains @Audit('stream')

  1. If you are not already in the rule, go to (Configure) > ESA Rules > Rules tab and in the Rule Library, open the ESA rule that you want to test.
  2. Scroll down to the Test Rule section.
    Test Rule section
  3. In the ESA Service field, select the ESA Correlation service to process the rule. Use the same ESA Correlation service that you plan to use in the ESA rule deployment that contains the rule.
  4. In the Input Data field, enter the input events to test the rule. Download the events from the Investigate view in JSON format, copy the events, and paste them in this field. You can do this from the Investigate > Navigate view or the Investigate > Events view.

    To download the events from the Investigate > Navigate view:
    1. In the main menu, go to Investigate > Navigate in a new tab, select a data source, and click Navigate.
    2. In the Navigate view, click Load Values and click a meta value to filter the events.
    3. Save the events as meta in the JSON file format [Save Events > Meta > (name the file) > Export Meta Format: choose JSON].
    4. In the toolbar click the Image of Jobs icon, which looks like a stop watch. (Jobs) icon and then click View Your Jobs.
    5. In the Jobs panel, download your extracted meta, for example: investigation-2020-May-19-08-30-20.json.
    6. Go to back to the (Configure) > ESA Rules tab opened previously and copy the contents of the JSON file into the Input Data field in your ESA rule.

    To download the events from the Investigate > Events view:

    1. In the main menu, go to Investigate > Events in a new tab.
    2. In the Events view, enter a query for the ESA rule test.
    3. Select the events to use and in the Download or Download All menu, select Visible Meta as JSON or All Meta as JSON, depending on the size of your selection.
    4. In the main menu, go to Dashboards and in the toolbar click the Image of Jobs icon, which looks like a stop watch. (Jobs) icon and then click View Your Jobs.
    5. In the Jobs panel, download your extracted meta, for example: Concentrator_ALL_EVENTS_ALL_META.json.
    6. Go to back to the (Configure) > ESA Rules tab opened previously and copy the contents of the JSON file into the Input Data field in your ESA rule.
  5. Click Test Rule. The Output field shows the output of your rule and you can determine if the results meet your requirements.

Advanced EPL Rule - Test Rule section

The following table describes the test rule output Engine Stats.

FieldDescription
Engine VersionEsper version running on the ESA service
Events OfferedNumber of events processed by the ESA service since the last service start
Offered RateThe rate that the ESA service processes current events / The maximum rate that the ESA service processed events
Runtime ErrorsIf applicable, this field can contain a link to runtime error messages related to the ESA rule deployment.

The following table describes the test rule output Rule Stats.

FieldDescription
DeployedA green checkmark indicates that the rule is deployed on the selected ESA service.
Statements FiredThe number of statements that fired the alerts
Alerts FiredThe number of alerts generated from the test data
Events in MemoryThe number of events placed in memory by the rule
Memory UsageThe total amount of memory used by the rule
CPU %The percentage of the deployment CPU used by the rule. For example, a deployment with 1 rule shows 100% CPU usage for that rule and a deployment with two equally CPU heavy rules show 50% each.
Events MatchedThe number of events that matched the rule
Alerted EventsIf applicable, this field can contain a link to events that caused an alert.
Runtime ErrorsIf applicable, this field can contain a link to runtime error messages related to the rule.
Debug LogsThis field contains a link to Esper debug (audit) logs.

 

You are here

Table of Contents > Add an Advanced EPL Rule

Attachments

    Outcomes