ESA Config: Mapping ESA Data Sources to Analytics Modules

Document created by RSA Information Design and Development on Sep 13, 2017Last modified by Janice Krogh on Oct 4, 2017
Version 6Show Document
  • View in full screen mode
 

This topic tells Administrators how to map specific ESA Analytics modules to multiple data sources and ESA Analytics services, which can make processing more efficient.

You can analyze the data that resides on one or more Concentrators with the RSA NetWitness Suite Automated Threat Detection functionality by selecting a preconfigured ESA Analytics module. The data analyzed by these modules is used to identify advanced threats. To better utilize your network resources and reduce unnecessary data flow, you can map multiple data sources, such as Concentrators, to multiple ESA Analytics services in order to process data more efficiently and take advantage of additional capacity.

An ESA Analytics module is a pipeline composed of activity objects that enrich an event with additional information through mathematical computations. ESA Analytics modules reside within ESA Analytics services.

When you deploy your mapping, the selected ESA Analytics services use query-based aggregation to collect the appropriate filtered events for the selected module from the Concentrators. Query-based aggregation is a predefined query that only transfers data for the selected ESA Analytics module. Only the data required by the module is transferred between the Concentrator and the ESA Analytics system.

There are currently two ESA Analytics modules available for Suspicious Domains: C2 for Packets (http-packet) and C2 for Logs (http-log).

Module Deployment Example - Two ESAs

To take advantage of your additional Concentrator capacity, you can map an ESA Analytics module to an ESA Analytics service and deploy it to analyze data from multiple data sources at the same time.

For example, if you have three Concentrators and two ESA Analytics services, you can create and deploy the following mappings:

  • Map Module 1 to the Concentrator 1 and 2 sources and the ESA Analytics 1 service. ESA Analytics Service 1 analyzes Module 1 filtered events from Concentrators 1 and 2.
  • Map Module 2 to the Concentrator 2 and 3 sources and the ESA Analytics 2 service. ESA Analytics Service 2 processes Module 2 filtered events from Concentrators 2 and 3.

In this example, Module 1 represents an ESA Analytics module, such as C2 for Packets (http-packet) and Module 2 represents another ESA Analytics module, such as C2 for Logs (http-logs) in another location.

Module Deployment Example - 2 ESAs

This example shows how both services can process data from the same Concentrator. Notice that ESA Analytics Services 1 and 2 can both process data from Concentrator 2. ESA Analytics Service 1 queries data for Module 1 events and ESA Analytics Service 2 queries different data for Module 2 events.

Module Deployment Example - One ESA

In addition to creating module mappings that are processed by different ESA Analytics services, you can map more than one module to the same ESA Analytics service.

For example, if you have three Concentrators and one ESA Analytics service, you can create and deploy the following mappings:

  • Map Module 1 to the Concentrator 1 and 2 sources and the ESA Analytics 1 service. ESA Analytics Service 1 analyzes Module 1 filtered events from Concentrators 1 and 2.
  • Map Module 2 to the Concentrator 2 and 3 sources and the ESA Analytics 1 service. ESA Analytics Service 1 also processes Module 2 filtered events from Concentrators 2 and 3.

Module Deployment Example - One ESA

This example shows how one service can process data from more than one module. Notice that ESA Analytics Service 1 can process data from Concentrators 1 and 2 for Module 1. It also processes data from Concentrators 2 and 3 for Module 2. ESA Analytics Service 1 queries data for Module 1 events and queries different data for Module 2 events.

Caution: Ensure that all NetWitness Suite host services are in sync with a consistent time source.

Prerequisites

  • All NetWitness Suite host services must be in sync with a consistent time source.
  • The Concentrator hosts and services must be discovered and available in the NetWitness Suite user interface.
  • All module-specific requirements must be followed.
    • For Suspicious Domains:
      • Configure log settings (Suspicious Domains for Logs only)
      • Create a whitelist using the Context Hub service.
      • Configure the Whois Lookup Service.
      • Verify that the C2 incident rule is enabled and monitor it for activity.
      • Verify that the incidents are grouped by Suspected C&C.

    For step-by-step procedures, see the NetWitness Suite Automated Threat Detection Guide.

Create ESA Analytics Mappings

The following procedure tells you how to map ESA Analytics modules to sources and services. After creating and reviewing the mappings, you deploy them so that they can start aggregating data.

  1. Go to ADMIN > System, and in the options panel, select ESA Analytics.
    The ESA Analytics Mappings panel is displayed.
    ESA Analytics Mappings panel
  2. Click to create an ESA Analytics mapping. Create a separate mapping for each module.
    The Create Mappings dialog is displayed.
    Create Mappings dialog - Empty
  3. In the Module list, select a module.
  4. Configure one or more data sources (Concentrators) for your mappings. Do the following for each Concentrator:
    1. Click Add icon .
      The Available Sources dialog shows the data sources that are available from the Admin > Services view.
      Available Sources dialog
    2. In the Available Sources dialog, select a Concentrator and click OK.
      The Add Source dialog is displayed.
      Add Source dialog - empty
    3. In the Add Source dialog, type the Administrator username and password for the Concentrator.
    4. Click Test Connection to make sure that it can communicate with the ESA Analytics service.
      Add Source dialog - Test Connection successful
    5. Click OK.
      After you configure your data sources and they appear in the Sources list, you can reuse them for additional mappings.
  5. In the Sources list, select one or more data sources to aggregate the data for the module.
    Create Mappings dialog
    A solid colored green circle indicates a running service and a white circle indicates a stopped service.
  6. In the Service list, select an ESA Analytics service to process the data for the module.
  7. If necessary, specify the time that will be used to query data from the selected Concentrators:

    Field

    Description

    Warm-up Period (Hours)

    Specifies a warm-up duration (in hours). A warm-up period is required to allow Automated Threat Detection to "learn" your traffic. The warm-up period should run when typical traffic is running. During this time, alerting for your module mapping is suppressed. The Warm-up Period primes the module with historical data and guarantees that the specified number of hours of data collection completes before sending alerts.

    RSA provides preconfigured ESA Analytics modules. Each module type has a default warm-up period defined, which you can adjust to your environment, if necessary. After this warm-up period, alerts can be viewed.

    For more information about Warm-up Period and Lag time, see Module Settings.

    Lag Time (Minutes)

    Specifies a constant time delay in minutes, which is added to avoid losing events being processed by the data sources during periods of heavy activity. For example, Concentrator performance varies depending on factors such as incoming load, ongoing queries, and indexing. Due to these factors, a Concentrator may not aggregate events in real-time, which leads to the delay.

    The Lag parameter gives the Concentrator a chance to finish aggregating all of the data.

    After the warm-up period completes, data aggregation continues at Current (System) Time - Lag Time. This is useful when a Concentrator is slow in aggregating data. The Lag time guarantees that the module does not process data that arrives to the Concentrator within the Lag time window so there is adequate delay to ensure all events that get generated in the enterprise can be processed by the module.

    For example, if Lag time is 30 minutes, and the current time is 2:00 PM, the Concentrator starts pulling records at 1:30 PM. The Lag time window, 30 minutes in this example, remains constant as time advances. When the current time advances to 2:01 PM, the Concentrator pulls the next minute of data at 1:31 PM, and so on.

    Important: The Lag time defines the buffer between the current time and the time when the module ingests the data.

    Caution: RSA recommends that Administrators adjust the Lag parameter dynamically based on the performance of each of the individual Concentrators to avoid missing any events during aggregation.

    For more information about Warm-up Period and Lag time, see Module Settings.

  8. Click Create.
    The mappings that you create appear in the list of existing mappings with a status of Undeployed.
    ESA Analytics Mappings panel
    Important: To start a module so that it starts aggregating data, you need to deploy it.

Deploy ESA Analytics Mappings

After you create your mappings, you need to deploy them in order to start aggregating data for the modules.

  1. In the list of mappings, verify that the status of the mappings that you want to deploy show as Undeployed.
  2. Select one or more mappings with a status of Undeployed and select Deploy Now.
    All selected mappings in the Undeployed state start to aggregate data as configured in the mapping. The mapping status changes to Deployed.
    You cannot deploy a mapping that has already been deployed.

Update a Mapping

You can only have one mapping per module. If you want to make changes to a deployed mapping, such as adding or removing Concentrators or changing the service, you must undeploy and delete the existing mapping and then create and deploy a new mapping for that module.

You can make the following updates to a deployed mapping without deleting it:

  • Undeploy the mapping
  • Change the warm-up period and lag time

You can also change the warm-up period and lag time for an undeployed module mapping.

Undeploy a Mapping

If you want to stop aggregating data for a module mapping, but you do not want to delete the mapping, you can undeploy it. This gives you the option of deploying it at a later time. When you undeploy a mapping, the specified ESA Analytics service stops pulling data from the data source for that module.

Caution: Undeploying a mapping with a status of Deployed will affect data aggregation for that module.

To undeploy a mapping:

  1. In the ESA Analytics Mappings panel, select the deployed mapping that you want to undeploy.
  2. In the Actions column, select Actions icon > Undeploy.
    The status changes from Deployed to Undeployed and data aggregation stops.

Delete a Mapping

You can delete a mapping with a status of Undeployed at any time. Since a mapping in the Undeployed state is not running, it does not affect data aggregation.

You should undeploy a mapping with a status of Deployed before deleting it. Undeploying and deleting a mapping clears the configuration on the ESA server, reverts the deployment for that mapping, and stops pulling data from the data source for that module.

Caution: Undeploying and deleting a mapping will affect data aggregation for that module.

To delete a mapping:

  1. In the ESA Analytics Mappings panel, select the mapping that you want to delete. You can only delete one mapping at a time.
  2. Click Delete icon.

Change the Warm-up Period and Lag Time

You may want to adjust the warm-up period for a specific module mapping. For example, after the warm up period is complete, you can increase the warm-up period setting to allow additional warm-up time. You can even increase the warm-up period when your module mapping is actively warming up.

If necessary, you can change the lag time for the module. The lag time defines the buffer between the current (system) time and the time when the module ingests the data.

  1. In the ESA Analytics Mappings panel, select the mapping that you want to change and in the Actions column, select Actions icon > Edit Module.
    The Module Settings dialog shows the selected module, ESA Analytics service, and data sources for the mapping. The data sources show the URLs used to communicate with ESA.
    Module Settings dialog
  2. Review the Warm-Up State section to determine the current warm-up state:
    • Warm Up Started At - The time when the first event was processed by the ESA Analytics module from the data source.
    • First Event Time - The time that the first event occurred. The warm-up time is based on this time.
    • Latest Event Time - The time that the latest event occurred.
    • Remaining Warm Up Time - The number of hours remaining in the warm-up period.
    • Is Completed? - Indicates whether the warm-up period is complete. If it is true, the warm-up period is complete. If it is false, the module is still warming up and you can view the number of hours remaining in the Remaining Warm Up Time field.
  3. In the Configuration section, you can update the Warm-Up Period (Hours) depending on whether or not the warm-up period is complete.
    • During the warm up period - You can add hours to the warm-up period or subtract any remaining warm-up time.
    • The warm-up period is complete - You can add hours to the warm-up period by adding the difference between the current time and the First Event Time to the hours that you want to add.
      For example, a warm-up period of 10 hours is complete and the First Event Time shows 12:00:00. The current time is 16:00:00 (4 hours later) and you want to add 5 more hours to the warm-up time. To do this, you need to add 9 hours (4+5=9) to the warm-up period of 10, so you would set the new warm-up period to 19 hours.
      You cannot decrease the warm-up period if it is complete, unless you delete the mapping and create a new one.
  4. If necessary, you can adjust the Lag Time (Minutes) to give the Concentrators in the mapping additional time to finish aggregating all of the data.
  5. Click Save.
    Changes DO NOT take effect immediately. For the settings to take effect, you need to undeploy and re-deploy the mapping.
  6. To undeploy the mapping, in the ESA Analytics Mappings panel, select the mapping that you want to undeploy and Actions icon > Undeploy.
    Data aggregation stops for the selected mapping.
  7. To re-deploy the mapping, select the mapping that you want to deploy and Actions icon > Deploy.
    The selected mapping deploys and starts to aggregate data as configured in the mapping.

 

You are here

Table of Contents > Configure ESA Analytics > Mapping ESA Data Sources to Analytics Modules

Attachments

    Outcomes