CBA: Mapping Cloud Gateway Analytic Streams

Document created by RSA Information Design and Development on Mar 15, 2018Last modified by RSA Information Design and Development on Mar 26, 2018
Version 2Show Document
  • View in full screen mode
 

You can configure the RSA Cloud Gateway to automatically upload Analytic Streams from one or more Concentrators to Cloud Behavioral Analytics (CBA). An Analytic Stream is a pipeline of selected traffic activity used for analytics processing. For example, Analytic Streams can include HTTP, FTP, SMB, or DNS traffic. By creating and deploying Analytic Stream mappings between Concentrator sources and Cloud Gateway services, data streams are automatically forwarded to the Cloud for analytics processing.

When you deploy your mapping, the selected Cloud Gateway service uses query-based aggregation to collect the appropriate filtered events for the selected Analytic Stream from the Concentrators. Query-based aggregation is a predefined query that only transfers data for the selected Analytic Stream. Only the data required by the Analytic Stream is transferred from the Concentrator to Cloud Behavioral Analytics.

Considerations

When creating and deploying your Analytic Stream mappings, keep the following important considerations in mind:

  1. Each Analytic Stream that you deploy places an additional load on the Internet egress points on the network.
  2. Every Analytic Stream that you add impacts the Concentrators.

  3. Ensure that you map Analytic Streams to Concentrators that actively collect that type of information. For example, HTTP Analytic Streams should only be activated on Concentrators that collect HTTP activity.

Analytic Stream Deployment Example - Two Gateways

To take advantage of your additional Concentrator capacity, you can map an Analytic Stream to a Cloud Gateway service and deploy it to analyze data from multiple data sources at the same time.

For example, if you have three Concentrators and two Cloud Gateway services, you can create and deploy the following mappings:

  • Map Analytic Stream 1 to the Concentrator 1 and 2 sources and the Cloud Gateway Server 1 service. Cloud Gateway Server 1 sends Analytic Stream 1 filtered traffic from Concentrators 1 and 2 to CBA in the Cloud.
  • Map Analytic Stream 2 traffic to the Concentrator 2 and 3 sources and the Cloud Gateway Server 2 service. Cloud Gateway Server 2 sends Analytic Stream 2 filtered traffic from Concentrators 2 and 3 to CBA in the Cloud.

In this example, Analytic Stream 1 represents an Analytic Stream, such as HTTP, and Analytic Stream 2 represents another Analytic Stream, such as FTP in another location. Concentrator 1 collects HTTP activity, Concentrator 2 collects HTTP and FTP activity, and Concentrator 3 collects FTP activity.

Module Deployment Example - 2 Gateways

This example shows how both services can process data from the same Concentrator. Notice that Cloud Gateway services 1 and 2 can both process data from Concentrator 2. Cloud Gateway Server 1 queries data for Analytic Stream 1 HTTP traffic and Cloud Gateway Server 2 queries different data for Analytic Stream 2 FTP traffic.

Analytic Stream Deployment Example - One Gateway

In addition to creating Analytic Stream mappings that are processed by different Cloud Gateway services, you can map more than one Analytic Stream to the same Cloud Gateway service.

For example, if you have three Concentrators and one Cloud Gateway service, you can create and deploy the following mappings:

  • Map Analytic Stream 1 to the Concentrator 1 and 2 sources and the Cloud Gateway Server 1 service. Cloud Gateway Server 1 sends Analytic Stream 1 filtered traffic from Concentrators 1 and 2 to CBA in the Cloud.
  • Map Analytic Stream 2 to the Concentrator 2 and 3 sources and the Cloud Gateway Server 1 service. Cloud Gateway Server 1 also sends Analytic Stream 2 filtered traffic from Concentrators 2 and 3 to CBA in the Cloud.

In this example, Analytic Stream 1 represents an Analytic Stream, such as HTTP, and Analytic Stream 2 represents another Analytic Stream, such as FTP in another location. Concentrator 1 collects HTTP activity, Concentrator 2 collects HTTP and FTP activity, and Concentrator 3 collects FTP activity.

Module Deployment Example - One Gateway

This example shows how one service can process data from more than one Analytic Stream. Notice that Cloud Gateway Server 1 can process data from Concentrators 1 and 2 for Analytic Stream 1. It also processes data from Concentrators 2 and 3 for Analytic Stream 2. Cloud Gateway Server 1 queries data for Analytic Stream 1 HTTP traffic and queries different data for Analytic Stream 2 FTP traffic and then sends that data to CBA in the Cloud for analytics processing.

Caution: Ensure that all NetWitness Suite host services are in sync with a consistent time source.

Prerequisites

  • All NetWitness Suite host services must be in sync with a consistent time source.
  • The Concentrator hosts and services must be discovered and available in the NetWitness Suite user interface.
  • The Cloud Gateway Server service must be provisioned. See Provision a Cloud Gateway.

Create Cloud Gateway Analytic Stream Mappings

The following procedure tells you how to map Analytic Streams to sources and services. After creating and reviewing the mappings, you deploy them so that they can start aggregating data.

  1. Go to ADMIN > System, and in the options panel, select Cloud Gateway.
    The Cloud Gateway Analytic Stream Mappings panel is displayed.
    Cloud Gateway Analytic Stream Mappings panel
  2. Click to create an Analytic Stream mapping. Create a separate mapping for each Analytic Stream.
    The Create Analytic Stream Mappings dialog is displayed.
    Create Analyti Stream Mappings dialog
  3. In the Analytic Stream list, select an Analytic Stream.
  4. Configure one or more data sources (Concentrators) for your mappings. Do the following for each Concentrator:
    1. Click Add icon .
      The Available Services dialog shows the data sources that are available from the ADMIN > Services view.
      Available Services dialog showing available data sources
    2. In the Available Services dialog, select a Concentrator and click OK.
      The Add Service dialog is displayed.
      Add Service dialog - empty
    3. In the Add Service dialog, type the Administrator username and password for the Concentrator.
    4. Click Test Connection to make sure that it can communicate with the Cloud Gateway service.
      Add Service dialog - Test Connection successful
    5. Click OK.
      After you configure your data sources and they appear in the Sources list, you can reuse them for additional mappings.
  5. In the Sources list, select one or more data sources to aggregate the data for the Analytic Stream.
    Create Analytic Stream Mappings dialog showing a mapping
    A solid colored green circle indicates a running service and a white circle indicates a stopped service.
  6. In the Service list, select a Cloud Gateway service to process the data for the Analytic Stream.
  7. If necessary, specify the Lag Time that will be used to query data from the selected Concentrators.
    Lag Time (Minutes) specifies a constant time delay in minutes, which is added to avoid losing events being processed by the data sources during periods of heavy activity. For example, Concentrator performance varies depending on factors such as incoming load, ongoing queries, and indexing. Due to these factors, a Concentrator may not aggregate events in real-time, which leads to the delay.
    The Lag Time parameter gives the Concentrator a chance to finish aggregating all of the data.
    Data aggregates at Current (System) Time - Lag Time. Setting Lag Time is useful when a Concentrator is slow in aggregating data. The Lag Time guarantees that Cloud Behavioral Analytics (CBA) does not process data that arrives to the Concentrator within the Lag Time window so there is adequate delay to ensure all events that get generated in the enterprise can be processed by CBA.
    For example, if Lag Time is 30 minutes, and the current time is 2:00 PM, the Concentrator starts pulling records at 1:30 PM. The Lag Time window, 30 minutes in this example, remains constant as time advances. When the current time advances to 2:01 PM, the Concentrator pulls the next minute of data at 1:31 PM, and so on.
    Important: The Lag Time defines the buffer between the current time and the time when the Analytic Stream ingests the data.
  8. Caution: RSA recommends that Administrators adjust the Lag Time parameter dynamically based on the performance of each of the individual Concentrators to avoid missing any events during aggregation.

  9. Click Create.
    The mappings that you create appear in the list of existing mappings with a status of Undeployed.
    CBA Gateway Mappings panel - Undeployed mapping
    Important: To start an Analytic Stream so that it starts aggregating data, you need to deploy it.

Deploy Cloud Gateway Analytic Stream Mappings

After you create your mappings, you need to deploy them in order to start aggregating data for the Analytic Streams.

  1. In the list of mappings, verify that the status of the mappings that you want to deploy show as Undeployed.
  2. Select one or more mappings with a status of Undeployed and select Deploy Now.
    All selected mappings in the Undeployed state start to aggregate data as configured in the mapping. The mapping status changes to Deployed.
    You cannot deploy a mapping that has already been deployed.

Update a Mapping

You can only have one mapping per Analytic Stream. If you want to make changes to a deployed mapping, such as adding or removing Concentrators or changing the service, you must undeploy and delete the existing mapping and then create and deploy a new mapping for that Analytic Stream.

You can make the following updates to a deployed mapping without deleting it:

  • Undeploy the mapping
  • Change the Lag Time

You can also change the Lag Time for an undeployed Analytic Stream mapping.

Undeploy a Mapping

If you want to stop aggregating data for an Analytic Stream mapping, but you do not want to delete the mapping, you can undeploy it. This gives you the option of deploying it at a later time. When you undeploy a mapping, the specified Cloud Gateway service stops pulling data from the data source for that Analytic Stream.

Caution: Undeploying a mapping with a status of Deployed will affect data aggregation for that Analytic Stream.

To undeploy a mapping:

  1. In the Cloud Gateway Analytic Stream Mappings panel, select the deployed mapping that you want to undeploy.
  2. In the Actions column, select Actions icon > Undeploy.
    The status changes from Deployed to Undeployed and data aggregation stops.

Delete a Mapping

You can delete a mapping with a status of Undeployed at any time. Since a mapping in the Undeployed state is not running, it does not affect data aggregation.

You should undeploy a mapping with a status of Deployed before deleting it. Undeploying and deleting a mapping clears the configuration on the Cloud Gateway Server, reverts the deployment for that mapping, and stops pulling data from the data source for that Analytic Stream.

Caution: Undeploying and deleting a mapping will affect data aggregation for that Analytic Stream.

To delete a mapping:

  1. In the Cloud Gateway Analytic Stream Mappings panel, select the mapping that you want to delete. You can only delete one mapping at a time.
  2. Click Delete icon.

Change the Lag Time

If necessary, you can change the Lag Time for the Analytic Stream. The Lag Time defines the buffer between the current (system) time and the time when the Analytic Stream ingests the data.

  1. In the Cloud Gateway Analytic Stream Mappings panel, select the mapping that you want to change and in the Actions column, select Actions icon > Edit stream.
    The Analytic Stream Settings dialog shows the selected Analytic Stream, Cloud Gateway service, and data sources for the mapping. The data sources show the URLs used to communicate with the Cloud Gateway service.
    Analytic Stream Settings dialog
  2. If necessary, you can adjust the Lag Time (Minutes) to give the Concentrators in the mapping additional time to finish aggregating all of the data.
  3. Click Save.
    Changes DO NOT take effect immediately. For the settings to take effect, you need to undeploy and re-deploy the mapping.
  4. To undeploy the mapping, in the Cloud Gateway Analytic Stream Mappings panel, select the mapping that you want to undeploy and then select Actions icon > Undeploy.
    Data aggregation stops for the selected mapping.
  5. To re-deploy the mapping, select the mapping that you want to deploy and then select Actions icon > Deploy.
    The selected mapping deploys and starts to aggregate data as configured in the mapping.

Monitor the Cloud Gateway

You can monitor the RSA Cloud Gateway service statistics in NetWitness Suite.

In the Services view, select the Cloud Gateway Server service and then select actions icon > View > Explore. In the Explore view, you will see important statistics that you should monitor for the Cloud Gateway. You can adjust your Analytic Stream Mappings as required.

The following figure shows two important statistics that you will want to look at for each stream:

  • upload-bytes-meter: This statistic shows the average bytes uploaded per second. It is the upload rate (bytes).
  • events-seen-meter: This statistic shows the number of events pulled from the Concentrator per second. It is the reading rate (number).

The first part of the statistic name shows the Analytic Stream name, in this example the Analytic Stream Name is C2:

C2/pipe/compressed/upload-bytes-meter

Cloud Gateway Server service Config Explore view - Top - showing the stream name "C2", upload-bytes-meter, and events-seen-meter

If you scroll down, you can see the rest of the statistics. If you have more than one Analytic Stream, you will see statistics for that stream, too. In this example, you can see statistics for the C2 and C2Packets Analytic Streams.

Cloud Gateway Server service Config Explore view - Bottom

Note:
- Each Analytic Stream that you deploy places an additional load on the Internet egress points on the network. Look at the upload statistics, such as upload-bytes-meter.
- Every Analytic Stream that you add impacts the Concentrators. Look at the events-seen-meter statistic and see the "Monitor Service Details" topic in the System Maintenance Guide.
- Ensure that you map Analytic Streams to Concentrators that actively collect that type of information. For example, HTTP Analytic Streams should only be activated on Concentrators that collect HTTP activity.

You are here
Table of Contents > Mapping Cloud Gateway Analytic Streams

Attachments

    Outcomes