ATD: Configuring Automated Threat Detection for Suspicious Domains

Document created by RSA Information Design and Development on Sep 13, 2017Last modified by Janice Krogh on Oct 12, 2017
Version 4Show Document
  • View in full screen mode
  

This topic tells administrators and analysts how to configure a Suspicious Domains module for NetWitness Suite Automated Threat Detection. The Automated Threat Detection functionality enables you to analyze the data that resides on one or more Concentrators by using preconfigured ESA Analytics modules. For example, using a Suspicious Domains module, an ESA Analytics service can examine your HTTP traffic to determine the probability that malicious activity is occurring in your environment.

There are two types of preconfigured Suspicious Domains modules available in NetWitness Suite: Command and Control (C2) for Packets and C2 for Logs. The Suspicious Domains module defines a subset of events and the activities executed on those events for identifying suspicious C2 domains.

Before you deploy an ESA Analytics module for Automated Threat Detection, it is important to note that there are many potential installation configurations that may be installed on the ESA, including: ESA Analytics, ESA Correlation Rules, and the Context Hub. Each of these may take up resources, so it is important to consider sizing before deploying Automated Threat Detection on your ESA.

Prerequisites

  • If you are using Packet data, you must have configured a Decoder for HTTP packet data, and you must have configured an HTTP Lua or Flex parser.
  • If you are using web proxy log data, you must have configured the appropriate Log Decoder with the correct parser for your web proxy.
  • If you are using web proxy log data, you must have updated to the latest log parsers. The following parsers are supported: Blue Coat Cache Flow (cacheflowelff), Cisco IronPort WSA (ciscoiportwsa), and Zscaler (zscalernss).
  • If you are using web proxy log data, for best results you should configure all web proxies the same way (set to the same time zone, use the same collection method -syslog or batch, and if you use batch use the same batching cadence).
  • A connection from the ESA host to the Whois service (same location as RSA Live cms:netwitness.com:443) must be opened on port 443. Verify with your System Administrator that this is complete.
  • To whitelist a domain, you need to enable the Context Hub service.

Important: Automated Threat Detection requires a "warm-up" period that acclimates the scoring algorithm to the traffic in your network. You should plan to configure Automated Threat Detection so that the warm-up period can run during normal traffic. For example, starting Automated Threat Detection on a Tuesday at 8:00 am in the timezone that contains the majority of your users allows the module to accurately analyze a day of normal traffic.

Configure Automated Threat Detection for Suspicious Domains

This procedure provides the steps needed to configure an ESA analytics Suspicious Domains module for Automated Threat Detection. ESA analytics modules, such as Suspicious Domains, are considered preconfigured because you do not have to manually create ESA rules for them.

The basic steps required are:

  1. Configure Log settings (for Logs only). Before you can use Automated Threat Detection for Logs, you must configure several settings. Skip this step if you plan to use Automated Threat Detection for Packets.
  2. Create a whitelist (optional) using the Context Hub service. Creating a whitelist allows you to ensure that commonly accessed websites are excluded from any Automated Threat Detection scoring.
  3. Configure the Whois Lookup service. The Whois service enables you to get accurate data about domains that you connect to. In order to ensure effective scoring, it is important that you configure the Whois Lookup service. Verify that the Whois Service is reachable from your environment.
  4. Map data sources to ESA Analytics modules. You define how NetWitness Suite Automated Threat Detection should automatically detect advanced threats by mapping a preconfigured ESA analytics module to multiple data sources, such as Concentrators, and an ESA analytics service.
  5. Verify the C2 incident rule is enabled and monitor for activity. After mapping your Suspicous Domains module, a period of time is required for the scoring algorithm to warm-up. After the warm-up period, verify that the C2 rule is enabled in the Incident Rules and monitor to see if the rule is triggered. 
  6. Verify that the incident rules are configured correctly. When you view incidents in the Respond view, it is helpful if the incidents are grouped by Suspected C&C.

Step 1: (For Logs Only) Configure Log Settings

To configure Automated Threat Detection for Logs, you need to complete a few extra configuration steps:

  • Verify that the supported parsers are enabled for your Log Decoder.
  • Get the latest versions of the appropriate web proxy parser from RSA Live.
  • Update the mapping on the Envision config file. This file is required to update the Log Decoder to work with the new meta available via the parsers.
  • Verify that the table-map.xml file was updated correctly.
  • Verify that the indexes were updated correctly.

To verify your parsers are running on your Log Decoder:

  1. Go to ADMIN Services.
  2. Select your Log Decoder and select Actions icon > View > Config
    The Service Parsers Configuration section shows a list of enabled parsers.
  3. Verify that the appropriate web proxy parser is enabled.

Log Decoder Configuration for Parsers

To get the latest parsers from RSA Live:

    1. Go to CONFIGURE > Live Content.
    2. Enter a search term for one of the supported web proxy parsers.
    3. Select the appropriate web proxy parser [for example, the Blue Coat ELFF (cacheflowelff) parser].

Note: You should have taken steps to configure logging to occur on your web proxy parser correctly.

  1. Click Deploy.
    The Deployment Wizard opens.
    Deployment Wizard
  2. Under Services, select the Log Decoder as the Service.
  3. Click Deploy to deploy the parser to your Log Decoder.

To Get the Latest Envision Config File:

  1. Go to CONFIGURELive Content.

  2. Enter envision as the key word for the search.
  3. Select the latest Envision Config file, and click Deploy.
    Live showing Envision Configuration File
  4. In the Deployment Wizard, under Services, select your Log Decoder.
  5. Click Deploy to deploy the Envision configuration file to the Log Decoder.

To Verify the Envision Configuration File was Updated Correctly:

  1. Go to ADMIN > Services, select the Log Decoder, and then select Actions icon > View > Config > Files tab.
    You can see the table-map.xml file. This file is modified when you update the Envision Configuration file.
  2. Search for the term, event.time. The field should now read, "event.time" flags ="None". This means that the event.time meta is now included in the mapping. Similarly, the fqdn flag should be set to "None".

To Verify the Indices for the index-concentrator.xml File are Updated:

You will need to verify that the index-concentrator.xml file includes both the event.time and fqdn meta.

  1. Go to ADMINServices, select your Concentrator, and then select Actions icon > View > Config
  2. On the Files tab, search for the index-concentrator.xml file.
  3. Verify that the following entry exists in your index-concentrator.xml file. If not, you will need to ensure your Concentrator is upgraded to the correct version:

<key description="FQDN" level="IndexValues" name="fqdn" format="Text" valueMax="100000" defaultAction="Open"/><key description="Event Time" format="TimeT" level="IndexValues" name="event.time" valueMax="0" />

Custom Index

Step 2: Create a Domains Whitelist (Optional)

This procedure is used when working with Automated Threat Detection to ensure that certain domains do not trigger a threat score. Sometimes, a domain you access regularly may trigger an Automated Threat Detection score. For example, a weather service might have similar beaconing behavior as a Command and Control communication and trigger an unwarranted negative score. When this happens, it is called a false positive. To prevent triggering a false positive with a specific domain, you can add the domain to a whitelist. Most domains do not need to be whitelisted because the solution only alerts on very suspect behaviors. The domains you may want to whitelist are valid automated services that do not have many host connections.

Note: For migrations from 10.6.x, if your previous Automated Threat Detection whitelist (Whitelisted Domains) appears on the Lists tab, you can rename it to domains_whitelist to use it for the Suspicious Domains modules.

  1. Create a whitelist for domains in Context Hub named domains_whitelist:
    1. Go to ADMIN > Services, select the Context Hub Server service, and then select View > Config > Lists tab.
      The Lists tab shows the current lists in the Context Hub.
      Context Hub Server service View > Config > Lists tab
    2. In the Lists panel, click Add icon to add a list. In the List Name field, type domains_whitelist. You must use this name in order for the module to recognize it.
      List tab showing domains_whitelist created
  2. Manually add domains to the list or import a .CSV file containing a list of domains.
    You can enter full domains, or you can use a wild card to include all sub-domains for a given domain. For example, you can enter *.gov to whitelist all government IP addresses. However, you cannot use other regex functions, such as [a-z]*.gov. This is because using *.gov replaces an entire string, such as www.irs.gov.
    1. To add domains manually, in the List Values section, click Add icon to add domains.
    2. To remove a domain, select the domain and click Delete icon.
    3. To import a .CSV file, in the List Values section, click Import icon, and in the Import List Values dialog, navigate to the .CSV file. Choose from the following delimiters: Comma, LF (Line Feed), and CR (Carriage Return) depending on how you have separated the values in your file. Click Upload
  3. Click Save.
    The domains_whitelist appears in the Lists panel. Analysts can add to this list from the Respond view and other parts of Investigation. The Context Hub Configuration Guide provides additional information.

Step 3: Configure the Whois Lookup Service

See the "Configure Whois Lookup Service" topic in the ESA Confguration Guide.

Step 4: Map Data Sources to ESA Analytics Modules

See the "Mapping ESA Data Sources to Analytics Modules" topic in the ESA Confguration Guide.

Step 5: Verify the Suspected Command & Control By Domain Rule is Enabled and Monitor the Rule

Verify the Suspected Command & Command Control by Domain rule in the Incident Rules.

  1. Go to CONFIGURE > Incident Rules > Aggregation Rules.
  2. Select the Suspected Command & Control Communication by Domain Rule, and double-click to open it. 
    Enable Incident Rule
  1. Verify that Enabled is selected.

The Rule displays a green Enabled button when it is enabled.

Result

After you deploy the ESA Analytics Suspicious Domains module mapping for Automated Threat Detection, your ESA will begin to perform analytics on the HTTP traffic. You can view detailed information for each incident in the Respond view.

Step 6: Verify the Incident is grouped by Suspected C&C

In order to group incidents correctly the Respond view, set the Group By condition to Domain.

  1. Go to CONFIGURE > Incident Rules > Aggregation Rules.
  2. Select the Suspected Command & Control Communication by Domain rule, and double-click to open it. 
  3. Verify that the Group By field is set to Domain.
    Automated Threat Detection Group option
    This will aggregate alerts and incidents will be created for "Suspected C&C".

Next Steps

Monitor the Respond view to see if the rule is triggered. The NetWitness Respond User Guide provides additional information.

 

You are here

Table of Contents > Configuring Automated Threat Detection for Suspicious Domains

Attachments

    Outcomes