ATD: Configuring Automated Threat Detection for Suspicious Domains

Document created by RSA Information Design and Development on Sep 13, 2017Last modified by RSA Information Design and Development on Sep 11, 2018
Version 6Show Document
  • View in full screen mode

This topic tells administrators and analysts how to configure a Suspicious Domains module for NetWitness Platform Automated Threat Detection. The Automated Threat Detection functionality enables you to analyze the data that resides on one or more Concentrators by using preconfigured ESA Analytics modules. For example, using a Suspicious Domains module, an ESA Analytics service can examine your HTTP traffic to determine the probability that malicious activity is occurring in your environment.

There are two types of preconfigured Suspicious Domains modules available in NetWitness Platform: Command and Control (C2) for Packets and C2 for Logs. The Suspicious Domains module defines a subset of events and the activities executed on those events for identifying suspicious C2 domains.

Before you deploy an ESA Analytics module for Automated Threat Detection, it is important to note that there are many potential installation configurations that may be installed on the ESA, including: ESA Analytics, ESA Correlation Rules, and the Context Hub. Each of these may take up resources, so it is important to consider sizing before deploying Automated Threat Detection on your ESA.


  • If you are using Packet data, you must have configured a Decoder for HTTP packet data, and you must have configured an HTTP Lua or Flex parser.
  • If you are using web proxy log data, you must have configured the appropriate Log Decoder with the correct parser for your web proxy.
  • If you are using web proxy log data, you must have updated to the latest log parsers. The following parsers are supported: Blue Coat Cache Flow (cacheflowelff), Cisco IronPort WSA (ciscoiportwsa), and Zscaler (zscalernss).
  • If you are using web proxy log data, for best results you should configure all web proxies the same way (set to the same time zone, use the same collection method -syslog or batch, and if you use batch use the same batching cadence).
  • A connection from the ESA host to the Whois service (same location as RSA Live must be opened on port 443. Verify with your System Administrator that this is complete.
  • To whitelist a domain, you need to enable the Context Hub service.

IMPORTANT: Automated Threat Detection requires a "warm-up" period that acclimates the scoring algorithm to the traffic in your network. You should plan to configure Automated Threat Detection so that the warm-up period can run during normal traffic. For example, starting Automated Threat Detection on a Tuesday at 8:00 am in the timezone that contains the majority of your users allows the module to accurately analyze a day of normal traffic.

Configure Automated Threat Detection for Suspicious Domains

This procedure provides the steps needed to configure an ESA analytics Suspicious Domains module for Automated Threat Detection. ESA analytics modules, such as Suspicious Domains, are considered preconfigured because you do not have to manually create ESA rules for them.

The basic steps required are:

  1. Configure Log settings (for Logs only). Before you can use Automated Threat Detection for Logs, you must configure several settings. Skip this step if you plan to use Automated Threat Detection for Packets.
  2. Create a whitelist (optional) using the Context Hub service. Creating a whitelist allows you to ensure that commonly accessed websites are excluded from any Automated Threat Detection scoring.
  3. Configure the Whois Lookup service. The Whois service enables you to get accurate data about domains that you connect to. In order to ensure effective scoring, it is important that you configure the Whois Lookup service. Verify that the Whois Service is reachable from your environment.
  4. Map data sources to ESA Analytics modules. You define how NetWitness Platform Automated Threat Detection should automatically detect advanced threats by mapping a preconfigured ESA analytics module to multiple data sources, such as Concentrators, and an ESA analytics service.
  5. Verify that the C2 incident rule is enabled and monitor for activity. After mapping your Suspicous Domains module, a period of time is required for the scoring algorithm to warm-up. After the warm-up period, verify that the C2 rule is enabled in the Incident Rules and monitor to see if the rule is triggered. 
  6. Verify that the incident rules are configured correctly. When you view incidents in the Respond view, it is helpful if the incidents are grouped by Suspected C&C.

Step 1: (For Logs Only) Configure Log Settings

To configure Automated Threat Detection for Logs, you need to complete a few extra configuration steps:

  • Verify that the supported parsers are enabled for your Log Decoder.
  • Get the latest versions of the appropriate web proxy parser from RSA Live.
  • Update the mapping on the Envision config file. This file is required to update the Log Decoder to work with the new meta available via the parsers.
  • Verify that the table-map.xml file was updated correctly.
  • Verify that the indexes were updated correctly.

To verify that your parsers are running on your Log Decoder:

  1. Go to ADMIN > Services.
  2. Select your Log Decoder and select Actions icon > View > Config
    The Service Parsers Configuration section shows a list of enabled parsers.
  3. Verify that the appropriate web proxy parser is enabled.

Log Decoder Configuration for Parsers

To get the latest parsers from RSA Live:

  1. Go to CONFIGURE > Live Content.
  2. Enter a search term for one of the supported web proxy parsers.
  3. Select the appropriate web proxy parser [for example, the Blue Coat ELFF (cacheflowelff) parser].
  4. Note: You should have taken steps to configure logging to occur on your web proxy parser correctly.

  5. Click Deploy.
    The Deployment Wizard opens.
    Deployment Wizard
  6. Under Services, select the Log Decoder as the Service.
  7. Click Deploy to deploy the parser to your Log Decoder.

To get the latest Envision Config file:

  1. Go to CONFIGURE > Live Content.

  2. Enter envision as the key word for the search.
  3. Select the latest Envision Config file, and click Deploy.
    Live showing Envision Configuration File
  4. In the Deployment Wizard, under Services, select your Log Decoder.
  5. Click Deploy to deploy the Envision configuration file to the Log Decoder.

To verify that the Envision Configuration file was updated correctly:

  1. Go to ADMIN > Services, select the Log Decoder, and then select Actions icon > View > Config > Files tab.
    You can see the table-map.xml file. This file is modified when you update the Envision Configuration file.
  2. Search for the term, event.time. The field should now read, "event.time" flags ="None". This means that the event.time meta is now included in the mapping. Similarly, the fqdn flag should be set to "None".

To verify that the Indices for the index-concentrator.xml file are updated:

You must verify that the index-concentrator.xml file includes both the event.time and fqdn meta.

  1. Go to ADMIN > Services, select your Concentrator, and then select Actions icon > View > Config
  2. On the Files tab, search for the index-concentrator.xml file.
  3. Verify that the following entry exists in your index-concentrator.xml file. If not, ensure that your Concentrator is upgraded to the correct version:

<key description="FQDN" level="IndexValues" name="fqdn" format="Text" valueMax="100000" defaultAction="Open"/><key description="Event Time" format="TimeT" level="IndexValues" name="event.time" valueMax="0" />

Custom Index

Step 2: Create a Domains Whitelist (Optional)

This procedure is used when working with Automated Threat Detection to ensure that certain domains do not trigger a threat score. Sometimes, a domain you access regularly may trigger an Automated Threat Detection score. For example, a weather service might have similar beaconing behavior as a Command and Control communication and trigger an unwarranted negative score. When this happens, it is called a false positive. To prevent triggering a false positive with a specific domain, you can add the domain to a whitelist. Most domains do not need to be whitelisted because the solution only alerts on very suspect behaviors. The domains you may want to whitelist are valid automated services that do not have many host connections.

Note: For migrations from 10.6.x, if your previous Automated Threat Detection whitelist (Whitelisted Domains) appears on the Lists tab, you can rename it to domains_whitelist to use it for the Suspicious Domains modules.

  1. Create a whitelist for domains in Context Hub named domains_whitelist:
    1. Go to ADMIN > Services, select the Context Hub Server service, and then select Actions icon > View > Config > Lists tab.
      The Lists tab shows the current lists in the Context Hub.
      Context Hub Server service View > Config > Lists tab
    2. In the Lists panel, click Add icon to add a list. In the List Name field, type domains_whitelist. You must use this name in order for the module to recognize it.
      List tab showing domains_whitelist created
  2. Manually add domains to the list or import a .CSV file containing a list of domains.
    You can enter full domains, or you can use a wild card to include all sub-domains for a given domain. For example, you can enter *.gov to whitelist all government IP addresses. However, you cannot use other regex functions, such as [a-z]*.gov. This is because using *.gov replaces an entire string, such as
    1. To add domains manually, in the List Values section, click Add icon to add domains.
    2. To remove a domain, select the domain and click Delete icon.
    3. To import a .CSV file, in the List Values section, click Import icon, and in the Import List Values dialog, navigate to the .CSV file. Choose from the following delimiters: Comma, LF (Line Feed), and CR (Carriage Return) depending on how you have separated the values in your file. Click Upload
  3. Click Save.
    The domains_whitelist appears in the Lists panel. Analysts can add to this list from the Respond view and the Investigate view. The Context Hub Configuration Guide provides additional information.

Step 3: Configure the Whois Lookup Service

See "Configure Whois Lookup Service" in the ESA Confguration Guide.

Step 4: Map Data Sources to ESA Analytics Modules

See "Mapping ESA Data Sources to Analytics Modules" in the ESA Confguration Guide.

Step 5: Verify that the Suspected Command & Control By Domain Rule is Enabled and Monitor the Rule

Note: The information in this procedure applies to version 11.1 and later.

Verify and monitor the Suspected Command & Command Control by Domain rule in the Incident Rules list.

  1. Go to CONFIGURE > Incident Rules.
  2. In the Incident Rules list, locate the Suspected Command & Control Communication by Domain rule and verify that it displays a green Enabled icon (Green enabled icon) next to the rule name.
    Incident Rules List view showing Suspected Command & Control Communications by Domain rule and status
  3. If the rule is not enabled:
    1. Click the link in the NAME field to open it.
    2. In the Incident Rule Details view, select Enabled and click Save. Incident Rule Details view for the Suspicious Command & Control Communication By Domain rule showing Enabled selected
  4. In the Incident Rules list, monitor the statistics in the following fields to see if the rule is triggered:
    • Last Matched: Shows the time when an alert was successfully matched with the rule.
    • Matched Alerts: Displays the number of alerts that matched the rule.
    • Incidents: Displays the number of incidents created by the rule.

    By default, these values reset to zero every 7 days. For more information, see "Set Counter for Matched Alerts and Incidents" in the NetWitness Respond Configuration Guide.

Step 6: Verify that the Incident is grouped by Suspected C&C

Note: The information in this procedure applies to version 11.1 and later.

In order to group incidents correctly in the Respond view, set the Group By condition to Domain for Suspected C&C.

  1. Go to CONFIGURE > Incident Rules.
  2. In the Incident Rules list, locate the Suspected Command & Control Communication by Domain rule and click the link in the NAME field to open it.
  3. In Grouping Options section, verify that the Group By field is set to Domain for Suspected C&C.
    Automated Threat Detection Group By option
    This aggregates alerts and incidents are created for "Suspected C&C."


After you deploy the ESA Analytics Suspicious Domains module mapping for Automated Threat Detection, your ESA begins to perform analytics on the HTTP traffic. You can view detailed information for each incident in the Respond view.

Next Steps

Monitor the Respond view to see if the rule is triggered. The NetWitness Respond User Guide provides additional information.

You are here
Table of Contents > Configuring Automated Threat Detection for Suspicious Domains