Reporting: Reporting Overview

Document created by RSA Information Design and Development on Sep 14, 2017Last modified by RSA Information Design and Development on Oct 15, 2017
Version 9Show Document
  • View in full screen mode

Reporting is a collection of data as a result of monitoring the network traffic, which can be used for further analysis. In NetWitness Suite you can run a report against NetWitness Suite Database core services to identify the network activities. For example, if you want to identify the Top Source Countries and Destination Countries, or top Threat and Risk trends that help monitor any changes to the normal categories or monitor the users and services that may potentially have malicious activities etc.

The reporting typically consist of: Reports and Charts. You can report on the log and packet data collected, and customize the reports and charts to enhance the visual appearance. You can create real-time reports for historical data. You can create charts and dashlets, that can be added in the real-time chart dashlets as well.

Reporting Engine

Reporting relies on the Reporting Engine to provide data for the reports, alerts and charts. Hence, you must configure the Reporting Engine as a service to NetWitness Suite before you can generate the reports. You must also specify the data source in the Reporting Engine from which the data is extracted.

The data that you can report or alert depends on the configuration of Reporting Engine and the data sources that you specify as part of the rule definition.

Note: Make sure you have access to the components in the Reporting.

Note: Make sure you have access to the required data sources. Only privileged users with access to sensitive information have the permission to certain data sources. To manage access control to data sources, see the "Add a Role and Assign Permissions for Warehouse Analytics" topic in the Warehouse Analytics Guide. However, for the existing reports, alerts and charts, if the user role or permissions are modified for the data sources, then it is not applicable unless you manually update the permissions.

Note: Reporting is accessible based on the role based access, defined for the user.

Report

A report is a combination of rules and other formatting objects such as headers and HTML-formatted notes that describe and identify data pertaining to a particular area of interest. Reports are defined and managed in the Build Report page and can be scheduled to run on an adhoc or timely basis. Once a report is run, results are stored centrally and can be automatically sent over email, SFTP, URL, and NFS to users, viewed via the NetWitness Suite web interface, downloaded as PDF and CSV files.

A report consists of the following:

                            
PropertyDescriptionExample
Report Name

Note: For Name field, the icon to extend the column size is not displayed at the end of the column field. You have to hover the mouse a little to the left side to see the icon for extending the column.

Used to identify the report to schedule them at a later time.Report1
Text

Pre-defined text fields used within a report to make the report more meaningful to the user.

Header1, Comment
Rules

The rules (queries) used to create a report.

select user.dst

where ip.src = 10.10.10.1

Note: In the Reporting user interface, the displayed date or time is always according to the user-selected time zone profile.

Rule

A rule is the basic and essential building block in the Reporting. You must create a rule which can be used in a Report, Chart or Alert.

A rule represents a unique query that detects and summarizes the requested information within a collection of network data.

The rule syntax is very similar to that of Standard Query Language (SQL) where you can use the select clause, where clause, sort and group options and limits for the result set. A rule consists of the following:

                                              
PropertyDescriptionExample
NameThe name of the rule.Windows System Account Activity
SelectList of meta types that are returned in the result set. The list of meta types is provided in the Meta Library. Meta Library in the Rule Builder is continually synchronized with the index configuration of the NetWitness Suite host to which NetWitness Suite is connected. The number of meta types that this property can represent depends on how the rule is to be sorted. If the Sort by property is 'None' or non-aggregate, a rule can have more than one select field, for example, for each match, include the ip.src, ip.dst, size, time in the rule result. If a rule is set to be sorted, either by session count, session size, or packet size, then there can only be one field on which to select. 
WhereA clause that is the base query for the rule.alert='cleartext_ftp_passwords'
Then (Rule Actions)A series of functions that manipulate the original result set of a rule in order to make the output in a report more meaningful or add additional functionality other than querying and displaying data. lookup_and_add ('username','ip.src',10);

Sort By

Determines how the data in the result set is sorted. The various possibilities are:

  • Total
  • Value
  • Column Name

Total

LimitDesignates how large a result set can be for the given rule. Users must note that if a result set is sorted by count or size, the limit represents the top (or bottom) N values to be returned. If the result set is not sorted, the first N values are returned.20

Note: In the User Interface (UI), the date or time displayed depends on the time zone selected by the user.

Rule Types

There are different rule types in the Reporting. Rule types designate the source of data for the report rule. Following are the rule types:

                     
Rule TypeDescription
NetWitness Database (NetWitness DB)The NetWitness database extracts the meta from a Reporting Engine configured to use a Concentrator, Broker and Archiver as the data sources and provides the meta for rules.
Warehouse Database (Warehouse DB)The Warehouse database, also referred to as the RSA NetWitness Warehouse, warehouses large volumes of data. The Warehouse is designed so that you can retrieve large volumes of data easily and efficiently. The Warehouse also extracts the meta from the Reporting Engine.
Respond Database (Respond DB)The Respond database reports on alerts and incidents. The Respond database contain alerts and incidents generated from different services and you can create a report on those alerts and incidents.

Note: In the User Interface (UI), the date or time displayed depends on the time zone selected by the user.

List

A list is a variable that refers to a series of comma-separated values (CSV).  You can insert a list into a rule or use it as an argument to a rule action. Lists can act as placeholders for other values, which you can populate and update as needed.

You can create, manage and view lists that can be used to define rules for Reporting and Alerting.

Lists cannot be empty or have duplicate or blank values.

Note: If you are defining a report with a rule which has lookup_and_add in the Then clause and direct the report output to a list, the list is not populated with the result.
For example, if you create a rule with ip.src in the Select clause and lookup_and_add ('ip.dst','ip.src', 10) in the Then clause, the report displays the result, but if you have redirected the output to a list, the list will be empty

Chart

Chart is a tabular or grid representation of data. It consists of the following:

                       
PropertyDescriptionExample
Chart NameIdentifies the chart.Chart1
Rule BasisIdentifies the rule path chosen from the folder hierarchy. 

Any NetWitness Suite DB rule in the Reporting Engine system which is not sorted by none can be used to instantly create a chart. In NetWitness Suite, the chart interval can be adjusted from the chart definition panel itself. Each time a chart runs, it stores its result data locally in the Reporting Engine, so that it can be reviewed in either the Dashboard View or Chart View without any performance considerations.

Note: In the Reporting user interface, the output for the field where Date and Time are displayed is always according to the user-selected time zone profile.

Note: The Reporting Engine (RE) will automatically check for the available disk space before you execute a Rule, Report, Chart and Alert. If the RE disk space (in percentage) is less than the minimum disk space threshold (default value is 5), the RE will stop the current execution and an error message 'Available disk space of Reporting engine home is <5%, please clean up the space to proceed further' is displayed. Additionally, you may also configure the minimum disk space threshold by using the following path: RE>Explore>com.rsa.soc.re>Configuration>CommonConfig>minDiskSpaceThreshold.

Reporting Guidelines

This section lists the RSA recommended guidelines to enhance the execution time of your reporting entities such as rules, reports, alerts, charts, and lists. The guidelines are provided for the following:

  • NWDB Rules
  • Timeout Configuration for NWDB Rules
  • Lookup and Add rule action
  • List value Reports

NWDB Rules

If the reporting entities such as report, alert, or chart contain NWDB rules (in most cases where the query contains Group By) takes a long time to execute, you may do the following: 

  1. Refine the Where clause: 
    You may limit the number of sessions scanned by using or refining the Where clause (especially when you use the Group By option). For example, consider the following rule.
    Normal Where Clause


    If you use a Where clause as mentioned above, the number of sessions aggregated is huge. To avoid this, you can filter only required sessions by specifying the list of IP addresses or creating a List (list of IP Address) that contains relevant IP addresses.
    Filtered Where Clause  
  2. Using indexed Meta keys in the Where clause:
    To understand if the Meta is indexed or not, mouse hover the Meta key. If the Value Type is INDEX_VALUE, then the Meta is indexed. The Value Type is INDEX_KEY or INDEX_NONE if the Meta is not indexed. 
    Below is a snapshot of a Meta key that is indexed.
    Indexed Meta key
     
  3. Configure the Timeout option:
    If the query is taking a long time and fails due to timeout issues, you can configure the timeout for the NWDB rule executions. For more information, see below section Timeout Configuration for NWDB Rules.  
  4. Schedule the queries to run at different times:
    If multiple query aggregates are concurrently executed and timeout occurs, you may schedule the queries to run at different times without much overlap. 

Timeout Configuration for NWDB Rules

Note: It is a good practice to check the statistics of the Reporting Engine and the NWDB data sources before you make any changes to the configuration. For more information, see the "Monitor Service Details" topic for Reporting Engine and "Monitor System Statistics" topic in the in the System Maintenance Guide.   

If NWDB rule execution fails due to timeout, you may get the following errors on the View a Report page: 

  • Reporting Engine timeout error
    • “Data source ‘10.31.x.x Concentrator' did not respond within the configured time 30 minutes for the ‘/sdk/values' request.” 
    • NWDB timeout error
      • "Error occurred while fetching data from source '10.31.x.x Concentrator'. {Timeout message from NWDB}
      In such cases, you may do the following:
      • Reporting Engine timeout 
        In case of Reporting Engine timeout, you may set the timeout to a longer duration so the long running queries can be executed. For more information on setting the NWDB Queries Time Out and NWDB Info Queries Time Out option for the Reporting Engine, see "Step 2. Configure Reporting Engine Settings" topic in the Reporting Engine Configuration Guide. RSA recommends you set the NWDB Query Time Out to zero minutes (implies no timeout) and NWDB Info Queries Time Out to 60 minutes.
      • NWDB timeout
        In case of NWDB timeout, you may need to configure the query.level.timeout and max.concurrent.queries parameters for the NWDB data source based on the recommendations in the Core Database Tuning Guide to fine tune the queries.
        The following figure is an example of Explorer view where you can set the parameters for NWDB data source.
        Set Parameters for NWDB Data Source
      • Schedule Reports at different times
        If the NWDB core devices are heavily utilized, you may schedule the reports to run at different times without overlap. 
      • Split the Report
        If you have many rules in a Report, split the report into multiple reports with each report containing logical set of rules. If you have multiple rules, all rules will begin to execute at the same time based on available threads, therefore you may group the rules logically into separate reports.

LookupAndAdd Rule Action

If a rule that consists of single or multiple lookup_and_add rule actions, takes a long time to execute the report, it is because each of the rule action triggers multiple lookup queries on the NWDB data source resulting in longer execution time.

To improve the report execution time, you may do the following:  

  • Refine the Where clause in the following:
    • Rule that contains the lookup_and_add rule action
    • lookup_and_add rule action
  • Set Limits
    You must set appropriate limits for the rule and rule actions. If the limit is high it will result in many queries being triggered and hence the report will take a long time to execute. 
  • Set the boolean aggregate parameter

    If you do not want the aggregate value such as sum(meta), count(meta) etc. for the lookup values, set the boolean aggregate parameter to false in the lookup_and_add rule action. For more information, see the NWDB Rule Syntax section in Rule Syntax .

    lookup_and_add(string select, string field, int limit, boolean inherit, string extraWhere, boolean aggregate)

    Consider the rule with lookup_and_add rule action:

    Rule with lookup and add rule action 

    The output is displayed:

    Rule with lookup and add rule action output 

  • Each lookup_and_add rule action triggers by default two concurrent lookup queries on the data source. RSA recommends that you retain the default setting, however if you want to increase the value you may want to ensure the value of Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine is less than the Max Concurrent Queries value in the NWDB data source configuration.
    If the NWDB data source is shared across other services, then you may retain a low value for the Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine as increasing it will impact the queries from other services. For more information, see "Reporting Engine General Tab" topic in Reporting Engine Configuration Guide
  • If you are interested only in unique values and not accurate aggregates, then set the Session Threshold to a non-zero value for the NWDB rule. For more information, see Create a Rule Using NetWitness Data Source section in Configure a Rule. The higher the value, the longer is the rule execution. If the value is set to zero it will take a longer time but will provide accurate aggregates. 
    Consider a rule with lookup_and_add rule action and Session Threshold set to 10.
    Rule with lookup_and_add rule action and Session Threshold set to 10

    The output is displayed:
    rule with lookup_and_add rule action and Session Threshold set to 10 output

List Value Reports

Use a Refined List:

In case of List value reports (for any data source type), individual reports will be generated for each value in the list. Therefore, more the number of values in the list the longer the reports will take to execute. Hence, you must use a refined list to generate such reports. 

Access Control for Reporting

Reporting Module provides you the option to set up access control for all the components in the module. In NetWitness Suite, you can define different roles and specify the access control for each of the role from the System Security module. You can define the access control to be provided for the Reporting module for each role. For more information, see "Step 1: Review the Pre-Configured NetWitness Suite Roles" and "Step 2: (Optional) Add a Role and Assign Permission" topics in the System Security and User Management Guide.

In the Reports module, you can modify the role permissions or access to the following Reporting objects:

The Following is an example of the hierarchy of the object groups, objects and dependents. This is an illustration of the Report Groups and Reports hierarchy.

Report Groups and Reports hierarchy

Report Groups and Reports Hierarchy

Permission for Object Groups

  • You must have the Read & Write permission to set the permissions for the Object Group, Objects, or Dependents. The dependents with “No Access” permission are grayed out and dependents with “Read-Only” permission are indicated with an icon.
  • When you set the permission for the Object Group, the Objects and Dependents in the Object Group do not inherit the permission automatically. You must select the "Apply these permissions to sub-groups and <Objects> in this group" option to achieve this. For example, if you do not want Operators roles to access reports in Report Group A, then you must set the permission on Group A to No access for the Operator  role and select the "Apply these permissions to sub-groups and Reports in this group" option.
  • When you set the permissions for the Object Group and select the "Apply these permissions to sub-groups and <Objects> in this group" option, the dependents such as rules or schedules in the objects do not inherit the permissions automatically. You must use the "Apply Read-only permission to Rules in the <Object>" option to apply the permission to the rules.
  • When you set the permissions for the Objects, you must ensure that the Objects in hierarchy should always have a permission that is less than or equal to the one above in the hierarchy for the permission to be applied. For example, if the reports in a Report Group have Read & Write permission and you apply a Read-Only or No Access permission at the Report Group level and select the "Apply these permissions to sub-groups and Reports in this group" option, then the permission on the rules will remain unchanged. 
  • The permissions are cascaded from top to down in the hierarchy and not vice-versa. For example, if you apply a permission to a rule, it does not change the permission of the Report that contains the rule. 

Permission for Objects or Dependents

  • You must have the Read & Write permission to set the permissions for the Objects or Dependents.
  • You can specify the permission for multiple objects at once instead of setting the permission for each object.
  • When you set the permission for the Object, the dependents in the Object do not inherit the permission automatically. You must select the "Apply Read-only permission to Rules in the <Object>" option to achieve this.

When you apply the permission to dependents the permission is applied based on the existing permission for the role. For example, consider an Analyst and a Operator with the following permissions for the different dependents (Report A object has Rule AA, Rule AB, and Rule AC as dependents). 

                               
Object or Dependent Analyst Operator
Report ARead & WriteNo Access
           Rule AARead & WriteNo Access
           Rule ABRead and WriteRead and Write
           Rule ACRead-onlyNo Access

When the Analyst applies a Read & Write permission for the Operator role and selects the option "Apply Read-only permission to Rules in the <Object>", then the permissions will be set for the different dependents as follows: 

Modify the Permissions

  • Group Level: Set the permissions at the Object Group level and for all the object and entities in the Group. For example, if you have 80 reports in the Administrators Reports group and you do not want anyone except the Administrator to add or modify these reports, you can set the permission for all the other roles at the group level to Read-Only and select the option to apply it to all the reports and sub-groups in the report group. 
  • Multiple Objects: Select multiple objects and specify the access for all the selected objects. For example, if you have 10 reports in the Network Traffic sub group with sensitive  information that you do not want anyone to access, select the 10 reports and then set the permission for all the roles as "No Access".
  • Single Object: Select only the object and specify the permission. For example, select the Network Traffic Report and specify the Read-Write permission for the Security Analyst role or select the Login Failure Alert and specify the Read-Write permission for a Security Analyst role.
                               
Object or DependentOperator (Before Permission is applied)Operator (After Permission is applied)
Report ANo AccessRead & Write
           Rule AANo AccessRead-only
           Rule ABRead and WriteRead & Write
           Rule ACNo AccessRead-only

Roles and Permissions for Reporting Module

Although NetWitness Suite has five pre-configured roles, you can add custom roles. For example, in addition to the pre-configured Analysts role, you can add custom roles for AnalystsEurope and AnalystsAsia.

                               
RolePermission
AdministratorsFull system access
OperatorsAccess to configurations but not to data
AnalystsAccess to data but not to configurations
SOC_ManagersSame access as Analysts and an additional permission to handle incidents
Malware_AnalystsAccess to malware events only

Depending on the user role, you can set the following access permissions to access the Reporting module components (Rules, Reports, Charts, Alerts, Lists):

  • Create
  • Delete
  • Export
  • Manage 
  • View 

Note: You must enable all these permissions for a user role to be able to define, delete, manage and view each of the Reporting modules. You must also have appropriate permissions for the data source to be listed, while defining the reports, charts, or alerts. For more information, see "Configure Data sources Permissions" topic in the Reporting Engine Configuration Guide.

For a detailed list of permissions and how to add a role and assign permissions, see "Role Permissions" and "Step 2. (Optional) Add a Role and Assign Permissions" topics in the System Security and User Management Guide

You are here
Table of Contents > Reporting Overview

Attachments

    Outcomes