Reporting: Reporting Guidelines

Document created by RSA Information Design and Development on May 2, 2016
Version 1Show Document
  • View in full screen mode
 
  

This topic lists the RSA recommended guidelines to enhance the execution time of your reporting entities. This topic lists the RSA recommended guidelines to enhance the execution time of your reporting entities such as rules, reports, alerts, charts, and lists. The guidelines are provided for the following:

  • NWDB Rules
  • Timeout Configuration for NWDB Rules
  • Lookup and Add rule action
  • List value Reports

NWDB Rules

If the reporting entities such as report, alert, or chart contain NWDB rules (in most cases where the query contains Group By) takes a long time to execute, you may do the following: 

  1. Refine the WHERE Clause: 
    You may limit the number of sessions scanned by using or refining the WHERE clause (especially when you use the Group By option). For example, consider the following rule.


    If you use a WHERE clause as mentioned above, the number of sessions aggregated is huge. To avoid this, you can filter only required sessions by specifying the list of IP addresses or creating a List (List of IP Address) that contains relevant IP addresses.

     
  2. Using indexed META keys in the WHERE clause:
    To understand if the META is indexed or not, mouse hover the META key. If the Value Type is INDEX_VALUE, then the META is indexed. The Value Type is INDEX_KEY or INDEX_NONE if the META is not indexed. 

    Below is a snapshot of a META key that is indexed.

     
  3. Configure the Timeout option:
    If the query is taking a long time and fails due to timeout issues, you can configure the timeout for the NWDB rule executions. For more information, see below section Timeout Configuration for NWDB Rules.  
  4. Schedule the queries to run at different times:
    If multiple query aggregates are concurrently executed and timeout occurs, you may schedule the queries to run at different times without much overlap. 

Timeout Configuration for NWDB Rules

Note: It is a good practice to check the statistics of the Reporting Engine and the NWDB data sources before you make any changes to the configuration. For more information, see the Monitor Appliances and Services topic for Reporting Engine and Monitor System Statistics topic in the in the System Maintenance Guide.   

If NWDB rule execution fails due to timeout, you may get the following errors on the View a Report page: 

  • Reporting Engine timeout error
    • “Data source ‘10.31.x.x Concentrator' did not respond within the configured time 30 minutes for the ‘/sdk/values' request.” 
  • NWDB timeout error
    • "Error occurred while fetching data from source '10.31.x.x Concentrator'. {Timeout message from NWDB}”.
  • In such cases, you may do the following:

    • Reporting Engine timeout 
      In case of Reporting Engine timeout, you may set the timeout to a longer duration so the long running queries can be executed. For more information on setting the NWDB Queries Time Out and NWDB Info Queries Time Out option for the Reporting Engine, see Configure Reporting Engine Settings topic in the Host and Services Configuration Guide. RSA recommends you set the NWDB Query Time Out to zero minutes (implies no timeout) and NWDB Info Queries Time Out to 60 minutes. 
    • NWDB timeout 
      In case of NWDB timeout, you may need to configure the query.level.timeout and max.concurrent.queries parameters for the NWDB data source based on the recommendations in the Database Tuning topic in the Host and Services Configuration Guide to fine tune the queries.

      Following is the snapshot of the Explorer view where you can set the parameters for NWDB data source. 
    • Schedule Reports at different times
      If the NWDB core devices are heavily utilized, you may schedule the reports to run at different times without overlap. 
    • Split the Report
      If you have many rules in a Report, split the report into multiple reports with each report containing logical set of rules. If you have multiple rules, all rules will begin to execute at the same time based on available threads, therefore you may group the rules logically into separate reports. 

LookupAndAdd Rule Action

If a rule that consists of single or multiple lookup_and_add rule actions, takes a long time to execute the report, it is because each of the rule action triggers multiple lookup queries on the NWDB data source resulting in longer execution time.

To improve the report execution time, you may do the following:  

  • Refine the WHERE clause in the following:
    • Rule that contains the lookup_and_add rule action
    • lookup_and_add rule action
  • Set Limits
    You must set appropriate limits for the rule and rule actions. If the limit is high it will result in many queries being triggered and hence the report will take a long time to execute. 
  • Set the boolean aggregate parameter

    If you do not want the aggregate value such as sum(meta), count(meta) etc. for the lookup values, set the boolean aggregate parameter to false in the lookup_and_add rule action. For more information, see the NWDB Rule Syntax.

    lookup_and_add(string select, string field, int limit, boolean inherit, string extraWhere, boolean aggregate)

    Consider the rule with lookup_and_add rule action:

    lookup_and_add_agg_false_106.png

    The output is displayed:

  • Each lookup_and_add rule action triggers by default two concurrent lookup queries on the data source. RSA recommends that you retain the default setting, however if you want to increase the value you may want to ensure the value of Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine is less than the Max Concurrent Queries value in the NWDB data source configuration.
    If the NWDB data source is shared across other services, then you may retain a low value for the Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine as increasing it will impact the queries from other services. For more information, see Reporting Engine General Tab topic in Host and Services Configuration Guide
  • If you are interested only in unique values and not accurate aggregates, then set the Session Threshold to a non-zero value for the NWDB rule. For more information, see Define a Rule Using NetWitness Data Source. The higher the value, the longer is the rule execution. If the value is set to zero it will take a longer time but will provide accurate aggregates. 
    Consider a rule with lookup_and_add rule action and Session Threshold set to 10.


    The output is displayed:

     

List Value Reports

Use a Refined List:

In case of List value reports (for any data source type), individual reports will be generated for each value in the list. Therefore, more the number of values in the list the longer the reports will take to execute. Hence, you must use a refined list to generate such reports. 

You are here: Reporting Overview > Reporting Guidelines

Attachments

    Outcomes