Streams Configuration

Document created by RSA Information Design and Development on Sep 14, 2017Last modified by RSA Information Design and Development on Nov 16, 2017
Version 5Show Document
  • View in full screen mode
 

The Streams tab for a Warehouse Connector in the Services Config view provides a way to manage stream configuration.

Workflow

add stream workflow

What do you want to do?

                                                          
RoleI want to...Refer to...
AdministratorInstall Warehouse Connector Service on a Log Decoder or DecoderInstall Warehouse Connector Service on a Log Decoder or Decoder or Hybrid
AdministratorConfigure a Warehouse Connector ServiceConfigure a Warehouse Connector Service
AdministratorConfigure the Data Source for Warehouse ConnectorConfigure the Data Source for Warehouse Connector

Administrator

Configure the Destination using NFS, SFTP, WebHDFS.

Configure the Destination Using NFS

Configure the Destination Using SFTP

Configure the Destination Using WebHDFS

AdministratorConfigure a Stream*Configure a Stream
AdministratorMonitor a Warehouse Connector*Monitor a Warehouse Connector

Administrator

Add Warehouse as Data Source to Reporting Engine

For more information, see 'Add Warehouse as a Data Source to Reporting Engine' in the Reporting Engine Configuration Guide.

AdministratorAnalyze a Warehouse ReportFor more information, see 'Step 4. Analyze a Warehouse Report' in the Warehouse Guide.

Administrator

Manage a Stream and Lockbox*

Manage a Stream and Lockbox

*You can complete these tasks here.

Related topics

Configure a Stream

Quick View

The following figure shows the Streams tab on the Warehouse Connector Services Config view.

StrmTabSelect.png

The Streams tab allows you to perform the following:

                                     
FeaturesDescription
Add a stream.
Delete a stream.
Edit the stream.
start icon Start the stream.
Stop the stream.
final icon Finalize the stream.
reload icon Reload the stream.
If you have added a new meta or if a new meta is added as part of content update to any of the sources, Log Decoder or Decoder, you need to reload the stream for the meta to be visible in the schema for the Reporting Engine. Reloading a stream does not have any impact on the data, but only the new meta list is fetched from the sources.

The following table describes the fields in the Streams tab:

                         
ParameterDescription
NameName of the stream.
SourcesThe sources associated with the stream.
DestinationThe destinations associated with the stream.
StatusStatus of the stream.

Stream Statistics

You can view the statistics of a configured stream. Click the  icon next to the name of the stream.

stream statistics

The following parameters are displayed in the Stream Statistics:

                                                            
SectionParameterDescription
Stream   
 Number of files in the permanent rejected folderDetermines the number of files in the permanent rejected folder (named, permfail) in the Warehouse Connector. The permanent rejected folder contains the files that Warehouse Connector failed to write to the destination.
 Rejected folder usage in percentageDetermines the disk usage of the rejected folder. 
 Session RateDetermines the rate at which the session is processed by the Warehouse Connector for the source.
Destination   
 StatusIndicates the status of the destination.
Source   
 StatusIndicate the status of the source. 
 Sessions BehindDetermines that number of sessions that needs to be processed by the Warehouse Connector.
 Last Session readDetermines the last session id processed by the Warehouse Connector.

Change Stream Configuration

You can change configuration of a stream in runtime. In the Streams tab, click Change Configuration to change the configuration of the selected stream.

change stream configuration

You can change the following parameters of the Stream Configuration:

Note: If you change the value of any parameter in stream configuration, make sure that you restart the stream.

After upgrading, if the values of Maximum Message Hold Count, Maximum Message Hold Interval and Maximum Message Hold Size are 3000000, 60 and 128 respectively, ensure that you assign the following values to the streams:

  • Maximum Message Hold Count - 2400000
  • Maximum Message Hold Interval - 600
  • Maximum Message Hold Size - 512

You can assign these values by modifying the existing Stream configuration.

                                                                    
SectionParameterDescription
Aggregation Configuration
 Aggregate max sessionsDetermines the maximum number of sessions in a response for an aggregation request from the Warehouse Connector to the source .
 Aggregation IntervalDetermines the time between the responses from the source.

Loader Settings

 Compress files on diskEnable to compress files on disk.
Supported values:
  • Deflate - Provides smaller compressed files and good performance while generating reports.
  • Off
By default, the parameter is set to deflate.
 Export Rollup IntervalDetermines the roll-up interval for export files and also the directory structure the Warehouse Connector writes to the destination. 
For example: 
If the parameter is set to:
ValueDirectory Structure
hour/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}/{hour}
minute/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}/{hour}/{minute}
day/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}
If you change the value of the parameter, ensure that you restart the stream. 
Recommended value is hour.
 Maximum Message Hold CountDetermines the maximum number of sessions to store in the memory before processing.

Note: If you have deployed a Warehouse Connector Virtual Appliance, make sure that you change the default  value of the parameter to 800000.

 Maximum Message Hold Interval (Seconds)Determines the maximum time (in seconds) to hold the sessions in memory before processing.
 Maximum Message Hold SizeDetermines the maximum size for the sessions to store in the memory before processing.
 Remote Export PathDetermines the  remote local mount point for HDFS (nfs://) and the location to export the data.
 Page Size

Stream Settings

 Auto StartupEnable to automatically start the stream whenever the Warehouse connector process is restarted. By default, the parameter is set to off.
Previous Topic:Add Stream Dialog
Next Topic:Lockbox Settings
You are here
Table of Contents > Warehouse Connector Configuration References > Streams Configuration

Attachments

    Outcomes