Warehouse Connector: Services Config View

Document created by RSA Information Design and Development on Jul 31, 2016
Version 1Show Document
  • View in full screen mode

The tabs for a Warehouse Connector in the Services Config view provide a way to manage basic service configuration, configure sources, configure destination, and configure streams.

The following are the tabs in the Warehouse Connector Services Config view:

  • Sources and Destinations
  • Streams
  • Lockbox Settings
  • General
  • Appliance Service Configuration

Sources and Destinations

The Sources and Destinations tab contains the following two sections:

  • Source Configuration
  • Destination Configuration


Source Configuration

The Source Configuration section allows you to configure the data sources form which the Warehouse Connector service needs to collect data.

The following is an example of the Source Configuration section.


The Source Configuration section allows you to perform the following.

add_icon.png Add the data source.
delete_icon.png Delete the data source.

Destination Configuration

The Destination Configuration section allows you to configure the destination to which the Warehouse Connector service needs to write the collected data.


The Destination Configuration section allows you to perform the following.

add_icon.png Add the destination.
delete_icon.png Delete the destination.
edit_icon.png Edit the destination.

Note: You can only edit the SFTP destination type.


The Streams tab allows you to perform all of the tasks related to a stream.


The Streams tab allows you to perform the following.

add_icon.png Add a stream.
delete_icon.png Delete a stream.
edit_icon.png Edit the stream.
start_icon.png Start the stream.
stop_icon.png Stop the stream.
finalize_icon.png Finalize the stream.
reload_icon.png Reload the stream.
If you have added a new meta or if a new meta is added as part of content update to any of the sources, Log Decoder or Decoder, you need to reload the stream for the meta to be visible in the schema for the Reporting Engine. Reloading a stream does not have any impact on the data but only the new meta list is fetched from the sources.

The Streams tab also lists all the created streams in a tabular format.

Column NameDescription
NameName of the stream.
SourcesThe sources associated with the stream.
DestinationThe destinations associated with the stream.
StatusStatus of the stream.

Stream Statistics

You can view the statistics of a configured stream. Click the info-sm.png icon next to the name of the stream.


The following parameters are displayed in the Stream Statistics.

 Number of files in the permanent rejected folderDetermines the number of files in the permanent rejected folder (named, permfail) in the Warehouse Connector. The permanent rejected folder contains the files that Warehouse Connector failed to write to the destination.
 Rejected folder usage in percentageDetermines the disk usage of the rejected folder. 
 Session RateDetermines the rate at which the session is processed by the Warehouse Connector for the source.
 StatusIndicates the status of the destination.
 StatusIndicate the status of the source. 
 Sessions BehindDetermines that number of sessions that needs to be processed by the Warehouse Connector.
 Last Session readDetermines the last session id processed by the Warehouse Connector.

Change Stream Configuration

You can change configuration of a stream in runtime. In the Streams tab, click Change Configuration to change the configuration of the selected stream.


You can change the following parameters of the Stream Configuration:

Note: If you change the value of any parameter in stream configuration, make sure that you restart the stream.

After upgrading, if the values of Maximum Message Hold Count, Maximum Message Hold Interval and Maximum Message Hold Size are 3000000, 60 and 128 respectively, ensure that you assign the following values to the streams:

Maximum Message Hold Count - 2400000

Maximum Message Hold Interval - 600

Maximum Message Hold Size - 512

You can assign these values by selecting Administration > Services > Select the Warehouse Connector service > Actions menu cropped > View > Config > Streams tab > Select the existing Stream > Change Configuration.

Aggregation Configuration   
 Aggregate max sessionsDetermines the maximum number of sessions in a response for an aggregation request from the Warehouse Connector to the source .
 Aggregation IntervalDetermines the time between the responses from the source.
Loader Settings   
 Compress files on diskEnable to compress files on disk.
Supported values:
  • deflate - Provides smaller compressed files and good performance while generating reports.
  • off
By default, the parameter is set to deflate.
 Export Rollup Interval

Determines the roll-up interval for export files and also the directory structure the Warehouse Connector writes to the destination. 

For example, iIf the parameter is set to:

Value Directory Structure
hour/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}/{hour}
minute/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}/{hour}/{minute}
day/rsasoc/v1/[logs | sessions]/data/{year}/{month}/{day}

If you change the value of the parameter, ensure that you restart the stream.

Recommended value is hour.

 Maximum Message Hold CountDetermines the maximum number of sessions to store in the memory before processing.

Note: If you have deployed a Warehouse Connector Virtual Appliance, make sure that you change the default  value of the parameter to 800000.

 Maximum Message Hold Interval (Seconds)Determines the maximum time (in seconds) to hold the sessions in memory before processing.
 Maximum Message Hold SizeDetermines the maximum size for the sessions to store in the memory before processing.
 Remote Export PathDetermines the  remote local mount point for HDFS (nfs://) and the location to export the data.
Stream Settings Page Size 
 Auto StartupEnable to automatically start the stream whenever the Warehouse connector process is restarted. By default, the parameter is set to off.

Lockbox Settings

The Lockbox Settings tab allows you can set, change, or refresh the lockbox password of the Warehouse Connector.



The General tab displays the system configuration parameters for the Warehouse Connector service.


When you add a Warehouse Connector service, default values are in effect. RSA designed the default values to accommodate most environments and recommends that you do not edit these values because it may adversely affect performance. The following table describes the System Configuration parameters.

NameConfig Value
CompressionDetermines the minimum amount of bytes before a message is compressed. If set to zero, messages are not compressed.
PortDetermines the port used by the service.

Note: If you change the port number, ensure that you restart the service.

SSLIf enabled, all the data transferred in the network will be encrypted using SSL.
Stat Update IntervalDetermines how often (in milliseconds) statistic nodes are updated in the system.
ThreadsDetermines the number of threads in the thread pool to handle incoming requests.

Appliance Service Configuration

The Appliance Service Configuration tab displays the appliance configuration settings for Warehouse Connector service. For more information, see Appliance Service Configuration in the Hosts and Services Getting Started Guide.


You are here: Configure Warehouse Connector > Warehouse Connector: Services Config View