Warehouse Connector collects meta and events from Decoder and Log Decoder and writes them in AVRO format into a Hadoop-based distributed computing system. You can set up Warehouse Connector as a service on existing Log Decoders or Decoders.
The Warehouse Connector contains the following components:
- Data Source
- Data Stream
A data source is the service from which the Warehouse Connector collects data to store in the destination. The supported data sources are Log Decoder and Decoder services. The Log Decoder collects log events and the Decoder collects packet and meta exclusively.
Destination is the Hadoop-based distributed computing system that collects, manages, and enables reporting on security data. The following are the supported destinations:
- RSA NetWitness Warehouse (MapR) deployments
- HortonWorks Data Platform
- Any Hadoop-based distributed computing system that supports WebHDFS or NFS mounting of HDFS file systems.
- Example: Commercial MapR M5 Enterprise Edition for Apache Hadoop
A data stream is a logical connection between the data source and destination. You can have multiple streams for different subsets of data collected. You can setup streams to segregate data from multiple Decoder and Log Decoder services. You can create a stream with multiple data sources and a single destination or with a single data source and destination.
The Warehouse Connector does the following:
- Aggregates session and raw log data from Decoders and Log Decoders.
- Transfers the aggregated data into supported destinations like Hadoop based deployments.
- Serializes the aggregated data that includes both schema and data into AVRO format.
In addition the Warehouse Connector also supports the following:
Meta filters enables you to filter the meta keys that should be written into the Warehouse. For more information, see Specify Meta Filters for a Stream.
Support for Multi-Valued Meta Keys
RSA NetWitness Warehouse supports multi-valued meta keys. The multi-valued meta keys is the meta field with the array type. You can use the meta keys library to determine the meta fields of type array and write HIVE queries with the correct syntax for arrays. By default, the following meta keys are treated as multi-valued and are defined in the file, multivalue-bootstrap.xml located at /etc/netwitness/ng in the Warehouse Connector:
Warehouse Connector enables you to validate the file integrity of the AVRO files that are transferred from the Warehouse Connector to the data destinations. You need to enable checksum validation while you configure the Warehouse Connector.
Lockbox provides an encrypted file that Warehouse Connector uses to store and protect sensitive data. You need to create the lockbox by providing a lockbox password while configuring the Warehouse Connector for the first time.
You can implement Warehouse Connector by setting up Warehouse Connector as a service on your existing Log Decoder or Decoder hosts.
The following is an overview of the entire process of installing and configuring the Warehouse Connector service on Log Decoder or Decoder, configuring the Warehouse Connector service on NetWitness, configuring data sources, destinations, streams for Warehouse Connector, and configuring alert notifications on NetWitness.
To install and configure the Warehouse Connector service, perform the following:
- Install Warehouse Connector service on a Log Decoder or Decoder
- Configure a Warehouse Connector service
- Configure the Data Source for Warehouse Connector
- Configure a Destination
- Configure a Stream
- Monitor a Warehouse Connector
- Add Warehouse as a Data Source to Reporting Engine
- Analyze a Warehouse Report
- Manage a Stream and Lockbox