Archiver: Group Aggregation

Document created by RSA Information Design and Development on Jul 21, 2016Last modified by RSA Information Design and Development on Jul 21, 2016
Version 2Show Document
  • View in full screen mode
 

This topic explains how Concentrators can be teamed into clusters that divide the work of aggregation between multiple hosts.

Note: Clustered aggregation does not apply to Brokers.

Clustered aggregation (also known as "gang" aggregation) allows multiple concentrators to efficiently aggregate from a single Decoder.  Any number of Concentrators can be grouped together to form an aggregation gang. The Concentrators in the gang divide all the sessions between themselves. Group or clustering aggregation allows multiple Concentrators to efficiently aggregate from a single Decoder in the type of architecture shown below:

Example

Any number of Concentrators can be grouped together to form a cluster aggregation. The Concentrators in the cluster divide all the sessions between themselves.

In a two-concentrator gang, the sessions aggregated by each concentrator might look something like this:

                     
Concentrator 0Concentrator 1
1 - 9,99910,000 - 19,999
20,000 - 29,99930,000 - 39,999
40,000 - 49,99950,000 - 59,999

Group Aggregation Parameters

You can configure the cluster as part of the Concentrator and Archiver configuration in the Security Analytics Services Config view. This table lists the parameters, which you can also configure in the Services Explore view.

                                   
ParameterWhere Parameter is SetDescription
Aggregate Max Sessions/concentrator/config/
aggregate.sessions.max
The number of sessions that a Concentrator receives from a Concentrator at any given time. When a Concentrator is part of a gang, it is also used to determine how the Decoder's sessions will be divided among the gang. The Decoder's sessions are divided into pieces of size aggregate.sessions.max, and then the pieces are evenly distributed among the Concentrators. For example, if aggregate.sessions.max is 10,000, then sessions 1-9,999 go to the first Concentrator and sessions 10,000-19,999 go to the second concentrator. All concentrators in the gang must use the same value for aggregate.sessions.max.
conGangName/concentrator/
devices/<device>/config/
options,gang=<gang name>
The gang name is used to determine the membership of a particular gang. There can be an unlimited number of gangs aggregating from a Decoder. The gang parameter is simply a mechanism that allows the decoder to identify which Concentrators are working together. All members of the gang must have the same gang name. If the gang name parameter is not set, gang aggregation is disabled. The gang name can be any string identifier.
Gang Size/concentrator/
devices/<device>/config/
options,gangSize=<number of devices
in gang>
This parameter sets the size of the gang. All Concentrators must have the same value for gang size. There is no limit on the size of the gang.
Gang Member ID/concentrator/
devices/<device>/config/
options,gangMember=<id number of
concentrator in gang>
This parameter sets the position of the concentrator in the group. For any gang of size N, gang member IDs from 0 to N-1 must be set on each of the gang members. For example, if the gang size is 2, one member gets gang member gang member ID 0, and the other gets gang member ID 1. If the gang size is 3, then the members are assigned IDs 0, 1, and 2.
Gang Membership Mode/concentrator/
devices/<device>/config/
options,gangMembership=(new|replace)
This parameter determines how this concentrator catches up when aggregation is started for the first time. The default behavior is replace. This means that when this Concentrator starts aggregation, it is intended to replace an existing gang member, or all members of the gang are being initialized at the same time. This means that the Concentrator begins aggregation from the oldest session available on the host it is aggregating from. If this parameter is set to new, it means this Concentrator is being added as a new member of an existing group. This Concentrator will not try to aggregate any existing sessions from the service. The other members of the group have already aggregated all the sessions on the service. This Concentrator will only aggregate new sessions as they appear on the service.

Examples

The following table shows sample parameters for a two-member gang with Gang ID foo.

                    
ParameterConcentrator 0Concentrator 1
aggregate.max.sessions 10,000 10,000
device options gang=foo gangSize=2
gangMember=0
gang=foo gangSize=2
gangMember=1


The following table shows sample parameters for a three-member gang with Gang ID baz.

                    
ParameterConcentrator 0Concentrator 1
aggregate.max.sessions 10,000 10,000
device options gang=baz gangSize=3
gangMember=0
gang=baz gangSize=3
gangMember=2

Cluster Size

When the division of sessions for a cluster needs to change, either to increase or decrease the cluster size, you must toggle to an offline state. While offline, all members need to be updated to have the same gangSize parameter. The replacement host aggregates only its division of sessions. Likewise, any cluster member can be data-reset and reaggregated at any time without affecting other members of the cluster.

Cluster Member

A cluster member can be replaced at any time by setting the gangMember parameter for the replacement host to be the same as the gangMember parameter for the old hold. The replacement host aggregates only its division of sessions. Likewise, any cluster member can be data-reset and reaggregated at any time without affecting other members of the cluster.

Attachments

    Outcomes