Configure ESA to Use a Memory Pool

Document created by RSA Information Design and Development on Nov 24, 2016Last modified by RSA Information Design and Development on Feb 9, 2017
Version 2Show Document
  • View in full screen mode
  

This topic tells administrators how to configure the ESA to use a memory pool.

 A memory pool is a customized implementation of virtual memory for events held by rules in ESA. This helps in scaling the capability of rules by an order of magnitude. When you want to create rules that cover a large time span or which are very complex, you may want to use a memory pool to handle memory more efficiently.  When you use a memory pool, instead of holding all of the events in memory, they can be written to disk. This is helpful because when a rule exists that is complex or extends over a long time frame, a large number of events must be held in memory.

You can configure memory pool to run in non-batch mode or batch mode:

  • Non-batch mode. In non-batch mode, events are written to disk as they enter the memory pool. To configure non-batch mode, set the MapPoolBatchWriteSize attribute to 1. Non-batch mode provides a more stable solution because each event is landed and fetched separately without creating memory spikes.
  • Batch mode. In batch mode, events are grouped into batches and then written to disk. To configure batch mode, set the batch size attribute MapPoolBatchWriteSize to a value greater than 1. Batch mode gives better performance since the disk activity for landing events to disk are optimized.

Note: Any changes to these settings will require you to restart the ESA. When ESA restarts, if any events are currently being held by the memory pool, they will be discarded upon restart. 

Caution: While this feature can be very helpful in managing memory, it can impact the event processing rate of the ESA.  Performance can be affected from 10 to 30 percent, depending on your rules and configuration settings.

Workflow

The following diagram shows the data flow using the memory pool for batch mode:

  1. Events are added to the memory pool and references to the events are stored in the memory pool.
  2. The events are then batched to be sent to disk (in non-batch mode, this step is skipped).
  3. Once the batch has met the threshold, the events are written to disk (in non-batch mode no threshold is required).
  4. When the EPL requires an event that was written to disk, the event is sent to the cache and used in the EPL rule. 

Procedure

Complete the following steps to configure an ESA memory pool. 

  1. From Administration > Services,  select your ESA service and then  > View > Explore
  2. Select  CEP > EsperPool > Configuration.
  3. Enter values for the following fields:
                                      
AttributeDescriptionConfiguration 
MapPoolPersistenceURILocation to store the memory pool file.

The default value is /opt/rsa/esa/pool/esperPool. RSA recommends you do not modify the default value.

 If you modify this setting to use a different partition, ensure the partition contains at least 10 times more space than the memory allocated for ESA.

Caution: If the memory pool is in use while this path is changed,a n ESA restart is required. When this occurs, ESA does not discard the stored events so you must manually purge them.

MapPoolEnableEnable or disable the memory pool.The default value is false. Set the value to true to enable the memory pool.  Requires a restart when you enable or disable memory pool.
MapPoolFlushIntervalSecs

Time interval to flush events to disk.  For example, any event held in Esper longer than 15 minutes gets flushed to disk.

The default value is 15 minutes. A smaller value ensures that the ESA is more stable when there are EPLs holding a large number of events in memory.  A larger value (greater than 30 minutes), ensures that only relevant events required over a longer period of time are flushed to disk. 

Note: Due to Java memory management design,  sometimes events not held by EPL may be sent to disk.  To help prevent this from occurring, you can set a higher value for MapPoolFlushIntervalSecs.

MapPoolBatchWriteSize

Specify the batch size (and whether to use batch mode). The events are batched into groups and then flushed to disk.

To use non-batch mode, set this value to 1.  

To use batch mode, set this value to greater than 1.

The default batch size is 100,000 events. At the end of flush interval, if the batch capacity is not reached, the batch expires in 30 seconds and all contents of the batch are written to disk as memory pool files.

A smaller value for the batch size (for example, 10,000 events) ensures that when events are fetched from disk, they do not pose a risk of bloating the memory, which creates more stability. However, a larger batch size (100,000 events) minimizes the input/output activity when writing events to disk, which can create better performance. 

MapPoolMinSizeMinimum size of the memory pool. This value is used for initialization, so it does not typically require editing. The default value is 10,000 events. A higher value may increase performance.  A lower value ensures that the system is more stable. 
MapPool Persist TypeThis is a view-only parameter that displays the type of optimization used.  The default value is RMSerialize

Note: The effectiveness of this feature depends on your environment. If you write rules that require frequent access of events over a period of time, this feature may degrade performance with no or minimal improvement in scalability.

Note: Memory pool files get deleted when all the events held in the pool file are no longer referenced by an EPL. 

Result

For a simple EPL rule, ESA typically improves memory approximately 8 to 9 times.

Previous Topic:Configure ESA Storage
You are here
Table of Contents > Additional ESA Procedures > Configure ESA to Use a Memory Pool

Attachments

    Outcomes