Archiver: Configure Log Storage Collections

This topic provides instructions for Administrators on how to configure log storage collections on an Archiver.

Security Analytics enables you to define individual storage collections for different log types. You can specify the maximum size of the Hot and Warm Storage space used by the collection, whether to use offline storage (Cold Storage), the number of days to retain the logs in the collection, the data compression, and whether to use a hash algorithm to be able to verify the data integrity of the files being saved. You should create collections based on your log retention storage requirements. Each collection that you create must be associated with at least one retention rule.

Prerequisites

Before you configure your log retention storage collections, configure total hot, warm, and cold storage.

Configure a Log Storage Collection

To configure a log retention storage collection on an Archiver:

1. In the Security Analytics menu, select Administration > Services.
2. Select the Archiver service and  > View > Config.
   The Services Config view of Archiver is displayed.
3. On the Data Retention tab, in the Collections section, click  to add a collection.
   (If you decide to make changes to an existing collection, you can select the collection and click  to change the settings.)
   
   
   The Collection dialog is displayed.
   
   
4. Configure the collection as described in the following table.                          

   Field	Description
   Collection Name	Specify a unique name for your collection, such as Compliance, MediumValue, or LowValue.
   Hot Storage	Specify the maximum size or percentage of hot storage to use for this collection. The free space available to use for hot storage and the total hot storage is shown next to this field.
   Warm Storage	(Optional) Specify the maximum size or percentage of warm storage to use for this collection. The free space available to use for warm storage and the total warm storage is shown next to this field.
   Cold Storage	(Optional) Specify whether to use cold storage for this collection. If you use cold storage for the collection, logs outside the storage limits are copied to cold storage before they are deleted from hot or warm storage.
   Retention	(Optional) Specify the number of days that logs are retained before they are removed or rolled over to cold storage.  For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first.
   Compression	Specify the type of compression to use for meta and raw logs in the collection. You can compress the meta and raw logs using GZIP or LZMA to save space. GZIP is very fast at compressing and decompressing, but it does not compress as well as LZMA. LZMA offers better compression at a cost of decompression speed (roughly three times slower than GZIP). Compression ratios are highly dependent on your data. The default compression is GZIP.
   Hash	Specify whether to enable or disable hash. When enabled, the hash algorithm is used to verify the data integrity of the files being saved. By default, the only data being hashed is raw logs and the hash files are saved in the same directory as data.

5. Click Save.
   
   Any errors in the collection appear in red text. A dotted underline indicates that a tooltip is available with information about the error. Your collection name appears in red text until at least one retention rule is defined for your collection.
   
   If you have a collection with editing disabled (grayed out), look at the associated tooltip for more information. 

Note: When decreasing collection storage allocations or lowering retention time, it may take several minutes to hours for the data to move and space to become available depending on the amount of moving (rolling) data. The default times are every 20 minutes for a size roll and every six hours for a time roll.

Next Step

Define retention rules for your collections.