This topic describes the Data Retention tab for an Archiver. Administrators use this tab to define the criteria for log retention and storage.
On the Administration > Services > Config view > Data Retention tab of an Archiver, Administrators can define the criteria for log retention and storage. As an Administrator, you can configure hot, warm, and cold storage as well as multiple storage collections with different locations and criteria for retaining logs. For example, you can create a Compliance collection that stores logs for a specific time period as required by government regulations. You can create another collection that stores low value logs in hot storage with a much shorter retention period. The flexibility of these collections enables you to have significantly less overall storage requirements.
Procedures related to this tab are described in Step 3. Configure Archiver Storage and Log Retention.
This tab has the following sections:
- Total Hot Storage: Enables you to configure the total amount of Hot Tier storage available. You can select or add mount points (paths) for your Hot Tier storage locations. These mount points are attached to fast direct storage, such as Direct-Attached Capacity (DAC) storage and SAN.
- Total Warm Storage: (Optional) Enables you configure the total amount of Warm Tier storage available. You can select or add mount points for your Warm Tier storage locations. These mount points are attached to secondary storage, such as NAS.
- Total Cold Storage: (Optional) Enables you to configure the total amount of Cold Tier storage available. You can add a mount point for a Cold Tier storage location to back up your log files. This mount point is attached to offline storage, such as NAS, or temporary storage before archiving to tape. Security Analytics does not manage cold storage.
- Collections: Enables you to define individual storage collections for different log types. You can specify the maximum size of the Hot and Warm Storage space, whether to use offline storage (Cold Storage), the number of days to retain the logs in the collection, the data compression, and whether to use a hash algorithm to ensure the data integrity of the files being saved.
- Retention Rule: Enables you to define rules for each of your log storage collections. You must define at least one rule for each collection.
To access the Data Retention tab for an Archiver:
- In the Security Analytics menu, select Administration > Services.
- Select an Archiver service and > View > Config.
- In the Services Config view for the service, click the Data Retention tab.
The Data Retention tab for the Archiver is displayed.
Total Hot, Warm, and Cold Storage
The Total Hot Storage section shows the total amount of Hot storage available and the number of hot storage mount points. The Total Hot Storage section shows the total amount of Warm storage available and the number of warm storage mount points. The Total Cold Storage section shows the total amount of Cold storage and the remaining free space available in Cold storage.
Hot, Warm, and Cold Storage Mount Points Dialogs
In the Hot, Warm, and Cold Storage Mount Points dialogs, you can specify the mount points for your storage locations. You can specify portions of this storage to use for your log storage collections.
The following table describes features of the Hot, Warm, and Cold Tier Storage dialogs.
The Collections section lists all of your storage collections along with Total Storage for Hot and Warm Storage.
The following table describes the features of the Collections section. You can hide some of the columns based on your requirements.
|Adds a storage collection. Collection Dialog provides additional details.|
|Removes the selected collection. Deleting the collection permanently removes all stored data from the collection, but the empty data directories remain.|
|Enables you to edit the selected collection. Collection Dialog provides additional details.|
|Refreshes collection information.|
|Selects a collection. For example, you can select a collection for editing or removal.|
|Collection||Shows the name of your collection, such as Default, Compliance, MediumValue, and LowValue. You can create multiple collections with different criteria for retaining logs. If you do not create any collections, the Default collection is used. |
If a collection has errors, the collection name and the columns with errors appear in red text.
|Usage / Hot Storage||Shows the current hot storage usage and the maximum hot storage for the collection. When the size of the logs reach the maximum hot storage amount, the logs are removed or they roll to the next available storage tier (warm or cold).|
|Usage / Warm Storage||Shows the current warm storage usage and the maximum warm storage for the collection. When the size of the logs reach the maximum warm storage amount, the logs are removed or they roll to available cold storage.|
|Cold Storage||Indicates whether cold storage is enabled or disabled. A solid colored green circle indicates that cold storage is enabled (). An blank white circle indicates that cold storage is disabled.|
|Retention||Shows the number of days that logs are retained before being removed or optionally moved to cold storage. No Limit indicates that log retention is not restricted by a specified number of days.|
For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first.
|Velocity (last hour)||Shows the number of logs captured over the last hour.|
|Oldest Date||Shows the date and time of the last log capture.|
|Duration||Shows how may days ago that the last log was captured. For example: 20 days.|
|Compression||Shows the compression type used for the meta and raw data in the collection.|
|Hash||Shows whether hash is enabled or disabled. When enabled, the hash algorithm is used to ensure the data integrity of the files being saved. By default, the only data being hashed is raw logs and the hash files are saved in the same directory as data.|
|# of Rules||Shows the number of rules applied to the collection.|
Define at least one rule for each collection. A collection without any associated rules shows a zero in red text as a warning: The collection name also appears in red text, which indicates an error in the collection.
Caution: If a collection does not have a rule, no logs will ever go into that collection.
|Actions||Enables you to see the rules associated with a collection in the Retention Rule section when you select <actions button> > Select Rules. In the Retention Rule section, you can change the overall priority of the collection rules.|
|Total Storage||Shows the current total hot storage usage and the maximum total hot storage at the bottom of the Usage / Hot Storage column. It also shows the current total warm storage usage and the maximum total warm storage at the bottom of the Usage / Warm Storage column.|
Any errors in the collection appear in red text. A dotted underline indicates that a tooltip is available with information about the error.
Collections that have editing disabled (grayed out) also have tooltips that provide information on the problem.
The Retention Rules section lists all of the retention rules used for your storage collections listed in the order of rule execution.
The following table describes the features of the Retention Rule section.
|Adds a retention rule to use in a storage collection. Rule Definition Dialog provides additional details.|
|Removes the selected retention rule. In order for your log collections to gather and store log data, you must associate them with at least one retention rule.|
|Enables you to edit the selected retention rule. Rule Definition Dialog provides additional details.|
|Refreshes retention rule information.|
|Move Up|| |
Moves the selected retention rule up in the Retention Rule priority list. Retention Rule order is very important. Security Analytics evaluates the the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.
You can also use drag and drop to reorder retention rules.
|Move Down||Moves the selected retention rule down in the Retention Rule priority list. Retention Rule order is very important. Security Analytics executes the the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.|
|Apply||Saves the rule order change.|
|Revert||Reverts the rule order change.|
|Selects or shows a selected retention rule.|
|Order||Shows the order of a rule in the overall list of retention rules.|
|Rule Name||Shows the name of rule, such as ComplianceDevices and GeneralWindowsLogs.|
|Condition||Shows the conditions for the rule. These conditions specify the type of logs to include in the collection. |
Rule and Query Guidelines presents the guidelines for all queries and rule conditions in Security Analytics Core services.
|Collection||Shows Collection name and how many days that the collection is retained. For example: MediumValue (30 Days)|