Archiver: Additional Procedures 55961

Document created by RSA Information Design and Development on Jul 26, 2016
Version 1Show Document
  • View in full screen mode
 

You can configure an Archiver to utilize tiered storage where the ingested data moves from primary to secondary storage or tier as the data gets older. The primary storage or hot tier will continue to retain the latest data and the secondary storage or warm tier handles the older data. You can schedule the data rollover from primary storage to the secondary storage by size of the database.

 

Note: The tiered storage can be configured only for SA 10.4 and later Archivers.

 

Primary storage has the following tiered level of storage:

  • Tier 0 (Hot) – This contains data that is actively leveraged as part of the business process. Data on this storage is often exposed to reporting and other tasks and has a faster access. This is generally a Direct Access Storage (DAC) or SAN storage.

The secondary storage can have the following tiered levels of storage:

  • Tier 1 (Warm) – This contains the older data aggregated by Archiver. Data on this storage can be leveraged for reporting and other tasks. Data access will not be as fast compared to a Hot tier. This is generally a Network Attached Storage (NAS).
  • Tier 2 (Cold) - This contains the oldest data that is either required for the operation of the business or mandated by regulatory requirements and cannot be accessed quickly. Data on this storage is no longer accessible by the Archiver for reporting and other tasks. However, if you want to access this data, you can restore it to the collections created on workbench service and use for Reporting. This is generally a offline storage like NAS or Scratch Space.

The data can be rolled over in the following ways:

  • Tier 0(Hot) to Tier 2(Cold)
  • Tier 0(Hot) to Tier 1(Warm) to Tier 2(Cold)

For example, if you set the MaxHotSize and MaxWarmSize each to 1 TB. When the data size reaches 1 TB on Hot tier, the oldest data from Hot will be moved to Warm tier and when the data size reaches 1 TB on Warm tier, the oldest data from Warm will be moved to Cold tier.

The format of the data that is presented for archiving on cold tier depends on the configuration. You can tune it by using different format type strings.

Cold directories support the following format type string:

%y - year of the data moved to the cold tier
%m - month of the data
%d - day
%h - hour
%##r - a block of hours for the current day.

For example, if you want three 8 hour blocks, you can set it to %8r. The first 8 hours of the day will return 0, second 8 hours will return 1 and last 8 hours of the day would return 2.

For example, if you configure a Cold tier as below and assume packet, meta, session, and index data are moved to Cold tier on 7/18/2014.

  • Metadb: cold-storage-%y-%m-%d/metadb
  • Packetdb: cold-storage-%y-%m-%d/packetdb
  • Sessiondb: cold-storage-%y-%m-%d/sessiondb
  • Index: cold-storage-%y-%m-%d/index

Archiver will create folders like:

  • cold-storage-2014-7-18/metadb
  • cold-storage-2014-7-18/packetdb
  • cold-storage-2014-7-18/sessiondb
  • cold-storage-2014-7-18/index

You can schedule the data rollover based on the size of the data collected. Once the online retention window for the primary storage expires, the data is rolled over to the secondary storage if it is configured. The data rollover can happen in one of the following ways:

  • Synchronous Rollover - This is configured by setting size values on the configuration for meta, logs/raw, session, and index when you configure the tier storage. The data will be rolled over from the database when the specified size is reached.
  • Asynchronous Rollover - This is configured by setting values of maxSize for Hot tier and maxSize for Warm tier. The data gets rolled over from a Hot tier when it exceeds the value set for maxSize for Hot and from a Warm tier when it exceeds the value set for maxSize for Warm.

Add a Storage Tier

Perform this procedure if you are adding the storage tiers in bulk.

  1. In the Security Analytics menu, select Administration > Services.
  2. Select an Archiver service.
  3. In the Actions column, select View > Config.
  4. Select the Storage tab.
  5. Under Tiered Storage Configuration, click .
    The Add Tier dialog is displayed.
  6. Select the required tier Hot, Warm or Cold.
    An example where Hot tier is selected is shown above.
  7. Provide the path details for Meta, Logs/Raw, Session, and Index.
    An example specifying the path for the Meta is shown in the figure above.
  8. Click Save.
    The storage tier added is displayed in the tiered storage configuration table.
  9. Click Apply.
    A warning dialog is displayed.

  10. Click Yes.
    The service reboots and the storage configuration becomes effective.

Note: Follow the same procedure to add a Warm tier or a Cold tier.

Add a Storage Configuration

Perform this procedure if you are adding the Meta, Packets, Session, or Index individually.

  1. In the Security Analytics menu, select Administration > Services.
  2. Select an Archiver service.
  3. In the Actions column, select View > Config.
  4. Select the Storage tab.
  5. Select one of the tabs Meta, Logs/Raw, Session, or Index depending on the database to be added.
  6. Under Tiered Storage Configuration, click to add a storage configuration.
    An editable row in the table is displayed.

  7. Type the path information for the selected database.
  8. Select the Tier type, Size, and Unit.
  9. Click Update.
  10. Type the Max Size Of A Hot Database and Max Size Of A Warm Database.
    The value of Hot Max Size provided will force the oldest data to be deleted when it reaches this value or moved to warm or cold tier if configured. If the value of Warm Max Size is used then it will force the oldest data to be deleted when it reaches this size or moved to cold tier if configured.
  11. Click Apply.
    A warning dialog is displayed.

  12. Click Yes.
    The service reboots and the storage configuration becomes effective.
You are here: Additional Procedures > Configure Archiver to Use Secondary Long-Term Data Storage

Attachments

    Outcomes