Data Privacy: Configure Data Retention

Document created by RSA Information Design and Development on Mar 21, 2017
Version 1Show Document
  • View in full screen mode
  

A Security Analytics user with the role of Administrator can configure Security Analytics to ensure that sensitive data has been removed after a specific retention period, regardless of system ingest rate. For instance, the policy might be to keep packets (both raw data and meta data) for no more than 24 hours, and to keep some logs (both raw data and meta data) for up to seven days. If sensitive data makes its way into another database on the Reporting Engine, Malware Analysis, Event Stream Analysis, and Security Analytics servers, data retention can be managed there as well. The administrator needs to set up each service individually across all Security Analytics components (except Event Stream Analysis) based on policy and data privacy laws.

Sensitive data may also be in cache.

  • Brokers can cache data and this needs to be cleared by configuring an independent rollover and other removal of cache as required. The administrator can configure cache rollover for a Broker by editing the Scheduler file in the Services Config view Files tab.
  • Investigation and the Security Analytics Server cache data, and this is cleared automatically every 24 hours.
  • If the Data Privacy Officer (DPO) exports data, that is the same as saving data on the Security Analytics server in the jobs queue. To clear this data, the administrator or DPO should clean up the jobs queue on a regular basis.

Data Retention

You can schedule a recurring job for Decoder, Log Decoder, and Concentrator services in Security Analytics to check if data is ready to be removed. The Data Retention Scheduler provides a means to configure basic scheduling (see below), and advanced Scheduler settings are also available by editing the Scheduler file in the Services Config view Files tab or the node in the Explorer view.

The Archiver has flexible data storage and retention options. You can place different types of log data into individual collections and manage them separately. These collections enable you to specify how much of the total storage space to use and how many days to store the logs in the collection. You can also determine whether to delete the log data or to move it to offline cold storage after it reaches the maximum specified storage space for the collection.

For example, you can put sensitive information in a collection and configure a limitation on how long to keep it, such as 30 days. To delete the data after 30 days, you would not enable warm or cold storage for that collection. 

Deleting versus Retaining Log Data

Administrators can configure hot, warm, and cold tiered storage on an Archiver. Cold storage contains the oldest log data that is either required for the operation of the business or mandated by regulatory requirements. When a collection reaches its retention limits for hot and warm storage, Security Analytics deletes the log data from hot or warm storage. With cold storage configured, a copy goes into cold storage before the logs are deleted from hot or warm storage. You can choose whether to enable cold storage for each log storage collection. Security Analytics does not manage cold storage. 

Enable or Disable Cold Storage in a Log Storage Collection

When log data in a collection reaches its retention limits for hot and warm storage, you can delete it or move it to offline (cold) storage. 

To enable or disable cold storage in a log retention storage collection on an Archiver:

  1. In the Security Analytics menu, select Administration > Services.
  2. Select the Archiver service and ic-actns.png > View > Config.
  3. Click the Data Retention tab.

    ArcDrTb.png

  4. In the Collections section of the Data Retention tab, select a collection and click .

    The Collection dialog is displayed.

    Note: If the maximum storage size of the collection does not allow full data retention for the retention period specified, Security Analytics deletes the data or it goes to warm or cold storage if specified in the collection.

  5. Enable or disable cold storage:

    • To delete log data when the collection reaches its specified retention limits, clear the Cold Storage checkbox.
    • To move log data to offline storage when the collection reaches its specified retention limits, select the Cold Storage checkbox. 
  6. Click Save.

Configure Log Retention and Storage on an Archiver

To configure log retention and storage on an Archiver, see the Configure Archiver Storage and Log Retention topic in the Archiver Configuration Guide.

Schedule a Recurring Job to Check Data Retention Thresholds

The data retention scheduler configuration ensures that the data residing in the Decoder, Log Decoder, and Concentrator components is deleted after a certain time. For example, data retention on a Decoder might be configured to execute a check every 15 minutes to determine if the specified duration threshold has been met. If the threshold is met, Security Analytics deletes data older then 4 hours in the relevant databases.

Caution: The schedule overwrites any previous schedule and becomes effective immediately. If the retention period is decreased, the data exceeding this retention period is removed.

For a  Decoder, Log Decoder, or Concentrator:

  1. In the Security Analytics menu, select Administration > Services.
  2. In the Services grid, select a Decoder, Log Decoder, or Concentrator service and click  > View > Config.
  3. Click the Data Retention Scheduler tab.

  4. Set the threshold based on the duration of time the data has been stored or the date on which the data was stored. Do one of the following:

    1. To define the duration of time that data can be stored before removal, select Duration, and then specify the number of days (365 maximum), hours (24 maximum), and minutes (60 maximum) that have elapsed since the time stamp on the data.
    2. To define the removal of data based on the date of the timestamp, select Date, and then specify the monthly date and time in the Calendar and Time fields.
      DateThreshold.png
  5. Do one of the following to configure the schedule for checking rollover criteria:

    1. If you want to set a regular interval at which the scheduled database check occurs, select Interval and specify the Hours and Minutes between the scheduled checks.

      RunInterval.png

    2. If you want to set a regular date and time at which the scheduled database check occurs, select Date and Time and specify the system clock time in hh:mm:ss format for the rollover.

      • To specify the day, select Every Day, Weekdays, or Weekends. The Scheduler defaults to Every Day.

        RunDateTim.png

      • To specify a different set of days of the week, select Custom and click on each day on which the database check occurs.

        Caution: The schedule overwrites any previous schedule and becomes effective immediately. If the retention period is decreased, the data exceeding this retention period is removed.

  6. Click Apply to complete the configuration.
You are here
Table of Contents > In-Depth Procedures > Configure Data Retention

Attachments

    Outcomes