New Feature: Database Purge

Blog Post created by Nathan Furze

on Sep 28, 2016

Like • Show 3 Likes3
Comment • 12

In the recent RSA Identity Governance and Lifecycle 7.0.1 Service Pack release we have added a database purging feature to remove unnecessary data from production tables. Administrators can schedule the periodic removal of unnecessary data from the system – this data includes application logs, run history & details, workflow events and health snapshot data that are older than 30 days.

Administrators can configure under Admin --> System --> Data Management. Configuration parameters include:

► Database Purging – On/Off

► Duration of Data Purge

► Set scheduled frequency

► The database purging feature is on by default on upgrade to 7.0 SP1

Outcomes

12 Comments

Tim Willemstein Sep 28, 2016 10:02 AM
Is there an overview of what tables are purged? Does this match the database purging as described in one of the KB articles?
Like • Show 1 Like1
Actions
- Nathan Furze @ Tim Willemstein on Oct 14, 2016 12:25 PM
  Do you have a link to that KB? We didn't document the specific tables because they may change over time but the types of data removed would not. To add some more clarity - the types of data purged are
  
  Access Requests Details
  Workflow Events
  Processing Run Details
  Application Logs
  Collection Run History
  Workflow Custom Tasks
  Health Snapshot
  
  Does that help?
  Like • Show 1 Like1
  Actions
  - Tim Willemstein @ Nathan Furze on Oct 17, 2016 1:45 AM
    The following KB articles are the ones I mentioned previously, these are mainly focused on log tables:
    
    000029239 How to Manage Database Logs
    000031223 How do I determine if my Via L&G database is in need of tables being purged
    
    Your list tells me that it is a much welcomed addition to what we had previously. The table I'm most curious about is the ever expanding WP_USER_DATA table, as this one keeps growing with the amount of new workflows that are executed.
    Like • Show 2 Likes2
    Actions
    - Russell Waliszewski @ Tim Willemstein on Oct 17, 2016 2:29 PM
      Access Requests Details
      
      Does include the WP_USER_DATA table. The purge process will remove data from this table which is no longer required by the system.
      3 people found this helpful
      Like • Show 1 Like1
      Actions
  - Andy Cheek @ Nathan Furze on Mar 15, 2017 9:24 AM
    Can't add a comment for some reason so had to reply to this. Is the data purged ALL system data that is no longer required, e.g. entries in various WP_xxxxxI tables that were created while instantiating and executing a workflow but once skipped or complete are no longer relevant?
    
    Or do we have the option to remove what I would call business data? For example - can we clear out all change requests that are more than n years old, say? As far as I understand it there is no concept of/support for archiving business data like this, but we could run reports, keep the output and then purge the data if that was supported.
    Like • Show 1 Like1
    Actions
    - Nathan Furze @ Andy Cheek on Mar 15, 2017 11:54 AM
      So it does not purge ALL the data that is not required but we did an analysis of certain data in tables that we are confident are no longer required that can be removed on a scheduled basis. For the purging feature is either on/off and you can change the schedule but the administrator cannot change what data is purged. The second part of your question does touch upon something we are looking at for a future release to incorporate a process for data archival.
      Like • Show 2 Likes2
      Actions
Razvan Dumitriu Apr 12, 2017 4:58 AM
Hi Nathan
Is the "data that are older than 30 days" filter hardcoded or can be adjusted?
(for example purge 60 or 90 days old data, not 30)

Thank you,

Răzvan
Like • Show 0 Likes0
Actions
- Nathan Furze @ Razvan Dumitriu on Apr 12, 2017 11:03 AM
  Răzvan,
  
  It is not configurable. What happens with the purge - the 1st time that it runs there will be a lot of data to remove - all in scope data that is older than 30 days. Assuming it is scheduled to run on a regular basis - each subsequent purge will only be the small amount of data that is greater than 30 days. Is there a use case you see where you would want data older than 30 days?
  
  Nathan
  Like • Show 1 Like1
  Actions
  - Razvan Dumitriu @ Nathan Furze on Apr 13, 2017 8:39 AM
    Thank you for the prompt answer.
    
    One quick thought was about the "Application Logs" (if we are talking about Admin > System > Logs). In 6.9.1 there is an editable retention value for the database.
    
    Some times the client comes to us after 30 days since an issue occured, and on some environments (UAT or PROD) we do not have (quick) access to the log files, so it is much more easier to filter and get the needed log entries from that section.
    
    Thanks,
    
    Razvan
    Like • Show 0 Likes0
    Actions
Andy Cheek Oct 2, 2017 11:09 AM
Am also curious about that. I can also see possibilities of business (or worse, auditors) looking for evidence of collection, etc. older than 30 days.

To my mind there is a distinction to be made between redundant and historical data, with different approaches to each.

Redundant data, i.e. entries in the various WP_xxxxxI tables created while instantiating and executing workflows which are no longer required/relevant once the workflow node is skipped or complete. Deleting this data has absolutely no impact on the ability of the business/auditors to look at an old change request and see what was executed and the data associated with the request.

Historical data, i.e. entries in logs, collection history is surely a different matter and could depend on individual client's circumstances/requirements? Or are RSA effectively saying - if you want to retain this information for your own purposes then you will need to extract it and keep it yourself?
Like • Show 0 Likes0
Actions
Andy Cheek Oct 2, 2017 11:12 AM
One more question (sorry but this is all new having just upgraded to IGL 702) ...

How can we work out what has been v. what might still be due for purging, especially since I suspect nothing similar has happened for a couple of years (since the previous upgrade)?
- Is the process is working forward from the earliest data to more recent or vice-versa or randomly?
- How does it retain consistency if, as on the first few runs, the process runs out of time?
- Given my previous comment/challenge about possible business/audit evidence requirements should we be concerned about recent data being purged imminently?
Like • Show 0 Likes0
Actions
- Nathan Furze @ Andy Cheek on Oct 6, 2017 12:23 PM
  Andy,
  
  Answers inline. Let me know if this covers your questions.
  
  Is the process is working forward from the earliest data to more recent or vice-versa or randomly? It is moving from the earliest to the most recent. Once it catches up to the current date it will purge data on the configured frequency.
  How does it retain consistency if, as on the first few runs, the process runs out of time? If the purge reaches the maximum run time it will restart on the earliest remaining day. The maximum duration is mostly there for the first few times that the purge runs and once caught up to the scheduled frequency the purges should be a smaller.
  Given my previous comment/challenge about possible business/audit evidence requirements should we be concerned about recent data being purged imminently? The date purge is targeting data that is not needed by the system any longer which may differ from a customer's audit requirements. If there is a requirement to retain this data for audit purposes then a backup would be most appropriate. The data purge feature is an automated, UI driven way to do something that many customers were doing manually. I should add that we are looking at a data archival feature that will address your concern about losing historical data. Data would be archived and then purged from the system.
  
  Nathan
  Like • Show 1 Like1
  Actions

New Feature: Database Purge

Outcomes

Related Content

Recommended Content

Incoming Links