This website uses cookies. By clicking OK, you consent to the use of cookies. Click Here to learn more about how we use cookies.
OK
  • RSA.com
  • Products
    • Archer®
      • Archer®
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Archer® Exchange
      • Training
      • Upcoming Events
      • Videos
    • RSA® Fraud & Risk Intelligence Suite
      • RSA® Fraud & Risk Intelligence Suite
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Web Threat Detection
      • Upcoming Events
      • Videos
    • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Cloud
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Upcoming Patch Content
      • Videos
    • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication Mobile SDK
      • Advisories
      • Events
      • Ideas
      • Knowledge Base
      • Request Access
      • Training
    • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® Adaptive Authentication for eCommerce
      • RSA® Adaptive Authentication for eCommerce
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® FraudAction Services
      • RSA® FraudAction Services
      • Advisories
      • Discussions
      • Documentation
      • Ideas
      • Videos
    • RSA® Web Threat Detection
      • RSA® Web Threat Detection
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Videos
    • RSA NetWitness® Platform
      • RSA NetWitness® Platform
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Integrations
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • RSA NetWitness® Detect AI
      • RSA NetWitness® Detect AI
      • Documentation
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA NetWitness® Investigator
      • RSA NetWitness® Investigator
      • Documentation
      • Download the Client
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA NetWitness® Orchestrator
      • RSA NetWitness® Orchestrator
      • Overview
      • Documentation
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA SecurID® Suite
      • RSA SecurID® Suite
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Knowledge Base
      • Ideas
      • Integrations
      • Training
      • Videos
    • RSA® Identity Governance & Lifecycle
      • RSA® Identity Governance & Lifecycle
      • Advisories
      • Blog
      • Community Exchange
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • RSA SecurID® Access
      • RSA SecurID® Access
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Integrations
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • Other RSA® Products
      • Other RSA® Products
      • RSA® Access Manager
      • RSA® Data Loss Prevention
      • RSA® Digital Certificate Solutions
      • RSA enVision®
      • RSA® Federated Identity Manager
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
  • Resources
    • Advisories
      • Product Advisories on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Hosted
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Product Advisories
    • Blogs
      • Blogs on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Blogs on RSA Link
    • Discussion Forums
      • Discussion Forums
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Discussion Forums on RSA Link
    • Documentation
      • Product Documentation
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Documentation on RSA Link
    • Downloads
      • Product Downloads
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Downloads on RSA Link
    • Ideas
      • Idea Exchange
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Documentation on RSA Link
    • Knowledge Base
      • Knowledge Base
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Knowledge Base Pages on RSA Link
    • Upcoming Events on RSA Link
      • Upcoming Events
    • Videos
      • Videos on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise
      • RSA® Adaptive Authentication On-Premise (Cassandra)
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Videos on RSA Link
  • Support
    • RSA Link Support
      • RSA Link Support
      • News & Announcements
      • Getting Started
      • Support Forum
      • Support Knowledge Base
      • Ideas & Suggestions
    • RSA Product Support
      • RSA Product Support
      • General Security Advisories and Statements
      • Product Life Cycle
      • Support Information
      •  
      •  
      •  
      •  
      •  
  • RSA Ready
  • RSA University
    • Certification Program
      • Certification Program
    • Course Catalogs
      • Course Catalogs
    • On-Demand Subscriptions
      • On-Demand Subscriptions
      • Archer®
      • RSA NetWitness® Platform
      • RSA SecurID® Suite
    • Product Training
      • Product Training
      • Archer®
      • RSA® Fraud & Risk Intelligence Suite
      • RSA® Identity Governance & Lifecycle
      • RSA NeWitness® Platform
      • RSA SecurID® Access
    • Student Resources
      • Student Resources
      • Access On-Demand Learning
      • Access Virtual Labs
      • Contact RSA University
      • Enrollments & Transcripts
      • Frequently Asked Questions
      • Getting Started
      • Learning Modalities
      • Payments & Cancellations
      • Private Training
      • Training Center Locations
      • Training Credits
      • YouTube Channel
    • Upcoming Events
      • Upcoming Events
      • Full Calendar
      • Conferences
      • Live Classroom Training
      • Live Virtual Classroom Training
      • Webinars
Sign In Register Now
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements

RSA Link website migration to new platform is in progress

View Status

RSA NetWitness® Platform Blog

Subscribe to the official RSA NetWitness Platform blog for information about new product features, industry insights, best practices, and more.
  • RSA Link
  • :
  • Products
  • :
  • RSA NetWitness Platform
  • :
  • Blogs
  • :
  • NetWitness Retention Script: Understanding The Nu...

NetWitness Retention Script: Understanding The Numbers

LeonardChvilice
Employee LeonardChvilice
Employee
0 0 549
  • Subscribe to RSS Feed
  • Mark as New
  • Mark as Read
  • Bookmark
  • Subscribe
  • Email to a Friend
  • Printer Friendly Page
  • Report Inappropriate Content
‎2021-01-19 08:19 PM

Overview

If you are looking at retention requirements for compliance, making decisions about the architecture, or to retain a decent investigation history, NetWitness retention is always at the top of these discussions.  As we all find out over time, retention is something that needs to be monitored for trends so informed decisions can be made to meet the corporate or regulatory retention requirements.  This article will shed some light on retention in the NetWitness Platform Systems and demonstrate how to view the retention numbers as a “stack”.  Scroll to the bottom to download the retention script related to this article.

 

Persistence

There are basically two levels of persistence in the NetWitness Platform:

  • Permanent - Final resting place for the data - It is NOT copied to another destination in the platform; i.e Network/Log Decoder raw packets/logs residing on the Network/Log Decoder
  • Temporary - Data that is copied (via aggregation) from this location to another device, i.e. The Concentrator consuming meta and sessions data from a Network/Log Decoder.  The meta and session data is considered temporary as only needs to reside on the Network/Log Decoder long enough to be consumed by the aggregating Concentrator.

 

Database Types

Below are the database types and the index used by the NetWitness Platform

  • PacketDB – Raw captured log/network data
    Present on Log/Network Decoders and Archivers
  • MetaDB – Meta data generated from Log/Network Decoder parsing and processing (App Rules/Feeds)
    Present on Log/Network Decoders, Concentrators and Archivers
  • SessionDB – Data that links the meta data and packet data together into sessions
    Present on Log/Network Decoders, Concentrators and Archivers
  • Index – Not really a database, but provides a method to lookup sessions using meta key values or session ID's.

 

NetWitness Systems

Let's take a look at how the database types, persistence, and the retention requirements relate to the individual NetWitness Systems.  

 

Log/Network Decoder

  • PacketDB
    • Permanent resting place (unless Log Archiver is deployed for Log Decoder)
      • Meet Your Requirement Retention Days
  • MetaDB
    • Temporary resting place
      • Typically like to see ~30 Days Retention
  • SessionDB
    • Temporary resting place
      • Typically like to see ~30 Days Retention

 

Concentrator

  • MetaDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • SessionDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • Index
    • Permanent Resting Place
      • Meet Your Requirement Retention Days

 

Log Archiver

  • PacketDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • MetaDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • SessionDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • Index
    • Permanent Resting Place
      • Meet Your Requirement Retention Days

 

Log Hybrid Retention

  • PacketDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • MetaDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • SessionDB
    • Permanent Resting Place
      • Meet Your Requirement Retention Days
  • Index
    • Permanent Resting Place
      • Meet Your Requirement Retention Days

 

Interpreting The Numbers

Viewing The "Stack"

When examining retention it is best to evaluate the systems as a "stack".  This will assist in viewing the relationship between the capture devices (Network/Log Decoders) and the upstream consumers (Concentrators) of the data in relation to the corporate goal or regulatory requirement.  The image below shows the NetWitness "stacks" in this particular sample architecture.  Each individual stack is separated by the "----------", so we can see that there are 15 stacks, two of which are archivers.

 

The-Full-Estate-Table.png

 

Determine The Retention

Determine if your goals or requirements are met by viewing the "Permanent" retention numbers.  *Note the Archiver permanent numbers are relative to a "Collection" name, there will always be a "default" collection.  If there were other collections, the row for each collection would be the "Permanent" retention numbers. 

 

The-Full-Estate-Table2.png

 

The Retention Script

Attached is the retention script used to provide the outputs shown above.

The script provides output to file in two text formats and outputs to local and/or syslog target:

  • Table for console viewing (text format)
  • CSV for use in other programs
  • Syslog (CEF Format), it can send to a syslog target (unencrypted only - Logger limitation)
  • It writes the output to /var/log/messages (CEF Format)

Requirements

In order for the script file to function as designed, you will need the following prerequisites:

  • Install the NwBackup script, particularly the following scripts:
    • Run get-all-systems11.sh
    • Run ssh-propagate11.sh

Installation

  1. Download the retention script at the end of this article
  2. SSH to the console of the NetwWitness Server (Node Zero)
  3. Login as the "root" user
  4. Create the /root/scripts/admin 
    mkdir /root/scripts
    mkdir /root/scripts/admin
  5. Create the retention directories to store the output files (these are the defaults).
    mkdir /root/retention
    mkdir /root/retention/table
    mkdir /root/retention/csv
  6. Using WinSCP or other client to copy the script to the NetWitness Server (Node Zero) /root/scripts/admin directory
  7. Make the script file executable
    chmod +x /root/scripts/admin/netwitness_retention_csv.sh
  8. Edit the script file variable values to match your output directories, syslog server, and number of history days to keep for the CSV and table text files.

Manual Execution

  1. SSH to the console of the NetwWitness Server (Node Zero)
  2. Login as the "root" user
  3. After you have completed the Installation Steps, type the following command
    /root/scripts/admin/netwitness_retention_csv.sh
  4. You will see ouput to the console similar to below

script-output.png

Crontab Scheduling Instructions

  1. SSH to NetWitness Server (Node-zero)
  2. Login as "root"
  3. Edit crontab
    crontab -e
  4. Add to crontab to execute once every 24 hours at 11pm UTC
    Press the following key
    Insert
    add the following text at the top or the bottom of the file
    ## Retention Script ##
    0 23 * * * /root/scripts/admin/netwitness_retention_csv.sh
    Press
    ESC
    Type the following keys
    :wq
    Press
    ENTER

 

Additional Notes

Since this data is now passed to the /var/log/messages it can ingested by the Log Decoder and partially parsed.  There will be a follow up article on properly parsing it and pulling this data into the Reporting Engine to spot trends.  Due to time constraints I was not able to add it to this article.

netwitness_retention_csv.zip
Labels
  • Resources
Tags (20)
  • Tags:
  • data
  • index
  • meta
  • metadb
  • NetWitness
  • netwitness.
  • numbers
  • NW
  • NWP
  • packet
  • packetdb
  • period
  • resource
  • retention
  • RSA NetWitness
  • RSA NetWitness Platform
  • Script
  • session
  • sessiondb
  • understanding
0 Likes
Share

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

  • Comment
About the Author
Sporting Clays Rock
Latest Articles
  • Network Cloud Visibility with AWS Traffic Mirrorin...
  • Analysing EVTX files in NetWitness through Winlogb...
  • NetWitness Retention Script: Understanding The Nu...
  • Interface Bonding - Putting it all together
  • Using RSA Logs and/or Packets to Send or Receive D...
  • video 35002
  • video 35001
  • Amazon Cloudwatch Event Source Log Configuration G...
Labels
  • Announcements 43
  • Events 2
  • Features 5
  • Resources 43
  • Tutorials 10
  • Use Cases 6
  • Videos 132
Powered by Khoros
  • Products
  • Resources
  • Solutions
  • RSA University
  • Support
  • RSA Labs
  • RSA Ready
  • About RSA Link
  • Terms & Conditions
  • Privacy Statement
  • Provide Feedback
© 2020 RSA Security LLC or its affiliates.
All rights reserved.