NetWitness Retention Script: Understanding The Numbers

[LeonardC]

Overview

If you are looking at retention requirements for compliance, making decisions about the architecture, or to retain a decent investigation history, NetWitness retention is always at the top of these discussions.  As we all find out over time, retention is something that needs to be monitored for trends so informed decisions can be made to meet the corporate or regulatory retention requirements.  This article will shed some light on retention in the NetWitness Platform Systems and demonstrate how to view the retention numbers as a “stack”.  Scroll to the bottom to download the retention script related to this article.

Persistence

There are basically two levels of persistence in the NetWitness Platform:

• Permanent - Final resting place for the data - It is NOT copied to another destination in the platform; i.e Network/Log Decoder raw packets/logs residing on the Network/Log Decoder
• Temporary - Data that is copied (via aggregation) from this location to another device, i.e. The Concentrator consuming meta and sessions data from a Network/Log Decoder.  The meta and session data is considered temporary as only needs to reside on the Network/Log Decoder long enough to be consumed by the aggregating Concentrator.

Database Types

Below are the database types and the index used by the NetWitness Platform

• PacketDB – Raw captured log/network data
  Present on Log/Network Decoders and Archivers
• MetaDB – Meta data generated from Log/Network Decoder parsing and processing (App Rules/Feeds)
  Present on Log/Network Decoders, Concentrators and Archivers
• SessionDB – Data that links the meta data and packet data together into sessions
  Present on Log/Network Decoders, Concentrators and Archivers
• Index – Not really a database, but provides a method to lookup sessions using meta key values or session ID's.

NetWitness Systems

Let's take a look at how the database types, persistence, and the retention requirements relate to the individual NetWitness Systems.  

Log/Network Decoder

• PacketDB
  • Permanent resting place (unless Log Archiver is deployed for Log Decoder)
    • Meet Your Requirement Retention Days
• MetaDB
  • Temporary resting place
    • Typically like to see ~30 Days Retention

• SessionDB
  • Temporary resting place
    • Typically like to see ~30 Days Retention

Concentrator

• MetaDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• SessionDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• Index
  • Permanent Resting Place
    • Meet Your Requirement Retention Days

Log Archiver

• PacketDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• MetaDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• SessionDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• Index
  • Permanent Resting Place
    • Meet Your Requirement Retention Days

Log Hybrid Retention

• PacketDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• MetaDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• SessionDB
  • Permanent Resting Place
    • Meet Your Requirement Retention Days
• Index
  • Permanent Resting Place
    • Meet Your Requirement Retention Days

Interpreting The Numbers

Viewing The "Stack"

When examining retention it is best to evaluate the systems as a "stack".  This will assist in viewing the relationship between the capture devices (Network/Log Decoders) and the upstream consumers (Concentrators) of the data in relation to the corporate goal or regulatory requirement.  The image below shows the NetWitness "stacks" in this particular sample architecture.  Each individual stack is separated by the "----------", so we can see that there are 15 stacks, two of which are archivers.

Determine The Retention

Determine if your goals or requirements are met by viewing the "Permanent" retention numbers.  *Note the Archiver permanent numbers are relative to a "Collection" name, there will always be a "default" collection.  If there were other collections, the row for each collection would be the "Permanent" retention numbers. 

The Retention Script

Attached is the retention script used to provide the outputs shown above.

The script provides output to file in two text formats and outputs to local and/or syslog target:

• Table for console viewing (text format)
• CSV for use in other programs
• Syslog (CEF Format), it can send to a syslog target (unencrypted only - Logger limitation)
• It writes the output to /var/log/messages (CEF Format)

Requirements

In order for the script file to function as designed, you will need the following prerequisites:

• Install the NwBackup script, particularly the following scripts:
  • Run get-all-systems11.sh
  • Run ssh-propagate11.sh

Installation

1. Download the retention script at the end of this article
2. SSH to the console of the NetwWitness Server (Node Zero)
3. Login as the "root" user
4. Create the /root/scripts/admin 
   mkdir /root/scripts
   mkdir /root/scripts/admin
5. Create the retention directories to store the output files (these are the defaults).
   mkdir /root/retention
   mkdir /root/retention/table
   mkdir /root/retention/csv
6. Using WinSCP or other client to copy the script to the NetWitness Server (Node Zero) /root/scripts/admin directory
7. Make the script file executable
   chmod +x /root/scripts/admin/netwitness_retention_csv.sh
8. Edit the script file variable values to match your output directories, syslog server, and number of history days to keep for the CSV and table text files.

Manual Execution

1. SSH to the console of the NetwWitness Server (Node Zero)
2. Login as the "root" user
3. After you have completed the Installation Steps, type the following command
   /root/scripts/admin/netwitness_retention_csv.sh
4. You will see ouput to the console similar to below

Crontab Scheduling Instructions

1. SSH to NetWitness Server (Node-zero)
2. Login as "root"
3. Edit crontab
   crontab -e
4. Add to crontab to execute once every 24 hours at 11pm UTC
   Press the following key
   Insert
   add the following text at the top or the bottom of the file
   ## Retention Script ##
   0 23 * * * /root/scripts/admin/netwitness_retention_csv.sh
   Press
   ESC
   Type the following keys
   :wq
   Press
   ENTER

Additional Notes

Since this data is now passed to the /var/log/messages it can ingested by the Log Decoder and partially parsed.  There will be a follow up article on properly parsing it and pulling this data into the Reporting Engine to spot trends.  Due to time constraints I was not able to add it to this article.

netwitness_retention_csv.zip

[JeremyKerwin]

The hosts that I've highlighted in the red box are endpoint log hybrids, shouldn't they be containing all the services like Log Decoder?

[JeremyKerwin]

Should the output be showing that for the Endpoint and Log Hybrids that they are consuming from 127.0.0.1. That seems like a mistake or something wrong because they are also only showing data from the Concentrator and not the Log Decoder service.

[JohnSnider]

For your endpoint server(s) what does the entry look like in the /var/netwitness/nw-backup/all-systems file? (specifically the DeviceType column #1, is this a "Native" EndpointLogHybrid, or did you "add" Endpoint to an existing LogHybrid?  If the latter, the DeviceType would show something like:

Endpoint+LogHybrid,hostname,ipaddress,uuid,serialnum

The get-all-systems11.sh script (mine) was updated to handle these service add-ons, and that broke Leonard's script, but I've sent him the fix, so hopefully he will post the updated script shortly.

[LeonardC]

Yes, please send me the all-systems file entry for the systems in question, so that I may confirm the issue is resolved in the new version.

[JeremyKerwin]

The all-systems file for the Hybrid's looks like this.

I have 2 endpoint log hybrids and 1 log hybrid, all three aren't showing the log decoder

EndpointLogHybrid,<HOSTNAME>,x.x.x.x,b6da819a-30b3-4d07-9acb-065af41571c1,VMware-4226d35a7196061c-4a1b9a36d4563858
EndpointLogHybrid,<HOSTNAME>,x.x.x.,6d035a6-b1bd-4cc8-a97c-c29aaa0bffb5,VMware-564d4e05711df378-9e1dd861935dafd0
LogHybrid,<HOSTNAME>,x.x.x.x,2fa919e6-a0d2-4a7b-9ffd-53808cd1be1a,VMware-4210931ddb85c9fd-481d2f78ce9f53fa

[LeonardC]

I have posted a newer version of the script v1.25 to handle the formatting of the new all-systems file that John was talking about.  But, I think this is an issue with the 127.0.0.1 address used as the Log Decoder aggregation IP, as the NwServer cannot remotely connect to the 127.0.0.1 address on the Log Decoder.  Your aggregation addresses in the Concentrator should be the actual IP address of the appliance, not 127.0.0.1.  The script executes on the NwServer so when the script pulls down the 127.0.0.1 it is trying to connect to the local loopback 127.0.0.1 on the NwServer so that is why your Log Decoder stats are empty.

[JeremyKerwin]

I've just run the v1.25 script and now I'm getting different errors.

The aggreation IPs are not 127.0.0.1, they are the actual IP addresses of the hosts.

[LeonardC]

I have seen this before on another system but was unable to determine root cause, as I did not have access to the system.  The addresses were all correct ( not 127.0.0.1) but when pulling the addresses via NwConsole it would come back as 127.0.0.1 instead of the proper expected address.  Has this system been upgraded from 10.x by chance?  Or perhaps build on the earlier versions of 11.x?

[JeremyKerwin]

This system has been upgraded from version 10.x yes

[LeonardC]

Try this command from the NWServer Console (Head unit) and see if you get 127.0.0.1 as a return.

NwConsole -q -c tlogin server="< YOUR LOGHYBRID IP HERE>" port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node.pem -c cd /concentrator/devices/ -c ls

My output from my 11.5.2 system is shown below:

2393e774-7f25-49df-a346-20ccc0914503:56002/ (10.0.0.48:56002)

[JeremyKerwin]

That command did return 127.0.0.1 

b6da819a-30b3-4d07-9acb-065af41571c1:56002/ (127.0.0.1:56002)
2fa919e6-a0d2-4a7b-9ffd-53808cd1be1a:56002/ (127.0.0.1:56002)
26d035a6-b1bd-4cc8-a97c-c29aaa0bffb5:56002/ (127.0.0.1:56002)

[LeonardC]

Let's confirm this by looking in "Explore" for the Concentrator service.

If the device(s) that are listed are NOT 127.0.0.1 then this behavior matches something I have seen on one other system but was unable determine root cause due to limited access to the system.

Also what "Host" and "IP" address is shown on your "HOSTS" page for that device?

[JeremyKerwin]

I've checked the explore section for the concentrators and they are all showing their respective IP addresses and not 127.0.0.1

The Host and IP address under Admin > Hosts is the IP address as well.

[LeonardC]

Ok, thanks for the info, I will need to modify the script to try and use another method to get around this issue.  I will post a new version once it is complete.

One quick question for verification.  In the /var/netwitness/nw-backup/all-systems file, are the proper IP addresses listed (Not 127.0.0.1)?

[JeremyKerwin]

Yep, confirmed the /var/netwitness/nw-backup/all-systems file is showing all IP addresses and no 127.0.0.1

[LeonardC]

A new version of the script has been posted that should resolve your issue.  Please download and give it a try.

[JeremyKerwin]

Hi Leonard, I've download the updated script and can confirm that it now works. Thanks very much for your work in getting this fixed for my use case 🙂 

[LeonardC]

You are welcome and I'm glad to hear it is working for you now.  Thanks for the confirmation.

[JeremyKerwin]

Hi @LeonardC  you mentioned at the bottom of your original post that there might be a follow up post on ingesting these results into the reporting engine.

I was just curious if that might be coming soon 🙂 

Thanks.

[LeonardC]

I will have some time coming up in June that I can work on getting the article and content posted.

Thanks..

[JeremyKerwin]

Thank you.