RSA NetWitness Storage Retention Script

Blog Post created by Naushad Kasu

on Jan 24, 2019

Like • Show 10 Likes10
Comment • 7

Although the RSA NetWitness platform gives administrators visibility into system metrics through the Health & Wellness Systems Stats Browser, we currently do not have a method to see all storage / retention across our deployment in a single instance or view.

Below you will find several scripts that will help us gain this visibility quickly and easily.

Update: Please grab the latest version of the script, some bugs were discovered that were fixed.

How It Works:

1. Dependency: get-all-systems.sh (attached) both v10 and v11 version for your particular environment. Please run this script prior to running the get-retention.py as it requires the 'all-systems' file which contains all of your appliances & services.

2. We then read through the all-systems file and look for services that have retention e.g. EndpointLogHybrid, EndpointHybrid, LogHybrid, LogDecoder, Decoder, Concentrator, Archiver.

3. Finally we use the 'tlogin' functionality of NwConsole to allow cert-based authentication, thus, no need to run this script with username/password as input to pull database statistics and output the retention (in days) for that particular service.

Instructions:

1. Run ./get-all-systems_v10.sh (for 10.x systems) or ./get-all-systems_v11.sh (for 11.x systems)

2. Run ./get-retention.py (without any arguments). This MUST be run from Puppetmaster (v10) or Node0 (v11).

Sample Run:

Please feel free to provide feedback, bug reports etc...

Attachments

Outcomes

Marinos Roussos Jan 22, 2019 9:21 AM

Great idea and effort but a shame that there has to be unofficial and unsupported scripts for eps, storage retention and a bunch of other basic things that should have been on Health and wellness since 10.2.

I will not bother raising an rfe because there is no point nor there is any hope with the process. Basic SIEM features are still basic features whether customers have asked for them or not.

1 person found this helpful

Actions

David Waugh @ Marinos Roussos on Jan 24, 2019 5:21 AM

I agree with Marinos.

Shouldn't this basic functionality be in the GUI?

Unless I am mistaken the get-allsystems.sh script still requires that you allow root login to all your systems, rather than first logging in as a normal user and then escalating your privileges. What about environments where direct login as root is not allowed?

Automatic Backup and the ability to restore functionality should be in the GUI.

Actions

Joshua Randall

@ David Waugh on Jan 24, 2019 1:48 PM

I'm certainly not going to disagree about the need for this in the UI, but I do want to point out that direct root login is not required or used in 11.x for the get-allsystems.sh script. 11.x uses SaltStack, which enables PKI-based auth between the salt master (AKA admin server; AKA node0) and any salt minion (everything that's not the salt master).

Eric Partington wrote a post a few months back with some examples of some of this functionality (Gathering Stats with Salt - BIOS/iDRAC/PERC Edition) and if you have an 11.x stack in a test or pre-prod environment I'd encourage you to check it out.

Actions

Marinos Roussos @ Joshua Randall on Jan 24, 2019 2:19 PM

Again, good effort and idea but it’s still an unofficial process that requires additional software to be installed, making it officially unsupported. This kinda defeats the purpose yet again.

I find it a little ironic that we are been persuaded from the vendor to use unsupported software and processes on the SIEM as the main solution and not as a workaround.

I’m not saying I wouldn’t try it in my lab or even installing some of these scripts in production but it’s not the way forward for a company that makes banners to celebrate that Gardner named them a leader!

Netwitness is not an open source software and should be treated as such.

Sent from my iPhone

Actions

Sean Griesheimer

@ Marinos Roussos on Jan 28, 2019 12:09 PM

Food for thought: these metrics (and a million more) are all available in the Health & Wellness -> System Stats Browser page, albeit a little more spread out. Here are the "Capture Packet Rate" stats:

Note the graph icon on the right can show that data over time, for any time range you want/need, and that is available for most stats (the exceptions being stats where a graph doesn't make sense, like date-based stats). All of those can be queried via API as well. I agree that making all of this more elegant and available through the UI is important - and I'm advocating for it - but having it available via a simple script that leverages baked-in functionality is EXCELLENT!

Actions

Marinos Roussos @ Sean Griesheimer on Jan 29, 2019 4:26 AM

Sure you have all these stats in H&W and in API which makes it sound more silly that users can’t pull stuff from API and put them on a dashboard.

I believe 10s of customers have been asking for a single pane of glass for monitoring the Netwitness kit for years but were ignored and now we are having discussions about why a lesser feature (or lack of) is great.

In terms of dashboards and stat visibility this product hasn’t had many if any changes since 4-5 years ago so let’s better not talk about it until there is something new.

Maybe v12 will have modular dashboards!

Sent from my iPhone

Actions

Jonathan Saxon

Feb 28, 2019 2:51 PM

Thanks! I used to advise customers to do this with a few simple CLI commands but your script is far more convenient. This shows the power of the API and the power of the community where customers and RSA staff can work together to meet real-world, daily, practical needs. Keeping an eye on retention will help a lot of customers.

Actions

7 Comments

Marinos Roussos Jan 22, 2019 9:21 AM
Great idea and effort but a shame that there has to be unofficial and unsupported scripts for eps, storage retention and a bunch of other basic things that should have been on Health and wellness since 10.2.
I will not bother raising an rfe because there is no point nor there is any hope with the process. Basic SIEM features are still basic features whether customers have asked for them or not.
1 person found this helpful
Like • Show 1 Like1
Actions
- David Waugh @ Marinos Roussos on Jan 24, 2019 5:21 AM
  I agree with Marinos.
  Shouldn't this basic functionality be in the GUI?
  
  Unless I am mistaken the get-allsystems.sh script still requires that you allow root login to all your systems, rather than first logging in as a normal user and then escalating your privileges. What about environments where direct login as root is not allowed?
  
  Automatic Backup and the ability to restore functionality should be in the GUI.
  Like • Show 1 Like1
  Actions
  - Joshua Randall @ David Waugh on Jan 24, 2019 1:48 PM
    I'm certainly not going to disagree about the need for this in the UI, but I do want to point out that direct root login is not required or used in 11.x for the get-allsystems.sh script. 11.x uses SaltStack, which enables PKI-based auth between the salt master (AKA admin server; AKA node0) and any salt minion (everything that's not the salt master).
    
    Eric Partington wrote a post a few months back with some examples of some of this functionality (Gathering Stats with Salt - BIOS/iDRAC/PERC Edition) and if you have an 11.x stack in a test or pre-prod environment I'd encourage you to check it out.
    Like • Show 1 Like1
    Actions
    - Marinos Roussos @ Joshua Randall on Jan 24, 2019 2:19 PM
      Again, good effort and idea but it’s still an unofficial process that requires additional software to be installed, making it officially unsupported. This kinda defeats the purpose yet again.
      
      I find it a little ironic that we are been persuaded from the vendor to use unsupported software and processes on the SIEM as the main solution and not as a workaround.
      
      I’m not saying I wouldn’t try it in my lab or even installing some of these scripts in production but it’s not the way forward for a company that makes banners to celebrate that Gardner named them a leader!
      
      Netwitness is not an open source software and should be treated as such.
      
      Sent from my iPhone
      Like • Show 0 Likes0
      Actions
      - Sean Griesheimer @ Marinos Roussos on Jan 28, 2019 12:09 PM
        Food for thought: these metrics (and a million more) are all available in the Health & Wellness -> System Stats Browser page, albeit a little more spread out. Here are the "Capture Packet Rate" stats:
        
        Note the graph icon on the right can show that data over time, for any time range you want/need, and that is available for most stats (the exceptions being stats where a graph doesn't make sense, like date-based stats). All of those can be queried via API as well. I agree that making all of this more elegant and available through the UI is important - and I'm advocating for it - but having it available via a simple script that leverages baked-in functionality is EXCELLENT!
        Like • Show 0 Likes0
        Actions
        Marinos Roussos @ Sean Griesheimer on Jan 29, 2019 4:26 AM
        Sure you have all these stats in H&W and in API which makes it sound more silly that users can’t pull stuff from API and put them on a dashboard.
        
        I believe 10s of customers have been asking for a single pane of glass for monitoring the Netwitness kit for years but were ignored and now we are having discussions about why a lesser feature (or lack of) is great.
        
        In terms of dashboards and stat visibility this product hasn’t had many if any changes since 4-5 years ago so let’s better not talk about it until there is something new.
        Maybe v12 will have modular dashboards!
        
        Sent from my iPhone
        Like • Show 0 Likes0
        Actions
Jonathan Saxon Feb 28, 2019 2:51 PM
Thanks! I used to advise customers to do this with a few simple CLI commands but your script is far more convenient. This shows the power of the API and the power of the community where customers and RSA staff can work together to meet real-world, daily, practical needs. Keeping an eye on retention will help a lot of customers.
Like • Show 1 Like1
Actions

RSA NetWitness Storage Retention Script

How It Works:

Instructions:

Attachments

Outcomes

Related Content

Recommended Content

Incoming Links