000017619 - JBOD chain settings may make some session data unavailable in RSA NetWitness NextGen 9.5

Document created by RSA Customer Support Employee on Jan 10, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000017619
Applies ToRSA NetWitness NextGen
RSA NetWitness NextGen 9.5
IssueJBOD chain settings may make some session data unavailable in RSA NetWitness NextGen 9.5.
NetWitness provides a script in the knowledgebase article How to install a JBOD on an RSA NetWitness NextGen 9.5 appliance that assists in the configuration changes necessary to add a JBOD to a Decoder.  An earlier version of this script (file date prior to March 28th, 2011) did not add chain database space to each JBOD resulting in a potentially too-small chain database.  Chain files are rolled out oldest-first like all other database-file-types in NextGen system.  When allotted space for the chain database reaches maximum, then the earliest chain files are removed.  In some cases (when more than 24TB of packet space has been added) this can lead to session data still being present on a Decoder but unavailable for Investigator drill and session recreation due to a missing chain file for that session data.
ResolutionIs This Applicable to Your Decoder?

In this case an additional 24TB would be figured like this:

1 JBOD with 12 2TB drives or
2 JBODs with 12 1TB drives

If your Decoder's additional JBOD space meets or exceeds this amount, you may be affected and should proceed, otherwise no action is required.

Sizing The New Chain Database Files

You will need to designate an additional 5% of each JBOD's disk space for chain database storage.  Since the packet data is set to occupy 90% of the partition already an additional 5% will not cause any space issues.  You will need to locate your JBOD partitions.  Each JBOD partition will end in "decoder" followed by a numeral.  The numeral will increment by "1" starting with "0".  For example if you have three JBODs, your JBOD partitions are:

/var/netwitness/decoder0
/var/netwitness/decoder1
/var/netwitness/decoder2

JBODs can come in different sizes so you'll want to note the size of each partition separately and record the value.  List all your partitions in megabytes by issuing the following command:

df -m /var/netwitness/decoder*

Look for the JBOD partitions.  "/var/netwitness/decoder" is on-board storage and will already have a chain database and can be skipped.  Below is an example with three JBODs each having 12 2TB drives each (and therefor affected):

/dev/mapper/decoder0-packetdb
                       19922944
M   17825792M  1992294.4M  90% /var/netwitness/decoder0
/dev/mapper/decoder1-packetdb
                       9961472
M     8912896M   996147.2M  90% /var/netwitness/decoder1
/dev/mapper/decoder2-packetdb
                       19922944M  
17825792M  1992294.4M  90% /var/netwitness/decoder2

For each partition, take the total partition size which is the first of three numbers ending in "M" and multiple it by .05 to calculate 5% of the partition volume for chain storage.

In the example above take 19922944M * .05 to give you 996147.2M (round down to the nearest whole number).  This gives you the value for the first JBOD.  If your JBODs are different sizes like in the example, you would perform the calculation for each JBOD and record its value.


Add The Chain Database to Each JBOD

Now that you know how big the new chain database should be for each JBOD, you'll need to add it to the Decoder's configuration.

Connect to the Decoder's Decoder service on port 50004 using NetWitness Administrator.  Once connected, locate the Decoder's Decoder service in the list of appliances and services in the left pane.  Right-click on it and select "Explorer".

Navigate to: Decoder > Database > Config and locate the value "chain.dir".

For the example above you would expect to see something similar to:

/var/netwitness/decoder/packetdb=584530

For each JBOD add an additional value separated by a semicolon ";".  Any values in addition to the original value require a double equals sign "==".

In our example the new value would be:

/var/netwitness/decoder/packetdb=548530;/var/netwitness/decoder0/packetdb==996147;/var/netwitness/decoder1/packetdb==498073;/var/netwitness/decoder2/packetdb==996147

Remove Packets That Are No Longer Referenced by the Chain Database

Since some packets on the appliance no longer have corresponding chain database files, they are no longer accessible by the system and should be removed.  First locate the oldest chain database file by noting the date on the file returned after issuing this command:

ls /var/netwitness/decoder/packetdb -l -1t *.nwcdb|tail -1

Example result:

-rw------- 1 root root 100663640 2011-02-22 14:37 chain-000000001.nwcdb

In the example above, the oldest chain database file dates from February 22nd, 2011.  Any packet data older than that will not be accessible by the system and will need to be rolled out.  Perform a "time roll" to remove any packet data older than this date.

For example, lets say the date in the example was 21 days ago.  Launch NetWitness Administrator.  Connect to this Decoder's Decoder service and click on the "Console" tab.  Issue the following command:

/database msg=timeroll params="days=21 type=session,chain,packet,meta"

This would remove all data older than 21 days.  With the new settings for the chain database in place, chain databases should now be able to go back as far as packet data so that all data on your Decoder going forward will be available and all data newer than 21 days will continue to be available.


Data Reset Option:

Doing a time roll may leave considerable meta on your Concentrators with no associated session data.  If you experience any instability or issues following the time roll you may wish to reset your systems by doing a data reset at the Decoder/Concentrator/Broker levels.  Please be aware this will remove all packet/session/meta/index/chain data from any appliance that is data reset. 

For Decoders, refer to the knowledgbase article How to perform a data reset on an RSA NetWitness decoder.
For Concentrators/Brokers, refer to the knowledgebase article How to perform a data reset on an RSA NetWitness concentrator or broker appliance.
Legacy Article IDa59751

Attachments

    Outcomes