000029446 - How to use the NwConsole and dbcheck to validate databases on an RSA NetWitness or Security Analytics Appliance.

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000029446
Applies ToRSA Product Set: Security Analytics and NextGen

RSA Product/Service Type: Security Analysis and NextGen Core Services

RSA Version/Condition: NextGen 9.8.5.x, and Security Analytics 10.0 and newer

Platform: CentOS

O/S Version: 5 or 6

 
IssueThe NextGen or Security Analytics core appliance experiences Initialization Errors when the service starts and the service fails to start or capture or aggregation fail to start.  
This may be due to corrupted meta, session or packet database files on the core appliance.  
Use the steps in this article to identify and fix corrupted database files using the dbcheck feature of the NwConsole tool. 
TasksThe NwConsole tool offers the dbcheck utility to verify the structure and integrity of database files on Linux based NextGen and Security Analytics core appliances.

Note: Core NetWitness and Security Analytics appliances run consistency checks when the services start up in NextGen 9.7.5.10 & 9.8.5.1 and all versions of Security Analytics.  In most cases, database corruption is detected and resolved during service initialization. 
ResolutionBackground
  • The Decoder/LogDecoder service has the following databases: sessiondb (nwsdb), metadb (nwmdb), packetdb (nwpdb) and NwServerLog (log)
  • The Concentrator service has the following databases: sessiondb (nwsdb) & metadb (nwmdb) and NwServerLog (log)
  • The Appliance service has the following databases: NwServerLog (log)
  • 9.8 also adds: statdb for appliance and service (statsdb)
Note: Appliances DACs may have multiple packetdb file systems. The first DAC will be found /var/netwitness/decoder/packetdb0.  Subsequent DACs will be found at /var/netwitness/decoder/packetdb1, etc. 
Prior to running dbcheck, you should assess the contents of /var/netwitness to determine which database files you want to verify.
Running NwConsole and dbcheck
  1. SSH to the appliance as root
  2. Stop services
    For CentOS 5.x-based appliances
For a decoder appliance:
monit stop nwdecoder
For a concentrator appliance:
monit stop nwconcentrator
For a hybrid appliance:
monit stop nwdecoder; monit stop nwconcentrator

For CentOS 6.x-based appliances

For a decoder appliance:
stop nwdecoder
For a concentrator appliance:
stop nwconcentrator
For a hybrid appliance:
stop nwdecoder; stop nwconcentrator

If you don't stop the service, you may receive the following error when attempting to run dbcheck on the latest db file:
fcntl failed: fd: 4 action: F_SETLK [OS Error: (11) - Resource temporarily unavailable]

  1. Run NwConsole
NwConsole

  1. Run dbcheck
Specific examples are provided below.  The syntax for the command is:
dbcheck [-header] [-autofix] [-chatty] [-dump] <filename or pathname, wildcards accepted>

  1. Exit NwConsole
Exit

  1. Restart services on the appliance
For CentOS 5.x-based appliances

For a decoder appliance:
monit start nwdecoder
For a concentrator appliance:
monit start nwconcentrator
For a hybrid appliance:
monit start nwdecoder; monit start nwconcentrator

For CentOS 6.x-based appliances

For a decoder appliance:
start nwdecoder
For a concentrator appliance:
Start nwconcentrator
For a hybrid appliance:
start nwdecoder; start nwconcentrator

 
Sample Output from dbcheck command
The output of the dbcheck command will look something like the following. 
---------------------------------------------
PERFORMING TEST: DB INTEGRITY
---------------------------------------------
Verifying file /var/netwitness/decoder/packetdb/packet-000000002.nwpdb
The index flags are 0, which translates to INDEX_64, fidelity is 1
The index file /var/netwitness/decoder/packetdb/packet-000000002.nwpdbindex does not contain a user defined header.
The index contains offsets for 3095886 objects.
The file "/var/netwitness/decoder/packetdb/packet-000000002.nwpdb" has a magic number of 1884076754 (0x704CBAD2).
Version is 8.
The file header size is 20.  File creation date from header is 2015-Jan-20 02:48:08 UTC.
The file flags value is 4128: . Second file flags is 1.
This is a VARIABLE object size database file.
The file flags value is 4128: . Second file flags is 1.
This is a VARIABLE object size database file.This file does not contain a serializer header.
The user defined header length is 8.
Header Dump:
00000000 DC C7 F9 00 00 00 00 00                          [........]
Header value as a unsigned 64 bit int: 16369628
 
0%  Object 0 has a size of 765 at offset 48.
1%  Object 31240 has a size of 1552 at offset 7774079.
2%  Object 60383 has a size of 781 at offset 15547386.
3%  Object 90369 has a size of 112 at offset 23320313.
[Output Snipped]
99%  Object 3066707 has a size of 104 at offset 769566957.
The object store contains 3095886 objects.
100%  File scan complete.  Total file size is 741.32 MB
File is valid
Example commands for running dbcheck on a decoder
Test specific databse files:
dbcheck -autofix /var/netwitness/decoder/sessiondb/session-000003057.nwsdb
dbcheck -autofix /var/netwitness/decoder/metadb/meta-000006446.nwmdb
dbcheck -autofix /var/netwitness/decoder/packetdb/packet-000102363.nwpdb
Test multiple files with wildcards:
Check and automatically fix the last 100 packetdb files:
dbcheck –autofix /var/netwitness/decoder/packetdb/packet-0000000??.nwdb
Check and automatically fix the last 1000 packetdb files:
dbcheck –autofix /var/netwitness/decoder/packetdb/packet-000000???.nwdb
 

Attachments

    Outcomes