000032720 - RSA NetWitness Logs & Network Health & Wellness indicates /var/netwitness partition is at 100% utilization

Document created by RSA Customer Support Employee on Jun 30, 2016•Last modified by RSA Customer Support on Apr 18, 2019
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000032720
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Core Appliance
RSA Version/Condition: 10.6.x, 11.x
Platform: CentOS
O/S Version: 6, 7
 
IssueNetWitness Health & Wellness indicates the /var/netwitness partition is at 100% utilization, but you cannot identify the files that are taking up the space on the partition.  

It is possible that a core service may start before a mount point has mounted.  

This can create directories and files on a filesystem other than what is intended, and it can result in the filesystem filling and reporting as full in Health & Wellness and from the Linux command line.

You may see something like the following:
 

[root@PacketDecoder01 netwitness]# df -hP
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root         7.8G  4.7G  2.8G  64% /
tmpfs                                48G     0   48G   0% /dev/shm
/dev/sda1                           496M   62M  409M  14% /boot
/dev/mapper/VolGroup00-usrhome      3.9G  8.1M  3.7G   1% /home
/dev/mapper/VolGroup00-opt          9.8G   53M  9.2G   1% /opt
/dev/mapper/VolGroup00-tmp           20G   44M   19G   1% /tmp
/dev/mapper/VolGroup00-var          7.8G   97M  7.3G   2% /var
/dev/mapper/VolGroup00-rabmq         20G   37M   20G   1% /var/lib/rabbitmq
/dev/mapper/VolGroup00-varlog       9.8G  369M  8.9G   4% /var/log
/dev/mapper/VolGroup00-nwhome        30G   30G   24K 100% /var/netwitness
/dev/mapper/VolGroup01-warec        400G  603M  400G   1% /var/netwitness/warehouseconnector
/dev/mapper/VolGroup00-vartmp       3.9G  8.1M  3.7G   1% /var/tmp
/dev/mapper/decodersmall-decoroot    10G  2.6G  7.5G  26% /var/netwitness/decoder
/dev/mapper/decodersmall-index       30G   69M   30G   1% /var/netwitness/decoder/index
/dev/mapper/decodersmall-metadb     5.2T  469G  4.7T   9% /var/netwitness/decoder/metadb
/dev/mapper/decodersmall-sessiondb  278G   11G  267G   4% /var/netwitness/decoder/sessiondb
/dev/mapper/decoder-packetdb         28T   16T   12T  58% /var/netwitness/decoder/packetdb


In this example the /var/netwitness filesystem appears to be full, but the NetWitness Decoder service continues to run and the /var/netwitness/decoder/packetdb filesystem is less than 60% full.

When examining the contents under the /var/netwitness mount something like the following shows:
 

[root@PacketDecoder01 ~]#  ls -lah /var/netwitness
total 8.0K
drwxr-xr-x.    5  root root   85 Nov  9 08:36 .
drwxr-xr-x.   21 root root  4.0K Dec 21 13:06 ..
drwxr-xr-x.    3 root root    19 Nov  9 08:36 appliance
drwxr-xr-x.    9 root root    98 Jan 27 12:44 decoder
-rw-------.    1 root root    12 Mar  7 19:55 NwDecoder.persist
drwxr-xr-x.    2 root root     6 Nov  6 09:41 warehouseconnector


In other words, no files are observed under the /var/netwitness mount that could be consuming nearly 30GB of disk space.
CauseIn certain circumstances, a service may start capturing or aggregating data before the specified filesystem is mounted.

When this happens, files may be stored under the /var/netwitness mount instead of the filesystem that would normally be mounted in this mount point.

Then, at a later time, when you mount a filesystem on a directory /mount-point, you can no longer access or view the files under the /mount-point directly. They still exist, but the /mount-point now refers to the root of the mounted filesystem, not to the directory that served as a mount point, so the contents of this directory cannot be accessed, at least in this way.

This effectively "hides" or "loses" the files that were previously and erroneously created directly under the filesystem, and makes the filesystem full even though you can't see the files. 
 
ResolutionTo reveal the hidden files, stop the NetWitness services such as nwapliance and nwdecoder then unmount all decoder filesystems using these commands: 
 

stop nwdecoder
stop nwappliance
umount /var/netwitness/decoder/sessiondb
umount /var/netwitness/decoder/index
umount /var/netwitness/decoder/metadb
umount /var/netwitness/decoder/packetdb
umount /var/netwitness/warehouseconnector
umount /var/netwitness/decoder


Change directories to the /var/netwitness folder and look for any files that remain.  These files are almost certainly unwanted files that were accidentally created at some point in time and should be the files consuming the space on the filesystem.

Once you have removed the unwanted files, mount all filesystems and start all services again using these commands:
 

mount -a
start nwdecoder
start nwappliance
NotesIn most cases, you will not need to retain the files found in /var/netwitness, but you may wish to copy the files off to an unused filesystem for further evaluation.  Often, the /var/netwitness/warehouseconnector will be unused and have a large amount of unused space that can be used for this purpose.

Though if it was actual Decoder packet or metadata files, then the files could be moved to proper mount points, making sure not to overwrite any existing file names.

Attachments

    Outcomes