000032720 - RSA Security Analytics Health and Wellness indicates /var/netwitness partition is at 100% utilization

Document created by RSA Customer Support Employee on Jun 30, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000032720
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.5.X, 10.4.x, 10.3.x, 10.2.x
Platform: CentOS
O/S Version: 6
IssueHealth and Wellness indicates the /var/netwitness partition is at 100% utilization but you cannot identify the files that are taking up space on the partition.  
It is possible that a core service may start before a mount point has mounted.  
This can create directories and files on a filesystem other than what is intended, and it can result in the filesystem filling and reporting as full in Health & Wellness and from the Linux command line.
You may see something like the following:
[root@PacketDecoder01 netwitness]# df -h
Filesystem            Size  Used Avail Use% Mounted on
                      7.8G  4.7G  2.8G  64% /
tmpfs                  48G     0   48G   0% /dev/shm
/dev/sda1             496M   62M  409M  14% /boot
                      3.9G  8.1M  3.7G   1% /home
                      9.8G   53M  9.2G   1% /opt
                       20G   44M   19G   1% /tmp
                      7.8G   97M  7.3G   2% /var
                       20G   37M   20G   1% /var/lib/rabbitmq
                      9.8G  369M  8.9G   4% /var/log
                       30G   30G   24K 100% /var/netwitness

                      400G  603M  400G   1% /var/netwitness/warehouseconnector
                      3.9G  8.1M  3.7G   1% /var/tmp
                       10G  2.6G  7.5G  26% /var/netwitness/decoder
                       30G   69M   30G   1% /var/netwitness/decoder/index
                      5.2T  469G  4.7T   9% /var/netwitness/decoder/metadb
                      278G   11G  267G   4% /var/netwitness/decoder/sessiondb
                       28T   16T   12T  58% /var/netwitness/decoder/packetdb

The /var/netwitness filesystem appears to be full, but the Decoder service continues to run and the /var/netwitness/decoder/packetdb filesystem is less than 60% full.
When you examine the contents of /var/netwitness you see something line the following:
[root@PacketDecoder01 ~]#  ls -lah /var/netwitness
total 8.0K
drwxr-xr-x.    5  root root   85 Nov  9 08:36 .
drwxr-xr-x.   21 root root  4.0K Dec 21 13:06 ..
drwxr-xr-x.    3 root root    19 Nov  9 08:36 appliance
drwxr-xr-x.    9 root root    98 Jan 27 12:44 decoder
-rw-------.    1 root root    12 Mar  7 19:55 NwDecoder.persist
drwxr-xr-x.    2 root root     6 Nov  6 09:41 warehouseconnector

In other words, you observe no files in the /var/netwitness partition that could be consuming nearly 30GB of space.
CauseIn certain circumstances, a service may start capturing or aggrgating data before the specified filesystem is mounted.
When this happens, files may be stored in the local /var/netwitness folder instead of the filesystem that would normally be mounted in this mount point.
Then, at a later time, when you mount a filesystem on a directory /mount-point, you can no longer access files under /mount-point directly. They still exist, but /mount-point now refers to the root of the mounted filesystem, not to the directory that served as a mount point, so the contents of this directory cannot be accessed, at least in this way.
This effectively "hides" or "loses" the files that were previously and erroneously created directly under the filesystem, and makes the filesystem full even though you can't see the files. 
ResolutionStop the appliance services such as nwapliance and nwdecoder then unmount all decoder filesystems using these commands: 
stop nwdecoder
stop nwappliance 
umount /var/netwitness/decoder/sessiondb 
umount /var/netwitness/decoder/index 
umount /var/netwitness/decoder/metadb 
umount /var/netwitness/decoder/packetdb 
umount /var/netwitness/warehouseconnector
umount /var/netwitness/decoder 
umount /var/netwitness

Change directories to the /var/netwitness folder and look for any files that remain.  These files are almost certainly unwanted files that were accidentally created at some point in time and should be the files consuming the space on the filesystem.
Once you have removed the unwanted files, mount all filesystems and start all services again using these commands:
mount -a
start nwdecoder
start nwappliance
NotesIn most cases you will not need to retain the files found in /var/netwitness, but you may wish to copy the files off to an unused filesystem for further evaluation.  Often, the /var/netwitness/warehouseconnector will be unused and have a large amount of unused space.