on Jun 30, 2016Article Content
| Article Number | 000032720 |
| Applies To | RSA Product Set: Security Analytics RSA Product/Service Type: SA Core Appliance RSA Version/Condition: 10.5.X, 10.4.x, 10.3.x, 10.2.x Platform: CentOS O/S Version: 6 |
| Issue | Health and Wellness indicates the /var/netwitness partition is at 100% utilization but you cannot identify the files that are taking up space on the partition. It is possible that a core service may start before a mount point has mounted. This can create directories and files on a filesystem other than what is intended, and it can result in the filesystem filling and reporting as full in Health & Wellness and from the Linux command line. You may see something like the following: [root@PacketDecoder01 netwitness]# df -hThe /var/netwitness filesystem appears to be full, but the Decoder service continues to run and the /var/netwitness/decoder/packetdb filesystem is less than 60% full. When you examine the contents of /var/netwitness you see something line the following: [root@PacketDecoder01 ~]# ls -lah /var/netwitness In other words, you observe no files in the /var/netwitness partition that could be consuming nearly 30GB of space. |
| Cause | In certain circumstances, a service may start capturing or aggrgating data before the specified filesystem is mounted. When this happens, files may be stored in the local /var/netwitness folder instead of the filesystem that would normally be mounted in this mount point. Then, at a later time, when you mount a filesystem on a directory /mount-point, you can no longer access files under /mount-point directly. They still exist, but /mount-point now refers to the root of the mounted filesystem, not to the directory that served as a mount point, so the contents of this directory cannot be accessed, at least in this way. This effectively "hides" or "loses" the files that were previously and erroneously created directly under the filesystem, and makes the filesystem full even though you can't see the files. |
| Resolution | Stop the appliance services such as nwapliance and nwdecoder then unmount all decoder filesystems using these commands: stop nwdecoder Change directories to the /var/netwitness folder and look for any files that remain. These files are almost certainly unwanted files that were accidentally created at some point in time and should be the files consuming the space on the filesystem. Once you have removed the unwanted files, mount all filesystems and start all services again using these commands: mount -a |
| Notes | In most cases you will not need to retain the files found in /var/netwitness, but you may wish to copy the files off to an unused filesystem for further evaluation. Often, the /var/netwitness/warehouseconnector will be unused and have a large amount of unused space. |