000017608 - The /tmp partition on an RSA  NetWitness Platform appliance is 100% utilized but no large files are present

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support on Sep 25, 2019
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000017608
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Core
RSA Version/Condition: 10.6.x, 11.x
Platform: CentOS
IssueThe /tmp partition on an RSA Security Analytics appliance is 100% utilized but no large files are present.

Issuing the df -h command on the appliance shows the /tmp partition as being full, as seen in the example below.




Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
                       20G  2.1G   17G  12% /
/dev/mapper/VolGroup00-tmp
                       20G  20G      0 100% /tmp
/dev/mapper/VolGroup00-usrhome
                      4.9G  139M  4.5G   3% /home
/dev/mapper/VolGroup00-var
                       20G  4.3G   15G  23% /var
/dev/mapper/VolGroup00-nwhome
                       62G  890M   58G   2% /var/netwitness
/dev/sdc1             251M   18M  221M   8% /boot
tmpfs                  48G     0   48G   0% /dev/shm
/dev/mapper/decodersmall-decoroot
                      9.9G  4.3G  5.1G  46% /var/netwitness/decoder
/dev/mapper/decodersmall-index
                       10G  1.1G  9.0G  11% /var/netwitness/decoder/index
/dev/mapper/decodersmall-sessiondb
                      600G  133G  468G  23% /var/netwitness/decoder/sessiondb
/dev/mapper/decoder-packetdb
                       19T   18T  945G  95% /var/netwitness/decoder/packetdb




Examining the /tmp partition with the ls -lah /tmp command shows no large files that account for the utilized disk space.
CauseWhile the /tmp partition is used to store physical files, it is also used to store "virtual files," meaning files that are in use by an active process.  If a process doesn't release the file correctly, even though the file is not is not actually on the filesystem structure, the space it used is still allocated.
Resolution

In order to confirm that the issue is caused by allocated space from files not properly released from processes, you may issue the following command:  lsof | grep /tmp | grep deleted



Issuing the command above will display a list of the files that have since been deleted but are still associated with an active process and claiming disk space.  You will also be able to see the amount of space that is being consumed.(See the screenshot in the section below)



After identifying the process (or processes) that is still linked to the files, you will be able to perform one of the following three actions to free the space:



  • Restart the service/daemon that is responsible.
  • Kill the associated process.
  • Reboot the appliance.

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article ID for further assistance.

Notes

The screenshot below is an example of output generated by the lsof | grep /tmp | grep deleted command that was issued on a Security Analytics server appliance.



Legacy Article IDa66756

Attachments

    Outcomes