000034623 - Puppet agent failure with "no space left on device" error in RSA Security Analytics 10.4 and above

Document created by RSA Customer Support Employee on Oct 19, 2017Last modified by RSA Customer Support Employee on Nov 17, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000034623
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Core Appliances, Security Analytics Head Unit
RSA Version/Condition: 10.4.x and above
Platform: CentOS

O/S Version: EL6
 
IssueSystem showing “No space left on device” when trying to write in to a linux partition or even run "puppet agent -t" command on partitions such as (/, /var, /usr…etc) even though "df -hP" output shows no issues with disk space.

# df -hP
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root          7.8G  5.1G  2.3G  69% /
tmpfs                                 48G     0   48G   0% /dev/shm
/dev/sda1                            243M   88M  143M  39% /boot
/dev/mapper/VolGroup00-usrhome       3.9G  8.1M  3.7G   1% /home
/dev/mapper/VolGroup01-rsahome        99G  588M   93G   1% /home/rsasoc
/dev/mapper/VolGroup01-tmp            20G  350M   19G   2% /tmp
/dev/mapper/VolGroup00-var           7.8G  364M  7.0G   5% /var
/dev/mapper/VolGroup01-sahome         50G  775M   50G   2% /var/lib/netwitness
/dev/mapper/VolGroup00-rabmq         245G   19G  226G   8% /var/lib/rabbitmq
/dev/mapper/VolGroup01-varlog        9.8G  3.1G  6.2G  34% /var/log
/dev/mapper/VolGroup00-nwhome         10G  7.8G  2.2G  79% /var/netwitness
/dev/mapper/VolGroup00-brokroot       50G  583M   50G   2% /var/netwitness/broker
/dev/mapper/VolGroup01-concroot       30G  475M   30G   2% /var/netwitness/concentrator
/dev/mapper/VolGroup02-concinde      300G  7.9G  293G   3% /var/netwitness/concentrator/index
/dev/mapper/VolGroup01-concmeta      2.2T  274G  2.0T  13% /var/netwitness/concentrator/metadb
/dev/mapper/VolGroup01-concsess      300G   17G  284G   6% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup00-redb          100G  7.6G   93G   8% /var/netwitness/database
/dev/mapper/VolGroup00-lcol          245G  544M  244G   1% /var/netwitness/logcollector
/dev/mapper/VolGroup00-ldecroot       30G  538M   30G   2% /var/netwitness/logdecoder
/dev/mapper/VolGroup00-ldecinde       10G   92M   10G   1% /var/netwitness/logdecoder/index
/dev/mapper/VolGroup00-ldecmeta      300G  282G   19G  94% /var/netwitness/logdecoder/metadb
/dev/mapper/VolGroup00-ldecpack      2.7T  418G  2.3T  16% /var/netwitness/logdecoder/packetdb
/dev/mapper/VolGroup00-ldecsess       30G   21G  9.3G  70% /var/netwitness/logdecoder/sessiondb
/dev/mapper/VolGroup01-ipdbext        30G   33M   30G   1% /var/netwitness/nwipdbextractor
/dev/mapper/VolGroup00-vartmp        3.9G  8.0M  3.7G   1% /var/tmp
/dev/mapper/concentrator0-sessiondb  1.7T   18G  1.7T   2% /var/netwitness/concentrator/sessiondb0
/dev/mapper/concentrator0-metadb      15T  301G   15T   2% /var/netwitness/concentrator/metadb0
/dev/mapper/logdecoder0-packetdb      17T  444G   16T   3% /var/netwitness/logdecoder/packetdb0
CauseThis can happen if the one of the partition inodes is 100% full. Running the command “df –iP” shows the following :

# df -iP
Filesystem                              Inodes  IUsed      IFree IUse% Mounted on
/dev/mapper/VolGroup00-root             524288 524288          0  100% /
tmpfs                                 12377304      1   12377303    1% /dev/shm
/dev/sda1                                65280     54      65226    1% /boot
/dev/mapper/VolGroup00-usrhome          262144     19     262125    1% /home
/dev/mapper/VolGroup01-rsahome         6553600    881    6552719    1% /home/rsasoc
/dev/mapper/VolGroup01-tmp             1310720  14716    1296004    2% /tmp
/dev/mapper/VolGroup00-var              524288  48003     476285   10% /var
/dev/mapper/VolGroup01-sahome         52428800   3638   52425162    1% /var/lib/netwitness
/dev/mapper/VolGroup00-rabmq         256016384   1551  256014833    1% /var/lib/rabbitmq
/dev/mapper/VolGroup01-varlog           655360    646     654714    1% /var/log
/dev/mapper/VolGroup00-nwhome          9220016   2584    9217432    1% /var/netwitness
/dev/mapper/VolGroup00-brokroot       52428800     43   52428757    1% /var/netwitness/broker
/dev/mapper/VolGroup01-concroot       31457280     28   31457252    1% /var/netwitness/concentrator
/dev/mapper/VolGroup02-concinde      314572800  51879  314520921    1% /var/netwitness/concentrator/index
/dev/mapper/VolGroup01-concmeta      472062336    279  472062057    1% /var/netwitness/concentrator/metadb
/dev/mapper/VolGroup01-concsess      314572800    137  314572663    1% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup00-redb          104857600   1274  104856326    1% /var/netwitness/database
/dev/mapper/VolGroup00-lcol          256016384    112  256016272    1% /var/netwitness/logcollector
/dev/mapper/VolGroup00-ldecroot       31457280     30   31457250    1% /var/netwitness/logdecoder
/dev/mapper/VolGroup00-ldecinde       10485760   1213   10484547    1% /var/netwitness/logdecoder/index
/dev/mapper/VolGroup00-ldecmeta       75896016    322   75895694    1% /var/netwitness/logdecoder/metadb
/dev/mapper/VolGroup00-ldecpack      561912192    261  561911931    1% /var/netwitness/logdecoder/packetdb
/dev/mapper/VolGroup00-ldecsess       31457280   1386   31455894    1% /var/netwitness/logdecoder/sessiondb
/dev/mapper/VolGroup01-ipdbext        31457280      3   31457277    1% /var/netwitness/nwipdbextractor
/dev/mapper/VolGroup00-vartmp           262144     13     262131    1% /var/tmp
/dev/mapper/concentrator0-sessiondb  351567872    183  351567689    1% /var/netwitness/concentrator/sessiondb0
/dev/mapper/concentrator0-metadb    3164091136    371 3164090765    1% /var/netwitness/concentrator/metadb0
/dev/mapper/logdecoder0-packetdb    3515659008    353 3515658655    1% /var/netwitness/logdecoder/packetdb0

As seen from above, the “/” inode is at 100% ,which could mean that there are a large number of small files and for this reason the error “No space left on device” is being produced as unable to create new files.
Resolution
  1. Display the Top 20 directories which have a high inode count (ascending order)

    find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n | tail -n20

    Note: A very common cause of high inode count are the contents of /var/spool/postfix/maildrop.
     
  2. Count the number of files in particular directory which are under 1 KB
     
    Syntax

    find <directory_from_command1_output> -mount -type f -size -1024c -print | wc -l


    Example

    find /var/spool/postfix/maildrop -mount -type f -size -1024c -print | wc -l

  3. Back up files smaller than 1 KB and older than a week then remove them.
     
    Example
    (assumes that the volume which has reached 100% is not hosting /var/netwitness/nwipdbextractor. If this is not available we could use /home. Use the df command to confirm the volume mounted on.)

    df -hP /var/netwitness/nwipdbextractor
    mkdir /var/netwitness/nwipdbextractor/manual_backup
    find /var/spool/postfix/maildrop -mount -type f -mtime +7 -size -1024c -print0 | tar -cvzf /var/netwitness/nwipdbextractor/manual_backup/`date +"%Y%m%d_%H%M"`_small_inode_backup.tar.gz --null -T -
    find /var/spool/postfix/maildrop -mount -type f -mtime +7 -size -1024c -delete

    If you are unsure which files are safe to remove, please contact RSA Technical Support before proceeding to Step 3.
  4. After deleting the small files, confirm the inode usage once again using the command below.

    df -iP

If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and quote this article number for further assistance.

Attachments

    Outcomes