System showing “No space left on device” when trying to write into a Linux partition or even run "puppet agent -t" command on partitions such as (/, /var, /usr…etc) even though "df -hP" output shows no issues with disk space.
As seen from above, the “/” inode is at 100 %, which could mean that there are a large number of small files and for this reason the error “No space left on device” is being produced as unable to create new files.
If you are on RSA Security Analytics version 10.6.5.2 and using a log collector there is a known core puppet permissions issue on /etc/netwitness/ng/logcollector/lctwin. This will cause a misalignment on /home/upload/.ssh/authorized_keys and /upload/.ssh/authorized_keys.
Display the Top 20 directories which have a high inode count (ascending order)
Back up files smaller than 1 KB and older than a week then remove them.
Example (assumes that the volume which has reached 100% is not hosting /var/netwitness/nwipdbextractor. If this is not available we could use /home. Use the df command to confirm the volume mounted on.)
If you are unsure which files are safe to remove, please contact RSA Technical Support before proceeding to Step 3.
After deleting the small files, confirm the inode usage once again using the command below.
Note: If you are on RSA Security Analytics 10.6.5.2 we recommend upgrading to RSA Security Analytics 10.6.6 to resolve this issue. For a list of resolved issues and how to upgrade to RSA Security Analytics 10.6.6 please review the 10.6.6 Release Notes and Deployment guides below: https://community.rsa.com/community/products/netwitness/1066
If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and quote this article number for further assistance.