|Applies To||RSA Product Set: Security Analytics|
RSA Product/Service Type: All-in-One, Hybrid
|Issue||The /var/log/messages file fills up the /var/log partition and prevent the nwlogcollector (or other services) from starting.|
|Cause||This issue may occur especially on busy systems such as an AIO or Hybrid appliance when /var/log/messages is constantly populated with logs sent by all of the SA services.|
In order to detect the problem login to the affected appliance using ssh and run the following commands, also consider the outputs in the examples below.
As you can see from the result of df -hT /var/log partition reached the 98% so it is going to fill up very soon.
A closer inspection on /var/log partition is necessary to see which directory is actually generating the issue by looking at the size:
for i in $(find /var/log/ -type d -xdev -maxdepth 2) ;do du -s $i ; done | sort -nr
Now we can use ls to look at the files in /var/log directory:
ls -liahS /var/log
We identified the issue now, /var/log/messages is causing /var/log to fill up quickly.
|Resolution||We need to find a way to adjust the logrotate default behaviour by changing the configuration file /etc/logrotate.d/syslog in order to make /var/log/messages rotate in a proper way and fix the issue.|
|Workaround||Looking at the /etc/logrotate.d/syslog we can see the current setting for the /var/log/messsages directive:|
We need to change the entry related with /var/log/messages by adding the following parameters:
In this way we are going to rotate 5 times /var/log/messages on a daily basis when the file reaches the size of 300 MB.
We also add another date format as filename for the rotated logs.
After several hours the result of the ls command now shows us that the file is rotate properly and doesn't fill up the /var/log/ partition anymore.
ls -liath /var/log
We could also double check the partition with to confirm that the issue is fixed:
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.
|Notes||If after applying the above steps, the files are not rotating. Restart of syslog service is required.|
service rsyslog restart