000032301 - The messages file fills up the /var/log partition and prevents services from starting on an RSA Security Analytics Hybrid or AIO appliance

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000032301
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: All-in-One, Hybrid
Platform: CentOS
IssueThe /var/log/messages file fills up the /var/log partition and prevent the nwlogcollector (or other services) from starting.
CauseThis issue may occur especially on busy systems such as an AIO or Hybrid appliance when /var/log/messages is constantly populated with logs sent by all of the SA services.
In order to detect the problem login to the affected appliance using ssh and run the following commands, also consider the outputs in the examples below.
df -hT

Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/sda1            ext3   9.9G  827M  8.6G   9% /
tmpfs                tmpfs  7.8G   46M  7.8G   1% /dev/shm
                     ext4   3.9G  1.4G  2.3G  39% /usr
                     ext4   2.0G  3.1M  1.9G   1% /home
                     ext4   3.9G  278M  3.4G   8% /var
                     ext4   3.9G  3.6G   87M  98% /var/log
                     ext4   5.8G   60M  5.5G   2% /tmp
                     ext4   2.0G  3.0M  1.9G   1% /var/tmp
                     ext4   3.9G  129M  3.5G   4% /opt
                     xfs     10G   41M   10G   1% /var/lib/rabbitmq
                     xfs     12G  383M   12G   4% /var/netwitness
                     xfs    104G  2.0G  103G   2% /var/netwitness/logcollector

As you can see from the result of df -hT /var/log partition reached the 98% so it is going to fill up very soon.
A closer inspection on /var/log partition is necessary to see which directory is actually generating the issue by looking at the size:
for i in $(find /var/log/ -type d -xdev -maxdepth 2) ;do du -s $i ; done | sort -nr
3496068 /var/log/
684772 /var/log/netwitness
684440 /var/log/netwitness/logcollector
42064 /var/log/rabbitmq
29604 /var/log/audit
25484 /var/log/sa
7256 /var/log/cluster
328 /var/log/netwitness/appliance
16 /var/log/lost+found
16 /var/log/install
8 /var/log/samba
4 /var/log/samba/old
4 /var/log/puppet
4 /var/log/ntpstats

Now we can use ls to look at the files in /var/log directory:
ls -liahS /var/log
35 -rw-------. 1 root root 2.3G Dec 16 16:52 messages
56 -rw-r--r--. 1 root root 232M Dec 16 16:57 sync_vlc_ha.log
22 -rw-------. 1 root root 50M Dec 16 16:57 secure
12 -rw-------. 1 root root 22M Dec 16 16:57 cron
97 -rw-------. 1 root root 16M Dec 15 20:01 messages-20151215.gz
93 -rw-------. 1 root root 9.7M Dec 16 00:01 messages-20151216.gz

We identified the issue now, /var/log/messages is causing /var/log to fill up quickly.
ResolutionWe need to find a way to adjust the logrotate default behaviour by changing the configuration file /etc/logrotate.d/syslog in order to make /var/log/messages rotate in a proper way and fix the issue.
WorkaroundLooking at the /etc/logrotate.d/syslog we can see the current setting for the /var/log/messsages directive:
cat /etc/logrotate.d/syslog
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

We need to change the entry related with /var/log/messages by adding the following parameters:
/var/log/messages { 
rotate 5
dateformat -%Y%m%d-%s
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

In this way we are going to rotate 5 times /var/log/messages on a daily basis when the file reaches the size of 300 MB.
We also add another date format as filename for the rotated logs.
After several hours the result of the ls command now shows us that the file is rotate properly and doesn't fill up the /var/log/ partition anymore.
ls -liath /var/log
38 -rw-------.  1 root      root     232M Dec 18 05:00 messages
35 -rw-------.  1 root      root     309K Dec 18 05:00 cron
98 -rw-------.  1 root      root     3.3M Dec 18 04:59 secure
56 -rw-r--r--.  1 root      root     5.4M Dec 18 04:59 sync_vlc_ha.log
91 -rw-r--r--.  1 root      root      67K Dec 18 04:59 vlc_ha.log
2 drwxr-xr-x. 12 root      root     4.0K Dec 18 02:01 .
66 -rw-------.  1 root      root     9.8M Dec 18 02:01 messages-20151218-1450404061.gz
8193 drwxr-xr-x.  2 root      root     4.0K Dec 18 00:00 sa
93 -rw-------.  1 root      root     9.7M Dec 17 21:01 messages-20151217-1450386061.gz
106 -rw-------.  1 root      root     9.8M Dec 17 17:01 messages-20151217-1450371661.gz
31 -rw-r--r--.  1 root      root     144K Dec 17 16:52 lastlog
61 -rw-rw-r--.  1 root      utmp     1.5K Dec 17 16:52 wtmp
111 -rw-------.  1 root      root     278K Dec 17 14:28 messages-20151217-1450362488.gz
108 -rw-------.  1 root      root     242K Dec 17 14:24 messages-20151217-1450362263.gz

We could also double check the partition with to confirm that the issue is fixed:
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-var ext4 3.9G 278M 3.4G 8% /var
/dev/mapper/VolGroup00-log ext4 3.9G 1.2G 2.5G 32% /var/log

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.
NotesIf after applying the above steps, the files are not rotating. Restart of syslog service is required.
service rsyslog restart