000037185 - RSA NetWitness Platform 11.x /var/log mount is full due to logstash directory

Document created by RSA Customer Support Employee on Feb 15, 2019Last modified by RSA Customer Support Employee on Oct 1, 2019
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000037185
Applies ToRSA Product Set: RSA NetWitness Platform, NetWitness Logs & Network
RSA Product/Service Type: NetWitness Server/Admin Server
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7
IssueAfter upgrading from NetWitness 10.x to NetWitness 11.x the /var/log mount is filling up, eventually becoming full.

User-added image

# du -hx --max-depth=2 /var/log | sort -h | tail
11M     /var/log/netwitness/investigate-server
11M     /var/log/netwitness/orchestration-server
12M     /var/log/nginx
36M     /var/log/audit
56M     /var/log/sa
74M     /var/log/netwitness
222M    /var/log/rabbitmq
613M    /var/log/mongodb
6.4G    /var/log/logstash
10G     /var/log

In the above example, the /var/log/logstash directory is using 6.4 GB of the 10 GB that has been utilized of the /var/log mount point.

# awk "BEGIN { printf \"/var/log/logstash is using %3.2f%% of the /var/log mount\n\", $(du -ksx /var/log/logstash | awk '{print $1}')/$(df -k /var/log | tail -n1 | awk '{print $2}') * 100, 4 }"

Sample Output:

/var/log/logstash is using 64.06% of the /var/log mount

In the above example the /var/log/logstash directory is using 64.06% of the 10 GB /var/log mount. Size of the /var/log mount is derived from a command equivalent to:

# df -hP /var/log
CauseIn NetWitness 10.x there was a /etc/logrotate.d/logstash configuration file to manage the logstash log files.
This configuration file may not exist by default in some versions of NetWitness 11.x.
  1. Obtain a console session on the problem host (using SSH, iDRAC console or local console)
  2. If the /var/log mountpoint is >90% utilized as seen in `df -h /var/log`, then move/remove a few of the oldest logstash-plain-*.log files under the /var/log/logstash directory, so that there is sufficient free disk space to compress the remaining files using gzip and for the logrotate command to be able to run.
  3. Create a /etc/logrotate.d/logstash file to have all the following lines:

    /var/log/logstash/*.log {
            maxsize 100M
            rotate 7
            dateformat -%Y-%m-%d
            extension .log
            su logstash logstash

  4. Make sure the permissions are correct on the logstash logrotate config file and also that the files in the logstash directory belong to the logstash group

# chmod 644 /etc/logrotate.d/logstash
# chown logstash:logstash /var/log/logstash/*

  1. Clean up the current directory and restart the logstash service

# find /var/log/logstash -name "logstash-plain-*.log" -type f -print0 | xargs -0 -I % --no-run-if-empty gzip -9 %
# sudo -u logstash systemctl restart logstash.service

  1. To test the newly created configuration file is correct, run the following command to force an immediate logrotate run:

# logrotate --force --verbose /etc/logrotate.d/logstash

NotesWhat if the logrotate command doesn't tidy up the files in the /var/log/logstash directory?

If the logrotate command encounters an error condition, then it will stop without doing anything.

Check if any of the below is the cause for no action.
  1. The file /var/log/logstash/logstash-plain.log doesn't exist or is empty, then logrotate stops. ls -l /var/log/logstash/logstash-plain.log
    -rw-r--r--.  1 logstash logstash       0 Mar  6 03:24 logstash-plain.log

    Restart the logstash service to create the file, or enter some text into the file.

    systemctl restart logstash

  2. A .gz file already exists for a particular .log file, then logrotate can't compress the .log file to overwrite the existing .gz file and it stops.

    -rw-r--r--. 1 logstash logstash 6.3M Mar 4 23:19 logstash-plain-2019-03-03.log
    -rw-r--r--. 1 logstash logstash 20 Mar 4 23:19 logstash-plain-2019-03-03.log.gz

    Move or rename the existing .gz file.
  3. Run the logrotate command with the debug (-d) switch. No action occurs with the debug switch selected, but the output shows what the logrotate command would do, and possibly show the error.

    -d, --debug Turns on debug mode and implies -v.  In debug mode, no changes will be made to the logs or to the logrotate state file.

    logrotate -df /etc/logrotate.d/logstash

  4. Can just manually delete the oldest /var/log/logstash logs files, keeping the newest 7 files.