000037185 - RSA NetWitness 11.x /var/log mount is full due to logstash directory

Document created by RSA Customer Support Employee on Feb 15, 2019Last modified by RSA Customer Support Employee on Mar 22, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000037185
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7
IssueAfter upgrading from NetWitness 10.x to NetWitness 11.x the /var/log mount is filling up, eventually becoming full.

User-added image

# du /var/log -h --max-depth=2 |sort -h |tail
11M     /var/log/netwitness/investigate-server
11M     /var/log/netwitness/orchestration-server
12M     /var/log/nginx
36M     /var/log/audit
56M     /var/log/sa
74M     /var/log/netwitness
222M    /var/log/rabbitmq
613M    /var/log/mongodb
6.4G    /var/log/logstash
10G     /var/log

In the above example the /var/log/logstash directory is using 6.4GB of the available 10GB /var/log mountpoint.

# echo;perl -e "print '/var/log/logstash is using '; print substr( `du -k /var/log/logstash/ |cut -f1` / `df -k /var/log |tail -1 |awk '{print $2}'` * 100, 0, 4);print '% of /var/log'";echo

/var/log/logstash is using 64.06% of /var/log

In the above example the /var/log/logstash directory is using 64.06% of the available 10GB /var/log mountpoint.
CauseIn NetWitness 10.x there was a /etc/logrotate.d/logstash configuration file to manage the logstash log files.
This configuration file doesn't exist in NetWitness 11.x.
  1. ssh login to the problem appliance.
  2. If the /var/log mountpoint is >90% full, then delete a few of the oldest logstash-plain-*.log files under the /var/log/logstash directory, so that there is sufficient free disk space for the logrotate command to be able to run.
  3. Create a /etc/logrotate.d/logstash file to have all the following lines:

    /var/log/logstash/logstash-plain.log {
            rotate 7
            dateformat -%Y-%m-%d
            extension .log

  4. To run the logstash logrotate now, and test the newly created configuration file is correct, run the command,

logrotate -vf /etc/logrotate.d/logstash

-v, Verbose output
-f, Force logrotation to occur.
NotesWhat if the logrotate command doesn't tidy up the files in the /var/log/logstash directory?

If the logrotate command encounters an error condition, then it will stop without doing anything.

Check if any of the below is the cause for no action.
  1. The file /var/log/logstash/logstash-plain.log doesn't exist or is empty, then logrotate stops. ls -l /var/log/logstash/logstash-plain.log
    -rw-r--r--.  1 logstash logstash       0 Mar  6 03:24 logstash-plain.log

    Restart the logstash service to create the file, or enter some text into the file.

    systemctl restart logstash

  2. A .gz file already exists for a particular .log file, then logrotate can't compress the .log file to overwrite the existing .gz file and it stops.

    -rw-r--r--. 1 logstash logstash 6.3M Mar 4 23:19 logstash-plain-2019-03-03.log
    -rw-r--r--. 1 logstash logstash 20 Mar 4 23:19 logstash-plain-2019-03-03.log.gz

    Move or rename the existing .gz file.
  3. Run the logrotate command with the debug (-d) switch. No action occurs with the debug switch selected, but output shows what the logrotate command would do, and possibly show the error.

    -d, --debug Turns  on debug mode and implies -v.  In debug mode, no changes will be made to the logs or to the logrotate state file.

    logrotate -df /etc/logrotate.d/logstash

  4. Can just manually delete the oldest /var/log/logstash logs files, keeping the newest 7 files.