000037892 - RSA NetWitness Platform - Error 'disk resource limit alarm has tripped' in sa@localhost.log due to file collection logs piled up in Collector

Document created by RSA Customer Support Employee on Sep 12, 2019
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000037892
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Collector
RSA Version/Condition: 10.6.X, 11.X
Platform: CentOS
O/S Version: 7
IssueRabbitmq server is not running due to 'disk resource limit alarm' as below.


=WARNING REPORT==== 29-Aug-2019::09:26:47 ===
disk resource limit alarm has tripped on node sa@localhost.  Collection will be blocked until this alarm clears!


Aug 24 04:05:12 Collector1 NwLogCollector[23993]: [MessageBrokerLogReceiver] [info] info 2019-08-24T04.05.12Z Disk free space insufficient. Free bytes:104907436032 Limit:104908750000
CauseThis issue is due to /var/netwitness/logcollector has reached 80% of available storage as below.

[root@Collector1 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on
                       20G  4.5G   14G  25% /
tmpfs                  48G     0   48G   0% /dev/shm
/dev/sdd1             248M  150M   85M  64% /boot
                      3.9G  417M  3.3G  12% /home
                       20G  171M   19G   1% /tmp
                      9.8G  3.6G  5.8G  39% /var/log
                       10G  935M  9.1G  10% /var/netwitness
                       30G  940M   30G   4% /var/netwitness/concentrator
                      300G   38G  263G  13% /var/netwitness/concentrator/index
                      2.4T  2.3T  130G  95% /var/netwitness/concentrator/metadb
                      300G  285G   16G  95% /var/netwitness/concentrator/sessiondb
                      489G  362G  127G  80% /var/netwitness/logcollector

                       30G  923M   30G   4% /var/netwitness/logdecoder
                       10G   37M   10G   1% /var/netwitness/logdecoder/index
                      300G  284G   17G  95% /var/netwitness/logdecoder/metadb
                      2.8T  2.7T  149G  95% /var/netwitness/logdecoder/packetdb
                       30G   29G  2.0G  94% /var/netwitness/logdecoder/sessiondb
                      400G   35G  366G   9% /var/netwitness/warehouseconnector
                      5.8G   12M  5.5G   1% /var/tmp
[root@hydsiemhyb01 ~]#
ResolutionPlease use the below steps to identify the cause for high consumption in /var/netwitness/logcollector.
  1. Run du -xh /var/netwitness/logcollector --max-depth=2|sort -h>collectingspace.txt command for consumption details.
  2. tail collectingspace.txt command shows as below.

    20M     /var/netwitness/logcollector/rabbitmq/log
    27M     /var/netwitness/logcollector/rabbitmq
    791M    /var/netwitness/logcollector/statdb
    3.3G    /var/netwitness/logcollector/metadb
    17G     /var/netwitness/logcollector/upload/microsoft_dhcp_2008
    373G    /var/netwitness/logcollector/upload/iis_tvm
    389G    /var/netwitness/logcollector/upload
    389G    /var/netwitness/logcollector/upload_chroot
    389G    /var/netwitness/logcollector/upload_chroot/home
    782G    /var/netwitness/logcollector/

  3. Above output indicates that the space consumption was due to /var/netwitness/logcollector/upload/iis_tvm which has huge files under /var/netwitness/logcollector/upload/iis_tvm/<eventSource>/save directory.
  4. These files were saved after a successful process of logs by Log Collector by choosing "Save on Success" as below for File Collection event source.
  5. Remove the files under /var/netwitness/logcollector/upload/iis_tvm/<eventSource>/save directory to free up space.
  6. service rabbitmq-server start command in collector.
  7. Uncheck the "Save On Success" option for all file collection event sources in Collector->Config->Event Sources->File/Config page.
More details on Save On Success