000034199 - RSA Netwitness Logcollection stopped due to corrupted XFS file system

Document created by RSA Customer Support Employee on Oct 24, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000034199
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Hybrid Appliance/LogDecoder
RSA Version/Condition: 10.5.x.x and higher releases
Platform: CentOS
 
Issue
  • LogCollection Stopped. 
  • Running command: df -kh shows that the LogCollector director is fully utilized [100% full].
 
  • User-added image
Cause1- Corrupted "xfs" filesystem within the /var/netwitness/logcollector directory .
Resolution1- vi the /etc/fstab file to hash the NFS mapping  of the LogCollector directory.
User-added image
Put # prior the line highlighted with Red to be like: 
#/dev/mapper/VolGroup00-lcol /var/netwitness/logcollector xfs     nosuid,noatime  1 2

2- Reboot the appliance using command: reboot.
3-  Check which process is accessing that directory using command: lsof
4- stop nwlogcollector service. 
5- Unhash the LogCollector NFS mapping. 
6- Unmount the directory using command: umount /dev/mapper/VolGroup00-lcol
7- Perform  filesystem check using command: xfs_check /dev/mapper/VolGroup00-lcol
8- Perfom filesystem repair using command: xfs_repair -L /dev/mapper/VolGroup00-lcol
9- Re-mount the LogCollector durectory using command: umount -a /dev/mapper/VolGroup00-lcol
10- Run command "Start nwlogcollector service" to start logcollection service. 
11- Issue command:  df -kh to check the storage status of the "/var/netwitness/logcollector" directory you will find it dropped down around 30% to reach 70% of utilized storage only. 
12- Run command tail -f /var/log/messages to track the operations of appliance. 
13- Perform required health-checks by checking the health &Wellness on the SA UI.

Attachments

    Outcomes