000034199 - RSA NetWitness Logcollection stopped due to corrupted XFS file system

Document created by RSA Customer Support Employee on Oct 24, 2016Last modified by RSA Customer Support on May 1, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000034199
Applies ToRSA Product Set: Security Analytics (10.5.x.x, 10.6.x.x), Netwitness for Logs and Network (11.x)
RSA Product/Service Type: Log Collector (Can be applied to any host that exhibits the same symptoms but was initially discovered on a Log Decoder/Log Collector)
RSA Version/Condition: 10.5.x.x and higher releases
Platform: CentOS 6, 7
Issue
  • LogCollection Stopped. 
  • Running command: df -kh shows that the LogCollector director is fully utilized [100% full].
 

  • User-added image
CauseCorrupted "xfs" filesystem within the /var/netwitness/logcollector directory .
Resolution
  1. vi the /etc/fstab file to hash the NFS mapping of the LogCollector directory.
    User-added image

    Put # prior the line highlighted with Red to be like: 



    #/dev/mapper/VolGroup00-lcol /var/netwitness/logcollector xfs     nosuid,noatime  1 2

  2. Reboot the appliance using command: reboot.
  3. Check which process is accessing that directory using command: lsof
  4. stop nwlogcollector service. 
  5. Unhash the LogCollector NFS mapping. 
  6. Unmount the directory using command: umount /dev/mapper/VolGroup00-lcol
  7. Perform  filesystem check using command: xfs_check /dev/mapper/VolGroup00-lcol
  8. Perfom filesystem repair using command: xfs_repair -L /dev/mapper/VolGroup00-lcol
  9. Re-mount the LogCollector durectory using command: umount -a /dev/mapper/VolGroup00-lcol
  10. Run command "Start nwlogcollector service" to start logcollection service. 
  11. Issue command:  df -kh to check the storage status of the "/var/netwitness/logcollector" directory you will find it dropped down around 30% to reach 70% of utilized storage only. 
  12. Run command tail -f /var/log/messages to track the operations of appliance. 
  13. Perform required health-checks by checking the health &Wellness on the SA UI.

Attachments

    Outcomes