000031139 - Reporting Engine stops logging events in reporting-engine.log after upgrading Security Analytics server

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000031139
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Reporting Engine
RSA Version/Condition: 10.4.x, 10.5.x
IssueAfter upgrading the Security Analytics server, no events are logged in the /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log file.
The Reporting Engine appears to run normally and reports, alerts and charts are displayed as expected.
In the Security Analytics UI, navigating to Administration > Services User-added image > View > Logs for the Reporting Engine does not show current Realtime events.
From an SSH session, running  the following command below confirms the file is empty.
tail /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log

The /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.sh_YYYYMMDD.log file shows the following error:
15/09/04 00:01:12 WARN i18n.MessageFactory: Can't find bundle for base name com.rsa.netwitness.carlos.management.proto.ManagementMessages, locale en_US; fabricating bundle dynamically.
log4j:ERROR An error occurred maintaining the log file index.
org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.1_index lockFactory=org.apache.lucene.store.NativeFSLockFactory@35dab4eb: files: [write.lock]
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:667)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
    at org.apache.lucene.index.IndexReader.open(IndexReader.java:273)
    at com.rsa.netwitness.carlos.logging.IndexingRollingFileAppender.initializeState(IndexingRollingFileAppender.java:225)
    at com.rsa.netwitness.carlos.logging.IndexingRollingFileAppender.activateOptions(IndexingRollingFileAppender.java:200)
    at com.rsa.soc.re.server.startup.ApplicationStartup.initLogging(ApplicationStartup.java:287)
    at com.rsa.soc.re.server.startup.ApplicationStartup.initApplication(ApplicationStartup.java:207)
    at com.rsa.soc.re.server.startup.ApplicationStartup.init(ApplicationStartup.java:97)
    at com.rsa.soc.re.server.startup.DeployJar.startApp(DeployJar.java:89)
    at com.rsa.soc.re.server.startup.DeployJar.init(DeployJar.java:79)
    at com.rsa.soc.re.server.startup.DeployJar.main(DeployJar.java:61)
CauseThe issue is caused by a corrupted Lucene log index file (reporting-engine.log.1_index from the above error).
ResolutionIn order to resolve the issue, identify the corrupted file by reviewing the most recent reporting-engine.sh_YYYYMMDD.log.
For example, the corrupted file is reporting-engine.log.1_index in the following log.
org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.1_index lockFactory=org.apache.lucene.store.NativeFSLockFactory@35dab4eb: files: [write.lock]

Then, delete the corrupted index folder and the log file as well as the folders and files created after the corruption by following the steps below.
  1. stop rsasoc_re
  2. mkdir /root/reporting-engine_bak
  3. cp /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log* /root/reporting-engine_bak
  4. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log
  5. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log_index/
  6. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.1
  7. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.1_index/
  8. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.x
  9. rm -rf /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log.x_index/ until the corrupted file and folder are deleted.
  10. start rsasoc_re
  11. tail -f /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log and confirm new events are logged.
 

Attachments

    Outcomes