000030769 - Web Threat Detection 5.1.0.7 Indexer core dumps with a segfault after upgrade

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 22, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000030769
Applies ToRSA Product Set: Web Threat Detection
RSA Product/Service Type: Web Threat Detection
RSA Version/Condition: 5.1.0.7
Platform: WTD 5.1.0.7
Platform (Other): na
O/S Version: Centos/RHEL 6.x
Product Name: Web Threat Detection
Product Description: Web Threat Detection
IssueEnvironments upgraded to 5.1.0.7 may experience a segfault when the Indexer process is attempting to run. When this happens Indexer will, if the OS if configured, will leave a core file in /var/log/silvertail.
Symptoms:

Syslog will have messages similar to the ones below:
Jun 2 07:00:25 <hostname> kernel: [6603643.182719] Callback[5016]: segfault at 0 ip (null) sp 00007fc46592cfe8 error 14
Jun 2 07:00:25 <hostname> kernel: [6603643.182723] Callback[5021]: segfault at 0 ip (null) sp 00007fc462727fe8 error 14 in indexer[400000+4aa000] in indexer[400000+4aa000]
There will be an hour that will not show any clicks in the Forensics UI.
If the OS is configured correctly there will be Indexer core files in /var/log/silvertail.

 
CauseThis is caused by enhancements made to WTD in 5.1.0.7.
ResolutionTo fix this issue Customers will need to upgrade to a fixed in version of WTD.
5.1.0.8 and higher versions will have the fix for this issue.
The latest version of 5.1 software will be available through SecurCare Online on Download Central.
Instructions to download new software for Web Threat Detection can be found here:
https://rsaportal.force.com/customer/articles/How_To/a67110-Silver-Tail-Web-Threat-Detection-Download-instructions
WorkaroundLogin via SSH to a system in the environment with access to the /var/opt/silvertail/data directory.
Move the .task file, for the hour missing traffic in the FUI, from /var/opt/silvertail/data/tasks/indexer/failed to the start point of the task chain, this is normally /var/opt/silvertail/data/tasks/organizer/completed. This can be looked up in SilverCat under Indexer>tasks>pending.
This will result in the tasks running again for that hour, only one task will be processed at a time, if multiple .task files are in the /failed directory they will be ran in order. 
If a task fails again, and the task file is deposited back in /var/opt/silvertail/data/tasks/indexer/failed, it may not be because of the Indexer Segfault issue. Open a case with support to verify that there is not another reason why Indexer is failing.
 
NotesThis issue is only found in version 5.1.0.7

Attachments

    Outcomes