000034437 - How to analyze /Data Log files in RSA Web Threat Detection

Document created by RSA Customer Support Employee on Jan 11, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000034437
Applies ToRSA Product Set: Web Threat Detection
RSA Product/Service Type: Mitigator
RSA Version/Condition: All
IssueCustomers might state that they have missing data, comparing two systems in same environments where they see data differences, missing clickstreams, or simply want to view raw data in their shards. 

TasksThese need to be investigated in the actual logfiles located in 
* the directories are arranged by
  • year
  • month
  • day
  • hour
ResolutionDo a webex session with the customer and ask them to go to the data in question to obtain the shard files. 
1. Determine the shard that needs to be extracted.  This will depend on the issue, however, in general the shards are arranged by the number of bits specified. The shards are distributed by IP's across the shards.
When you list the directory, the name of the shard will contain a bit value. 
2. The customer or CS Engineer will use a built-in utility to decrypt the targeted data files called logcat.  This utility is not officially supported but can be used. 
A.The command example will decrypt and move to a temp folder for collection.
logcat -f conf FILE.
The location of conf file is required and is usually in two places  var/opt/silvertail/etc/logcat.conf  or universal_conf.txt
Example --
/var/opt/silvertail/bin/logcat –f /var/opt/silvertail/etc/logcat.conf <logcat file>.crypt >  <temp folder>
B. Ask the customer to send these via SFTP so FRI CS Engineer can analyze on their lab system. 
Note -- Make your own .conf file that includes this information from Universal_conf of the system that will use logcat
------------- contents of file -----------------
        <x509 cert="/var/opt/silvertail/certs/silvertail.crt"
               key="/var/opt/silvertail/certs/silvertail.key" />
    <logger priority="INFO" />
3. Further analysis on the shard files using getshard 
     getshard –i <ip> -b 8
(8 is seen in the naming of the shards is the number of bits in size)
Example -- 
    getshard -b 8 -i
4. Explore shards with imported utility loggrep
loggrep can be obtained by FRI CS Engineers and can be used to look through the contents of a single shard. 
Example --# var/opt/silvertail/bin/loggrep –I <ip address> -m POST
Example --
cat the shard and run loggrep to find a particular IP address and REQUEST in the contents  do a count and print the unique entries (this will show if there are too many counts for one page, or too many unique pages because page normalization has not been performed).
cat <shardname.txt> | loggrep –I <ip address> | awk ‘/REQUEST/ {print $3} | awk –F’&’ ‘{print $2}’ | sort | uniq –c |sort –n
There are many ways to approach the analysis of shard files but this is an introductory to using these tools 

NotesNote-- decrypted shards will contain raw data and may contain sensitive information, these files should be handled in a secure fashion. 
Use of getshard --
[root@wtd etc]# /var/opt/silvertail/bin/getshard
Silver Tail shard generator version
getshard -i <ip> -b <bits> [-H] -s <string> [-v]
    -i <ip>      IP address to show the shard (required)
    -b <bits>    Number of shard bits. Default 0.
    -H           Display the hash value of ip (optional)
    -s <string>  Show the hash of string.
    -v           Display the version
Use of logcat -- 
    logcat -f conf FILE...
Use of loggrep
[root@gcVarzArchive verizon]# cat prodshard23Test2.txt |loggrep
Usage: /varz/bin/loggrep [options] ...
Options :
        -i <IP>
        -a <User-agent>
        -u <User>
        -m <Method>
        -p <Page>
        -s <Status>   (HTTP Response Code)
        -h <Host>     (HTTP Host Header)
        -z <Other>    (Any other header - see below)
        -v            Invert matching
All fields are Regex.  IP Address is anchored match, all others are partial.
"Other" format is TTTT&ffff&vvvv  where TTTT is the log entry type,
ffff is the field name, and vvvv is the value.  eg, to find the "test"
cookie being set to "true" use :
-z SETCOOKIE&test&^true$
Examples :
Find all POST requests from IP :
        /varz/bin/loggrep -i -m POST
Find all requests from Firefox with an argument of "search" with any value
        /varz/bin/loggrep -a firefox -z 'ARGS&search&.'