000034437 - How to analyze the data log files in RSA Web Threat Detection

Document created by RSA Customer Support Employee on Jan 11, 2017Last modified by RSA Customer Support on Jun 26, 2018
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000034437
Applies ToRSA Product Set: Web Threat Detection
RSA Product/Service Type: Mitigator
RSA Version/Condition: 5.x, 6.x
IssueCustomers might state that they have missing data, comparing two systems in same environments where they see data differences, missing clickstreams or simply want to view raw data in their shards. 
TasksThese need to be investigated in the actual log files located in the /var/opt/silvertail/data directory.

The directories are arranged by the following:
  • year
  • month
  • day
  • hour
ResolutionDo a webex session with the customer and ask them to go to the data in question to obtain the shard files. 
  1. Determine the shard that needs to be extracted.  This will depend on the issue, however, in general the shards are arranged by the number of bits specified. The shards are distributed by IP's across the shards.  When you list the directory, the name of the shard will contain a bit value.
  2. The customer or CS Engineer will use a built-in utility to decrypt the targeted data files called logcat.  This utility is not officially supported but can be used. 
    1. The command below will decrypt and move to a temp folder for collection.

      logcat -f conf FILE

        The location of conf file is required and is usually in two places  var/opt/silvertail/etc/logcat.conf  or universal_conf.txt.  For example:


      /var/opt/silvertail/bin/logcat –f /var/opt/silvertail/etc/logcat.conf <logcat file>.crypt > <temp folder>

    2. Ask the customer to send these via SFTP so FRI CS Engineer can analyze on their lab system.

      Note:  Make your own .conf file that includes this information from Universal_conf of the system that will use logcat.


      ------------- contents of file -----------------
              <x509 cert="/var/opt/silvertail/certs/silvertail.crt"
                     key="/var/opt/silvertail/certs/silvertail.key" />
          <logger priority="INFO" />

  3. Further analysis on the shard files using getshard.

    getshard –i <ip> -b 8

    The 8 is seen in the naming of the shards is the number of bits in size. For example:

    getshard -b 8 -i

  4. Explore shards with imported utility loggrep. The loggrep utility can be obtained by FRI CS Engineers and can be used to look through the contents of a single shard.   For example:

    # var/opt/silvertail/bin/loggrep –I <ip address> -m POST

Cat the shard and run loggrep to find a particular IP address and REQUEST in the contents, do a count and print the unique entries. (This will show if there are too many counts for one page, or too many unique pages because page normalization has not been performed.)

cat <shardname.txt> | loggrep –I <ip address> | awk ‘/REQUEST/ {print $3} | awk –F’&’ ‘{print $2}’ | sort | uniq –c |sort –n

There are many ways to approach the analysis of shard files but this is an introductory to using these tools.
NotesDecrypted shards will contain raw data and may contain sensitive information. These files should be handled in a secure fashion.

Use of getshard

[root@wtd etc]# /var/opt/silvertail/bin/getshard
Silver Tail shard generator version
getshard -i <ip> -b <bits> [-H] -s <string> [-v]
    -i <ip>      IP address to show the shard (required)
    -b <bits>    Number of shard bits. Default 0.
    -H           Display the hash value of ip (optional)
    -s <string>  Show the hash of string.
    -v           Display the version

Use of logcat

logcat -f conf FILE...

Use of loggrep

[root@gcVarzArchive verizon]# cat prodshard23Test2.txt |loggrep
Usage: /varz/bin/loggrep [options] ...

Options :
        -i <IP>
        -a <User-agent>
        -u <User>
        -m <Method>
        -p <Page>
        -s <Status>   (HTTP Response Code)
        -h <Host>     (HTTP Host Header)
        -z <Other>    (Any other header - see below)
        -v            Invert matching

All fields are Regex.  IP Address is anchored match, all others are partial.

The "Other" format is TTTT&ffff&vvvv  where TTTT is the log entry type, ffff is the field name, and vvvv is the value.  eg, to find the "test" cookie being set to "true" use:

-z SETCOOKIE&test&^true$


  • Find all POST requests from IP

    /varz/bin/loggrep -i -m POST

  • Find all requests from Firefox with an argument of "search" with any value

    /varz/bin/loggrep -a firefox -z 'ARGS&search&.'