000015501 - Silvertail: How to determine which shard contains an IP

Document created by RSA Customer Support Employee on Jun 16, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000015501
IssueNeed to find which log file contains a specific IP address
Logs are stored as shards and it is hard to tell which log contains which IP addresses.

Convert IP(s) to shard value(s) using /var/opt/silvertail/bin/getshard -i <IP> -b 08


/var/opt/silvertail/bin/getshard -i 113.XX.XX.XX -b 08

Silver Tail shard generator .99

shard d5


How to gather logs based on that shard value

1. create temp folder

2. cd into temp folder

3. execute the following command per IP address (with updated month, shard, and ip address):

for i in {days separate by space - 01 02 31}; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2012/mm-mmm/$i/*/sb08/shard_###.log.gz.crypt | /home/silvertail/loggrep -i <IP> >> mm-mmm-$i.log; done

example with output files:

 for i in 01 02 03; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2013/11-Nov/$i/*/sb08/shard_fd.log.gz.crypt | /sts-scripts/scripts/loggrep -i >> Nov-11-$i.log; done

-rw-r--r-- 1 root       root       854476 Nov 19 10:00 Nov-11-01.log
-rw-r--r-- 1 root       root       856399 Nov 19 10:00 Nov-11-02.log
-rw-r--r-- 1 root       root       855758 Nov 19 10:00 Nov-11-03.log

NOTE: you will need to install or create loggrep utility which is an internal silvertail tool


Loggrep is here: Copy this entire section (starting with the #) and save as a unix text file, and give it execute permissions.

use strict;
use warnings;
use Getopt::Std;

# Version 1.00 - SH - Initial version
#         1.01 - SH - Added -v option

sub Usage() {
 print << "EOT";
Usage: $0 [options] ...

Options :
 -i <IP>
 -a <User-agent>
 -u <User>
 -m <Method>
 -p <Page>
 -s <Status>   (HTTP Response Code)
 -h <Host>     (HTTP Host Header)
 -z <Other>    (Any other header - see below)
 -v            Invert matching

All fields are Regex.  IP Address is anchored match, all others are partial.

"Other" format is TTTT&ffff&vvvv  where TTTT is the log entry type,
ffff is the field name, and vvvv is the value.  eg, to find the "test"
cookie being set to "true" use :
-z SETCOOKIE&test&^true\$

Examples :
Find all POST requests from IP :
 $0 -i -m POST

Find all requests from Firefox with an argument of "search" with any value
 $0 -a firefox -z 'ARGS&search&.'



sub GetVariable($$$) {
 my ($t, $Str, $V) = @_;

 return $1 if (($t=~/^cookie$/i)&&($Str =~ /^(?:.*; )?$V=([^;]*)/i));
 return $1 if ($Str =~ /^(?:.*&)?$V=([^&]*)/i);
 return undef;


my $IntReq;
my $Interesting=0;
my $Rec="";
my %opts;
my $Invert=0;

Usage() if ($#ARGV<0);

getopts('vi:u:a:m:p:z:s:h:', \%opts) || Usage();

my $scount=0;
my @Search;
push @Search, ($scount++).'&STTX&ip&^'.$opts{i}.'$' if ($opts{i});
push @Search, ($scount++).'&HEADERS&user-agent&'.$opts{a} if ($opts{a});
push @Search, ($scount++).'&USER&id&'.$opts{u} if ($opts{u});
push @Search, ($scount++).'&METHOD&method&'.$opts{m} if ($opts{m});
push @Search, ($scount++).'&REQUEST&page&'.$opts{p} if ($opts{p});
push @Search, ($scount++).'&STATUS&val&'.$opts{s} if ($opts{s});
push @Search, ($scount++).'&HEADERS&host&'.$opts{h} if ($opts{h});
push @Search, ($scount++).'&'.$opts{z} if ($opts{z});


$Invert++ if ($opts{v});

while(<>) {
 next unless (/^[tTD][0-9:.]+\t([A-Za-z0-9-]+)\t(.*)$/);
 my ($Type, $Val) = ($1,$2);

 if (/^T/) {
  print $Rec if ((($Interesting==$IntReq)&&!$Invert)||(($Interesting!=$IntReq)&&$Invert));
 my $x;

 foreach my $s (@Search) {
  my ($i, $t, $h, $v) = split('&',$s,4);
  $Interesting|=(2**$i)  if (($Type =~ /^$t$/i) && ($x = GetVariable($Type, $Val, $h)) && ($x =~ /$v/i));

The number of shard bits is the number of bits of the IP address that we use when determining the number of shards.  When determining the shard file, the IP hashed using FNV32.  We then take the low n bits of that hash (n is the number of shard bits), the resulting string is incorporated into the shard name, e.g. shard_a0.log.gz.crypt
Legacy Article IDa63996