000015501 - Silvertail: How to determine which shard contains an IP

Convert IP(s) to shard value(s) using /var/opt/silvertail/bin/getshard -i <IP> -b 08

example:

/var/opt/silvertail/bin/getshard -i 113.XX.XX.XX -b 08

Silver Tail shard generator .99

shard d5

How to gather logs based on that shard value

1. create temp folder

2. cd into temp folder

3. execute the following command per IP address (with updated month, shard, and ip address):

for i in {days separate by space - 01 02 31}; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2012/mm-mmm/$i/*/sb08/shard_###.log.gz.crypt | /home/silvertail/loggrep -i <IP> >> mm-mmm-$i.log; done

example with output files:

 for i in 01 02 03; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2013/11-Nov/$i/*/sb08/shard_fd.log.gz.crypt | /sts-scripts/scripts/loggrep -i 10.101.99.151 >> Nov-11-$i.log; done

-rw-r--r-- 1 root       root       854476 Nov 19 10:00 Nov-11-01.log
-rw-r--r-- 1 root       root       856399 Nov 19 10:00 Nov-11-02.log
-rw-r--r-- 1 root       root       855758 Nov 19 10:00 Nov-11-03.log

NOTE: you will need to install or create loggrep utility which is an internal silvertail tool

Loggrep is here: Copy this entire section (starting with the #) and save as a unix text file, and give it execute permissions.

#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Std;

#
# Version 1.00 - SH - Initial version
#         1.01 - SH - Added -v option
#

sub Usage() {
 print << "EOT";
Usage: $0 [options] ...

Options :
 -i <IP>
 -a <User-agent>
 -u <User>
 -m <Method>
 -p <Page>
 -s <Status>   (HTTP Response Code)
 -h <Host>     (HTTP Host Header)
 -z <Other>    (Any other header - see below)
 -v            Invert matching

All fields are Regex.  IP Address is anchored match, all others are partial.

"Other" format is TTTT&ffff&vvvv  where TTTT is the log entry type,
ffff is the field name, and vvvv is the value.  eg, to find the "test"
cookie being set to "true" use :
-z SETCOOKIE&test&^true\$

Examples :
Find all POST requests from IP 192.0.2.55 :
 $0 -i 192.0.2.55 -m POST

Find all requests from Firefox with an argument of "search" with any value
 $0 -a firefox -z 'ARGS&search&.'

EOT

 exit(1);
}

sub GetVariable($$$) {
 my ($t, $Str, $V) = @_;

 return $1 if (($t=~/^cookie$/i)&&($Str =~ /^(?:.*; )?$V=([^;]*)/i));
 return $1 if ($Str =~ /^(?:.*&)?$V=([^&]*)/i);
 return undef;
}

my $IntReq;
my $Interesting=0;
my $Rec="";
my %opts;
my $Invert=0;

Usage() if ($#ARGV<0);

getopts('vi:u:a:m:p:z:s:h:', \%opts) || Usage();

my $scount=0;
my @Search;
push @Search, ($scount++).'&STTX&ip&^'.$opts{i}.'$' if ($opts{i});
push @Search, ($scount++).'&HEADERS&user-agent&'.$opts{a} if ($opts{a});
push @Search, ($scount++).'&USER&id&'.$opts{u} if ($opts{u});
push @Search, ($scount++).'&METHOD&method&'.$opts{m} if ($opts{m});
push @Search, ($scount++).'&REQUEST&page&'.$opts{p} if ($opts{p});
push @Search, ($scount++).'&STATUS&val&'.$opts{s} if ($opts{s});
push @Search, ($scount++).'&HEADERS&host&'.$opts{h} if ($opts{h});
push @Search, ($scount++).'&'.$opts{z} if ($opts{z});

$IntReq=(2**$scount)-1;;

$Invert++ if ($opts{v});

while(<>) {
 $Rec.=$_;
 chomp;
 next unless (/^[tTD][0-9:.]+\t([A-Za-z0-9-]+)\t(.*)$/);
 my ($Type, $Val) = ($1,$2);

 if (/^T/) {
  print $Rec if ((($Interesting==$IntReq)&&!$Invert)||(($Interesting!=$IntReq)&&$Invert));
  $Interesting=0;
  $Rec="";
  next;
 }
  
 my $x;

 foreach my $s (@Search) {
  my ($i, $t, $h, $v) = split('&',$s,4);
  $Interesting|=(2**$i)  if (($Type =~ /^$t$/i) && ($x = GetVariable($Type, $Val, $h)) && ($x =~ /$v/i));
 }
}