Article Content
Article Number | 000015501 |
Issue | Need to find which log file contains a specific IP address Logs are stored as shards and it is hard to tell which log contains which IP addresses. |
Resolution | Convert IP(s) to shard value(s) using /var/opt/silvertail/bin/getshard -i <IP> -b 08 example: /var/opt/silvertail/bin/getshard -i 113.XX.XX.XX -b 08 Silver Tail shard generator .99 shard d5
How to gather logs based on that shard value 1. create temp folder 2. cd into temp folder 3. execute the following command per IP address (with updated month, shard, and ip address): for i in {days separate by space - 01 02 31}; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2012/mm-mmm/$i/*/sb08/shard_###.log.gz.crypt | /home/silvertail/loggrep -i <IP> >> mm-mmm-$i.log; done example with output files: for i in 01 02 03; do /var/opt/silvertail/bin/logcat -f /var/opt/silvertail/etc/logcat.conf /var/opt/silvertail/data/logs/2013/11-Nov/$i/*/sb08/shard_fd.log.gz.crypt | /sts-scripts/scripts/loggrep -i 10.101.99.151 >> Nov-11-$i.log; done -rw-r--r-- 1 root root 854476 Nov 19 10:00 Nov-11-01.log NOTE: you will need to install or create loggrep utility which is an internal silvertail tool |
Notes | Loggrep is here: Copy this entire section (starting with the #) and save as a unix text file, and give it execute permissions. #!/usr/bin/perl #
Options : All fields are Regex. IP Address is anchored match, all others are partial. "Other" format is TTTT&ffff&vvvv where TTTT is the log entry type, Examples : Find all requests from Firefox with an argument of "search" with any value EOT exit(1); sub GetVariable($$$) { return $1 if (($t=~/^cookie$/i)&&($Str =~ /^(?:.*; )?$V=([^;]*)/i));
my $IntReq; Usage() if ($#ARGV<0); getopts('vi:u:a:m:p:z:s:h:', \%opts) || Usage(); my $scount=0; $IntReq=(2**$scount)-1;; $Invert++ if ($opts{v}); while(<>) { if (/^T/) { foreach my $s (@Search) { The number of shard bits is the number of bits of the IP address that we use when determining the number of shards. When determining the shard file, the IP hashed using FNV32. We then take the low n bits of that hash (n is the number of shard bits), the resulting string is incorporated into the shard name, e.g. shard_a0.log.gz.crypt |
Legacy Article ID | a63996 |