on Jan 11, 2017Article Content
| Article Number | 000034437 |
| Applies To | RSA Product Set: Web Threat Detection RSA Product/Service Type: Mitigator RSA Version/Condition: All |
| Issue | Customers might state that they have missing data, comparing two systems in same environments where they see data differences, missing clickstreams, or simply want to view raw data in their shards. |
| Tasks | These need to be investigated in the actual logfiles located in /var/opt/silvertail/data/* * the directories are arranged by
|
| Resolution | Do a webex session with the customer and ask them to go to the data in question to obtain the shard files. 1. Determine the shard that needs to be extracted. This will depend on the issue, however, in general the shards are arranged by the number of bits specified. The shards are distributed by IP's across the shards. When you list the directory, the name of the shard will contain a bit value. 2. The customer or CS Engineer will use a built-in utility to decrypt the targeted data files called logcat. This utility is not officially supported but can be used. A.The command example will decrypt and move to a temp folder for collection. logcat -f conf FILE. The location of conf file is required and is usually in two places var/opt/silvertail/etc/logcat.conf or universal_conf.txt Example -- /var/opt/silvertail/bin/logcat –f /var/opt/silvertail/etc/logcat.conf <logcat file>.crypt > <temp folder> B. Ask the customer to send these via SFTP so FRI CS Engineer can analyze on their lab system. Note -- Make your own .conf file that includes this information from Universal_conf of the system that will use logcat ------------- contents of file ----------------- <build> <tls> <x509 cert="/var/opt/silvertail/certs/silvertail.crt" key="/var/opt/silvertail/certs/silvertail.key" /> </tls> <logger priority="INFO" /> </build> 3. Further analysis on the shard files using getshard getshard –i <ip> -b 8 (8 is seen in the naming of the shards is the number of bits in size) Example -- getshard -b 8 -i 192.168.1.2. 4. Explore shards with imported utility loggrep loggrep can be obtained by FRI CS Engineers and can be used to look through the contents of a single shard. Example --# var/opt/silvertail/bin/loggrep –I <ip address> -m POST Example -- cat the shard and run loggrep to find a particular IP address and REQUEST in the contents do a count and print the unique entries (this will show if there are too many counts for one page, or too many unique pages because page normalization has not been performed). cat <shardname.txt> | loggrep –I <ip address> | awk ‘/REQUEST/ {print $3} | awk –F’&’ ‘{print $2}’ | sort | uniq –c |sort –n There are many ways to approach the analysis of shard files but this is an introductory to using these tools |
| Notes | Note-- decrypted shards will contain raw data and may contain sensitive information, these files should be handled in a secure fashion. Use of getshard -- [root@wtd etc]# /var/opt/silvertail/bin/getshard Silver Tail shard generator version 5.1.1.5 getshard -i <ip> -b <bits> [-H] -s <string> [-v] -i <ip> IP address to show the shard (required) -b <bits> Number of shard bits. Default 0. -H Display the hash value of ip (optional) -s <string> Show the hash of string. -v Display the version Use of logcat -- logcat -f conf FILE... Use of loggrep [root@gcVarzArchive verizon]# cat prodshard23Test2.txt |loggrep Usage: /varz/bin/loggrep [options] ... Options : -i <IP> -a <User-agent> -u <User> -m <Method> -p <Page> -s <Status> (HTTP Response Code) -h <Host> (HTTP Host Header) -z <Other> (Any other header - see below) -v Invert matching All fields are Regex. IP Address is anchored match, all others are partial. "Other" format is TTTT&ffff&vvvv where TTTT is the log entry type, ffff is the field name, and vvvv is the value. eg, to find the "test" cookie being set to "true" use : -z SETCOOKIE&test&^true$ Examples : Find all POST requests from IP 192.0.2.55 : /varz/bin/loggrep -i 192.0.2.55 -m POST Find all requests from Firefox with an argument of "search" with any value /varz/bin/loggrep -a firefox -z 'ARGS&search&.' |