000030743 - Parsing a tab delimited log file into RSA Security Analytics

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 5Show Document
  • View in full screen mode

Article Content

Article Number000030743
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.X
Platform: CentOS
O/S Version: 6
IssueThis article describes how to parse a tab delimited log file into Security Analytics.
The log file is of the following form and is tab delimited: (Note: the contents below should all be on a single line.)
 
#Fields: datatime c-ip x-ss-company-id cs(X-Forwarded-For) cs-username cs-method cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs(User-Agent) 
cs(Content-Type) cs-bytes sc-bytes sc-status sc(Content-Type) s-ip x-ss-category x-ss-last-rule-name x-ss-last-rule-action x-ss-block-type x-ss-block-value 
x-ss-external-ip x-ss-referer-host 2015-07-10 11:39:37 GMT 10.106.21.99 2164457336 10.106.21.99 CONNECT https www.ibm.com 443 / curl/7.19.7 (x86_64-redhat-linux-gnu) 
libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - 0 0 0 23.32.171.219 c:busi default block adv-rule-match No exception exists to allow 
this web page 128.221.224.200 2015-07-10 11:41:40 GMT 10.106.21.99 2164457336 10.106.21.99 CONNECT https www.ibm.com 443 / curl/7.19.7 (x86_64-redhat-linux-gnu) 
libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - 0 0 0 23.32.171.219 c:busi default block adv-rule-match No exception exists to allow 
this web page 168.159.213.199
TasksThis procedure uploads the file to a Log Collector using the File collection method. This enables replacing the TAB delimiters with an alternative delimiter and also append some text onto the start of each line of the log to assist with parsing.
Create a New Event Source File Type
On the Log Collector create the following file /etc/netwitness/ng/logcollection/content/collection/file/ciscocwsctm.xml  
<?xml version="1.0" encoding="UTF-8"?>
<typespec>
   <name>ciscocwsctm</name>
   <type>file</type>
   <prettyName>ciscocwsctm</prettyName>
   <version>1.0</version>
   <author>administrator</author>
   <description>FileCollection specification for eventsource type Cisco CWS Custom using file handler type "ciscocwsctm"</description>
<device>
  <name>ciscocwsctm</name>
</device>
<configuration>
</configuration>
<collection>
         <file>
                <parserId>file.ciscocwsctm</parserId>
                <processorType>generic</processorType>
  <dataStartLine>2</dataStartLine>
          <fieldDelim>0x09</fieldDelim>
          <idField></idField>
          <lineDelim>\n</lineDelim>
          <transformPrefixTag>CISCOCWSCTM</transformPrefixTag>
          <transformReplaceFieldDelim>1</transformReplaceFieldDelim>
          <transformPrefixFilename>0</transformPrefixFilename>
          <transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
          <transformReplacementFieldDelim>^^</transformReplacementFieldDelim>
         </file>
</collection>
</typespec>

Important points about this file:
  • Replace TAB delimiters (0x09) with ^^ delimters to aid parsing
  • Add %CISCOWSCTM to each line of the log file to aid parsing
  • Restart the logcollector service (restart nwlogcollector) to apply this or the changes to take effect.
Create a New Parser
Create a new directory called  /etc/netwitness/ng/envision/etc/devices/ciscocwsctm
on the logdecoder.
Copy the attached files to this location
/etc/netwitness/ng/envision/etc/devices/ciscocwsctm/ciscocwsctm.ini 

/etc/netwitness/ng/envision/etc/devices/ciscocwsctm/ciscocwsctmmsg.xml 

Edit /etc/netwitness/ng/envision/etc/table-map-custom.xml xml configuration file to include the following:
<!--My Custom Parser --> 
<mapping envisionName="csusername" nwName="user.src" flags="None"/>
<mapping envisionName="xforwardfor" nwName="xfactor" flags="None"/>
<mapping envisionName="csmethod" nwName="web_method" flags="None"/>
<mapping envisionName="csurischeme" nwName="protocol" flags="None"/>
<mapping envisionName="cshost" nwName="alias.host" flags="None"/>
<mapping envisionName="csuriport" nwName="dport" flags="None"/>
<mapping envisionName="csuripath" nwName="url.raw" flags="None"/>
<mapping envisionName="csuriquery" nwName="query" flags="None"/>
<mapping envisionName="csuseragent" nwName="user.agent" flags="None"/>
<mapping envisionName="cscontenttype" nwName="content" flags="None"/>
<mapping envisionName="csbytes" nwName="sbytes" flags="None"/>
<mapping envisionName="scbytes" nwName="rbytes" flags="None"/>
<mapping envisionName="scstatus" nwName="result.code" flags="None"/>
<mapping envisionName="sccontenttype" nwName="content" flags="None"/>
<mapping envisionName="xsscategory" nwName="category" flags="None"/>
<mapping envisionName="xssname" nwName="policy.name" flags="None"/>
<mapping envisionName="xsslastruleaction" nwName="action" flags="None"/>
<mapping envisionName="xssblocktype" nwName="policy.name" flags="None"/>
<mapping envisionName="xssblockvalue" nwName="result" flags="None"/>
<mapping envisionName="xssexternalip" nwName="dtransaddr" flags="None"/>
<mapping envisionName="xssreferhost" nwName="referer" flags="None"/>
<!-- My Custom Parser -->

Restart the log decoder with 
restart nwlogdecoder
Resolution

The parsing works because the log file is in a simple structured format.
This means that the parser only needs to expect one message.
This is defined as follows in the ciscowsctmmsg.xml file: (Note: the contents below should all be on a single line.)


content="&lt;saddr&gt;^^&lt;xsscompanyid&gt;^^&lt;xforwardfor&gt;^^&lt;csusername&gt;^^&lt;csmethod&gt;^^&lt;csurischeme&gt;^^&lt;cshost&gt;^^&lt;csuriport&gt;^^&lt;csuripath&gt;
^^&lt;csuriquery&gt;^^&lt;csuseragent&gt;^^&lt;cscontenttype&gt;^^&lt;csbytes&gt;^^&lt;scbytes&gt;^^&lt;scstatus&gt;^^&lt;scontenttype&gt;^^&lt;daddr&gt;^^&lt;xsscategory&gt;^^&lt;
xssname&gt;^^&lt;xsslastruleaction&gt;^^&lt;xssblocktype&gt;^^&lt;xssblockvalue&gt;^^&lt;xssexternalip&gt;^^&lt;xssreferhost&gt;"/>


The table-map-custom.xml file maps these fields into our Security Analytics meta keys.
 

Outcomes