Nikolay Klender

Automated Threat Detection for Log Decoder

Discussion created by Nikolay Klender on May 23, 2016
Latest reply on May 23, 2016 by Mark Karlstrand

New component of ESA server Automated Threat Detection (ATD) is very promising. According to manual it analyzes HTTP packets which are parsed by the Decoder and sent to the ESA device. But if you have Log Decoder only you are out of luck (according to manual). When I asked support team of RSA about possibility to use ATD with logs from proxy device (squid, BlueCoat, etc) I received following answer:

Automated Threat Detection is an analytics engine that examines your HTTP data hence only supported for packets in combination with LUA parser.

There are no plans to expand it with logs functionality.

http://sadocs.emc.com/0_en-us/088_SA106/50_Alrt/66_C2Thrt/00_TDInfo

But all this seems strange because web proxy logs contains all necessary information.

If we look at meta from http_lua parser and contains of file /opt/rsa/esa/topology/C2.topology we can understood that following meta are analyzed by ATD:

  • action (GET, POST, etc)
  • ip_src
  • alias_host - web domain
  • client - user agent
  • referer
  • time
  • service (must be equal 80)

Log decoder do not fill service meta, that is why ATD do not analyze logs from LogDecoder. Thats why we need to modify parser to extract required meta and inject service meta of value 80. Lets take cacheflowelff parser as an example.

1) In  table-map-custom.xml we associate new xservice field with service meta

          <mapping envisionName="xservice" nwName="service" flags="None" format="UInt32"/>

2) open v20_cacheflowelffmsg.xml and edit content section of GET, POST, etc message id.

So here we inject xservice field with value 80 (will be moved to service meta), extract User-Agent to agent field (will be moved to client meta), and extract referer field. Please look table-map.xml for more information about mapping fields from envision parsers to netwitness metas

 

content="&lt;@xservice:80&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@fld63:*PARMVAL(username)&gt;&lt;@fld61:*PARMVAL(h_code)&gt;&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@domain:*URL($DOMAIN,url)&gt;&lt;@web_root:*URL($ROOT,url)&gt; &lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld1,fld2)&gt;&lt;@event_source:*PARMVAL(hostname1)&gt;&lt;@dhost:*PARMVAL(web_host)&gt;date=&lt;fld1&gt;,time=&lt;fld2&gt;,time-taken=&lt;processing_time&gt;,c-ip=&lt;saddr&gt;,s-action=&lt;action&gt;,sc-status=&lt;resultcode&gt;,sc-bytes=&lt;sbytes&gt;,cs-bytes=&lt;rbytes&gt;,cs-method=&lt;web_method&gt;,cs-user=&lt;username1&gt;,cs-username=&lt;username&gt;,cs-uri-username=&lt;username&gt;,s-hierarchy=&lt;h_code&gt;,cs-host=&lt;web_host&gt;,rs(Content-Type)=&lt;content_type&gt;,cs-uri-port=&lt;dport&gt;,s-ip=&lt;hostip&gt;,r-ip=&lt;dtransaddr&gt;,r-supplier-ip=&lt;fld37&gt;,r-dns=&lt;fld38&gt;,c-port=&lt;sport&gt;,cs-category=&lt;category&gt;,cs-uri-scheme=&lt;network_service&gt;,duration=&lt;duration&gt;,s-supplier-ip=&lt;daddr&gt;,cs-auth-group=&lt;group_object&gt;,s-supplier-name=&lt;fld68&gt;,sc-filter-result=&lt;disposition&gt;,sc-filter-category=&lt;filter&gt;,cs(User-Agent)=&lt;agent&gt;,x-virus-id=&lt;virusname&gt;,s-sitename=&lt;service&gt;,cs-uri=&lt;url&gt;,cs-uri-path=&lt;webpage&gt;,x-exception-id=&lt;fld88&gt;,cs-categories=&lt;category&gt;,cs(Referer)=&lt;referer&gt;,cs-uri-query=&lt;web_query&gt;,cs-uri-extension=&lt;web_extension&gt;,cs(Cookie)=&lt;web_cookie&gt;,s-computername=&lt;hostname1&gt;,s-port=&lt;network_port&gt;,cs-version=&lt;version&gt;,cs-auth-groups=&lt;group_object&gt;,cs-uri-stem=&lt;fld3&gt;,localtime=&lt;fld5&gt;,x-bluecoat-application-name=&lt;fld4&gt;,x-bluecoat-application-operation=&lt;fld6&gt;,s-icap-status=&lt;fld7&gt;,s-icap-info=&lt;fld8&gt;"

3)restart logdecoder

stop nwlogdecoder && start nwlogdecoder

4) configure ATD according to manual. After 24 hours you should see some new incidents

Outcomes