AnsweredAssumed Answered

Parsing McAfee Advanced Threat Defense Logs

Question asked by David Waugh Employee on Dec 18, 2015
Latest reply on Dec 7, 2016 by Thomas Sanda

I recently had a customer who needed to parse McAfee Advanced Threat Detection Logs. This is not officially supported by Security Analytics so it was necessary to write a custom parser.


The log format for McAfee ATD is similar to the following:


Dec 15 03:27:05 localhost ATD2ESM[13207]: {"Summary": { "Event_Type": "ATD File Report","MISversion": "","SUMversion": "","OSversion":"win7sp1x64","fileId": "Not Available","Parent MD5": "Not Available","ATD IP":"","Src IP": "","Dst IP": "","TaskId":"37","JobId": "37","JSONversion": "1.001.0718","hasDynamicAnalysis":"true","Subject": {"Name": "","Type": "PE32+ executable (console) x86-64","md5":"6AF8F4E3601156A59F050AAB4FAB5153","sha-1":"11BBBA1E7B39E1E193C6740B61F2A32E30ADD01A","size": "56832","Timestamp": "2014-12-15 11:24:12","parent_archive": "Not Available"},"Selectors": [{"Engine":"Sandbox","MalwareName": "Malware.Dynamic","Severity": "5"}],"Verdict":{"Severity": "5","Description": "Sample is malicious"},"Stats": [{"ID":"0","Category": "Persistence, Installation Boot Survival","Severity": "5"},{"ID":"1","Category": "Hiding, Camouflage, Stealthiness, Detection and Removal Protection","Severity": "0"},{"ID": "2","Category": "Security Solution / Mechanism bypass, termination and removal, Anti Debugging, VM Detection","Severity": "5"},{"ID": "3","Category": "Spreading","Severity": "2"},{"ID": "4","Category":"Exploiting, Shellcode","Severity": "0"},{"ID": "5","Category":"Networking","Severity": "3"},{"ID": "6","Category": "Data spying, Sniffing,Keylogging, Ebanking Fraud","Severity": "4"}],"Behavior": ["Created content under Windows system directory","Deleted AV auto-run registry key","Created a socket bound to a specific service provider and listen to an open port","Installed low level keyboard hook procedure","Deleted a key from auto-run registry entry","Altered auto-run registry entry that executed at next Windows boot"]}}


This is JSON and in a more human readable format would look as follows:


   "Summary": {

      "Event_Type": "ATD File Report",

      "MISversion": "",

      "SUMversion": "",

      "OSversion": "win7sp1x64",

      "fileId": "Not Available",

      "Parent MD5": "Not Available",

      "ATD IP": "",

      "Src IP": "",

      "Dst IP": "",

      "TaskId": "37",

      "JobId": "37",

      "JSONversion": "1.001.0718",

      "hasDynamicAnalysis": "true",

      "Subject": {

         "Name": "",

         "Type": "PE32+ executable (console) x86-64",

         "md5": "6AF8F4E3601156A59F050AAB4FAB5153",

         "sha-1": "11BBBA1E7B39E1E193C6740B61F2A32E30ADD01A",

         "size": "56832",

         "Timestamp": "2014-12-15 11:24:12",

         "parent_archive": "Not Available"


      "Selectors": [


            "Engine": "Sandbox",

            "MalwareName": "Malware.Dynamic",

            "Severity": "5"



      "Verdict": {

         "Severity": "5",

         "Description": "Sample is malicious"


      "Stats": [


            "ID": "0",

            "Category": "Persistence, Installation Boot Survival",

            "Severity": "5"



            "ID": "1",

            "Category": "Hiding, Camouflage, Stealthiness, Detection and Removal Protection",

            "Severity": "0"



            "ID": "2",

            "Category": "Security Solution / Mechanism bypass, termination and removal, Anti Debugging, VM Detection",

            "Severity": "5"



            "ID": "3",

            "Category": "Spreading",

            "Severity": "2"



            "ID": "4",

            "Category": "Exploiting, Shellcode",

            "Severity": "0"



            "ID": "5",

            "Category": "Networking",

            "Severity": "3"



            "ID": "6",

            "Category": "Data spying, Sniffing,Keylogging, Ebanking Fraud",

            "Severity": "4"



      "Behavior": [

         "Created content under Windows system directory",

         "Deleted AV auto-run registry key",

         "Created a socket bound to a specific service provider and listen to an open port",

         "Installed low level keyboard hook procedure",

         "Deleted a key from auto-run registry entry",

         "Altered auto-run registry entry that executed at next Windows boot"





The difficulty I had with these messages is that the Behaviour fields and Stats field could contain any number of items. In this case the behaviour field contains 7 or more entries ID:0 to ID:6 and the behaviour field contains 6 entries. These are not necessarily maximum values and actual test logs would be needed to determine the maximum and minimum possible entries.


To parse this log I did the following:

  • Create a single message for each possible Stats field. Eg a Message for ID=0, A message for ID=0 ,ID=1 all the way up to the possibility that there were 7 entries in the stats field.
  • Used the {a | a,b | a,b,c|a,b,c,d|a,b,c,d,e|a,b,c,d,e,f} to handle the behaviour part of the field.

Other possibilities that I tried were:

  • Using a JSON parser. Unfortunately, the maximum amount of meta that a variable can hold is 256 characters and so I was unable to capture the complete string. The JSON parsers I found also used functions that were not supported by Security Analytics so could not be imported.


Here is the necessary table-map-custom.xml keys to use with the parser. These can be mapped to meta fields as desired.


<!-- BEGIN List of keys Not in table-map-custom.xml -->

<mapping envisionName="matd.behaviour" nwName="matd.behaviour" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour1" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour2" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour3" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour4" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour5" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour6" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.description" nwName="event.desc" flags="None" format="Text"/>

<mapping envisionName="matd.fileid" nwName="matd.fileid" flags="None" format="Text"/>

<mapping envisionName="matd.filename" nwName="url" flags="None" format="Text"/>

<mapping envisionName="matd.filetype" nwName="filetype" flags="None" format="Text"/>

<mapping envisionName="matd.isdynamic" nwName="matd.isdynamic" flags="None" format="Text"/>

<mapping envisionName="matd.jobid" nwName="matd.jobid" flags="None" format="Text"/>

<mapping envisionName="matd.jsonversion" nwName="matd.jsonversion" flags="None" format="Text"/>

<mapping envisionName="matd.md5checksum" nwName="matd.md5checksum" flags="None" format="Text"/>

<mapping envisionName="matd.misversion" nwName="matd.misversion" flags="None" format="Text"/>

<mapping envisionName="matd.osversion" nwName="matd.osversion" flags="None" format="Text"/>

<mapping envisionName="matd.parentarch" nwName="matd.parentarch" flags="None" format="Text"/>

<mapping envisionName="matd.parentmd5" nwName="matd.parentmd5" flags="None" format="Text"/>

<mapping envisionName="matd.selector" nwName="matd.selector" flags="None" format="Text"/>

<mapping envisionName="matd.severity" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sha1checksum" nwName="matd.sha1checksum" flags="None" format="Text"/>

<mapping envisionName="matd.size" nwName="matd.size" flags="None" format="Text"/>

<mapping envisionName="matd.stats" nwName="matd.stats" flags="None" format="Text"/>

<mapping envisionName="matd.sumversion" nwName="matd.sumversion" flags="None" format="Text"/>

<mapping envisionName="matd.taskid" nwName="matd.taskid" flags="None" format="Text"/>

<mapping envisionName="matd.time" nwName="matd.time" flags="None" format="Text"/>

<mapping envisionName="matd.cat0" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat1" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat2" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat3" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat4" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat5" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat6" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.sev0" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev1" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev2" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev3" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev4" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev5" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev6" nwName="severity" flags="None" format="Text"/>

<!-- END List of keys Not in table-map-custom.xml -->