AnsweredAssumed Answered

Parsing McAfee Advanced Threat Defense Logs

Question asked by David Waugh Employee on Dec 18, 2015
Latest reply on Dec 7, 2016 by Thomas Sanda

I recently had a customer who needed to parse McAfee Advanced Threat Detection Logs. This is not officially supported by Security Analytics so it was necessary to write a custom parser.

 

The log format for McAfee ATD is similar to the following:

 

Dec 15 03:27:05 localhost ATD2ESM[13207]: {"Summary": { "Event_Type": "ATD File Report","MISversion": "3.4.4.2.43772","SUMversion": "3.4.4.2.43772","OSversion":"win7sp1x64","fileId": "Not Available","Parent MD5": "Not Available","ATD IP":"10.213.248.17","Src IP": "10.213.248.69","Dst IP": "10.213.248.107","TaskId":"37","JobId": "37","JSONversion": "1.001.0718","hasDynamicAnalysis":"true","Subject": {"Name": "http://10.213.248.107/Apoorv/samples/automation_samples/vtest64.exe","Type": "PE32+ executable (console) x86-64","md5":"6AF8F4E3601156A59F050AAB4FAB5153","sha-1":"11BBBA1E7B39E1E193C6740B61F2A32E30ADD01A","size": "56832","Timestamp": "2014-12-15 11:24:12","parent_archive": "Not Available"},"Selectors": [{"Engine":"Sandbox","MalwareName": "Malware.Dynamic","Severity": "5"}],"Verdict":{"Severity": "5","Description": "Sample is malicious"},"Stats": [{"ID":"0","Category": "Persistence, Installation Boot Survival","Severity": "5"},{"ID":"1","Category": "Hiding, Camouflage, Stealthiness, Detection and Removal Protection","Severity": "0"},{"ID": "2","Category": "Security Solution / Mechanism bypass, termination and removal, Anti Debugging, VM Detection","Severity": "5"},{"ID": "3","Category": "Spreading","Severity": "2"},{"ID": "4","Category":"Exploiting, Shellcode","Severity": "0"},{"ID": "5","Category":"Networking","Severity": "3"},{"ID": "6","Category": "Data spying, Sniffing,Keylogging, Ebanking Fraud","Severity": "4"}],"Behavior": ["Created content under Windows system directory","Deleted AV auto-run registry key","Created a socket bound to a specific service provider and listen to an open port","Installed low level keyboard hook procedure","Deleted a key from auto-run registry entry","Altered auto-run registry entry that executed at next Windows boot"]}}

 

This is JSON and in a more human readable format would look as follows:

{

   "Summary": {

      "Event_Type": "ATD File Report",

      "MISversion": "3.4.4.2.43772",

      "SUMversion": "3.4.4.2.43772",

      "OSversion": "win7sp1x64",

      "fileId": "Not Available",

      "Parent MD5": "Not Available",

      "ATD IP": "10.213.248.17",

      "Src IP": "10.213.248.69",

      "Dst IP": "10.213.248.107",

      "TaskId": "37",

      "JobId": "37",

      "JSONversion": "1.001.0718",

      "hasDynamicAnalysis": "true",

      "Subject": {

         "Name": "http://10.213.248.107/Apoorv/samples/automation_samples/vtest64.exe",

         "Type": "PE32+ executable (console) x86-64",

         "md5": "6AF8F4E3601156A59F050AAB4FAB5153",

         "sha-1": "11BBBA1E7B39E1E193C6740B61F2A32E30ADD01A",

         "size": "56832",

         "Timestamp": "2014-12-15 11:24:12",

         "parent_archive": "Not Available"

      },

      "Selectors": [

         {

            "Engine": "Sandbox",

            "MalwareName": "Malware.Dynamic",

            "Severity": "5"

         }

      ],

      "Verdict": {

         "Severity": "5",

         "Description": "Sample is malicious"

      },

      "Stats": [

         {

            "ID": "0",

            "Category": "Persistence, Installation Boot Survival",

            "Severity": "5"

         },

         {

            "ID": "1",

            "Category": "Hiding, Camouflage, Stealthiness, Detection and Removal Protection",

            "Severity": "0"

         },

         {

            "ID": "2",

            "Category": "Security Solution / Mechanism bypass, termination and removal, Anti Debugging, VM Detection",

            "Severity": "5"

         },

         {

            "ID": "3",

            "Category": "Spreading",

            "Severity": "2"

         },

         {

            "ID": "4",

            "Category": "Exploiting, Shellcode",

            "Severity": "0"

         },

         {

            "ID": "5",

            "Category": "Networking",

            "Severity": "3"

         },

         {

            "ID": "6",

            "Category": "Data spying, Sniffing,Keylogging, Ebanking Fraud",

            "Severity": "4"

         }

      ],

      "Behavior": [

         "Created content under Windows system directory",

         "Deleted AV auto-run registry key",

         "Created a socket bound to a specific service provider and listen to an open port",

         "Installed low level keyboard hook procedure",

         "Deleted a key from auto-run registry entry",

         "Altered auto-run registry entry that executed at next Windows boot"

      ]

   }

}

 

The difficulty I had with these messages is that the Behaviour fields and Stats field could contain any number of items. In this case the behaviour field contains 7 or more entries ID:0 to ID:6 and the behaviour field contains 6 entries. These are not necessarily maximum values and actual test logs would be needed to determine the maximum and minimum possible entries.

 

To parse this log I did the following:

  • Create a single message for each possible Stats field. Eg a Message for ID=0, A message for ID=0 ,ID=1 all the way up to the possibility that there were 7 entries in the stats field.
  • Used the {a | a,b | a,b,c|a,b,c,d|a,b,c,d,e|a,b,c,d,e,f} to handle the behaviour part of the field.

Other possibilities that I tried were:

  • Using a JSON parser. Unfortunately, the maximum amount of meta that a variable can hold is 256 characters and so I was unable to capture the complete string. The JSON parsers I found also used functions that were not supported by Security Analytics so could not be imported.

 

Here is the necessary table-map-custom.xml keys to use with the parser. These can be mapped to meta fields as desired.

 

<!-- BEGIN List of keys Not in table-map-custom.xml -->

<mapping envisionName="matd.behaviour" nwName="matd.behaviour" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour1" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour2" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour3" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour4" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour5" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.behaviour6" nwName="threat.desc" flags="None" format="Text"/>

<mapping envisionName="matd.description" nwName="event.desc" flags="None" format="Text"/>

<mapping envisionName="matd.fileid" nwName="matd.fileid" flags="None" format="Text"/>

<mapping envisionName="matd.filename" nwName="url" flags="None" format="Text"/>

<mapping envisionName="matd.filetype" nwName="filetype" flags="None" format="Text"/>

<mapping envisionName="matd.isdynamic" nwName="matd.isdynamic" flags="None" format="Text"/>

<mapping envisionName="matd.jobid" nwName="matd.jobid" flags="None" format="Text"/>

<mapping envisionName="matd.jsonversion" nwName="matd.jsonversion" flags="None" format="Text"/>

<mapping envisionName="matd.md5checksum" nwName="matd.md5checksum" flags="None" format="Text"/>

<mapping envisionName="matd.misversion" nwName="matd.misversion" flags="None" format="Text"/>

<mapping envisionName="matd.osversion" nwName="matd.osversion" flags="None" format="Text"/>

<mapping envisionName="matd.parentarch" nwName="matd.parentarch" flags="None" format="Text"/>

<mapping envisionName="matd.parentmd5" nwName="matd.parentmd5" flags="None" format="Text"/>

<mapping envisionName="matd.selector" nwName="matd.selector" flags="None" format="Text"/>

<mapping envisionName="matd.severity" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sha1checksum" nwName="matd.sha1checksum" flags="None" format="Text"/>

<mapping envisionName="matd.size" nwName="matd.size" flags="None" format="Text"/>

<mapping envisionName="matd.stats" nwName="matd.stats" flags="None" format="Text"/>

<mapping envisionName="matd.sumversion" nwName="matd.sumversion" flags="None" format="Text"/>

<mapping envisionName="matd.taskid" nwName="matd.taskid" flags="None" format="Text"/>

<mapping envisionName="matd.time" nwName="matd.time" flags="None" format="Text"/>

<mapping envisionName="matd.cat0" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat1" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat2" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat3" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat4" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat5" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.cat6" nwName="threat.category" flags="None" format="Text"/>

<mapping envisionName="matd.sev0" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev1" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev2" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev3" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev4" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev5" nwName="severity" flags="None" format="Text"/>

<mapping envisionName="matd.sev6" nwName="severity" flags="None" format="Text"/>

<!-- END List of keys Not in table-map-custom.xml -->

 

 

Outcomes