Ian Redden

LUA - Common Event Format (CEF) Script Template

Blog Post created by Ian Redden Employee on Oct 19, 2016

The CEF helper script attempts to fill the gap of CEF extensions that the system parser does not currently parse. For a complete list of what we do parse, please refer to Supported CEF Meta Keys - RSA Security Analytics Documentation .

 

This CEF helper template was written to be highly configurable as well as adhere to revision 16 of the Common Event Format (CEF) standards document. Its aim is to be able to parse meta data into any meta key in RSA Netwitness from any security appliance with the least amount of programming.


This is performed by several routines within the code that associates “csX” and “cnX” within a CEF extension with their appropriate “csXLabel” and “cnXLabel”. If no “csLabel” exists, it will use “csX” or “cnX” as the key name (where X is a numeric value). For example, a CEF message containing “cs3=name cs3Label=Ian” will set the key name to “name” and the value to “Ian” whereas a CEF message containing “cs3=Ian” will set the key name to “cs3” and the value to “Ian”. However, using scripts configuration, you can translate cs3 to any meta key name you want (i.e. username). This is configured in the t_keys_to_use variable.  More information on that below!

 

As CEF messages can also contain carriage returns (\n), if a carriage return is found, it will parse each line into a separate meta value of the same key name. For example, a CEF message containing “cs2=user.names cs2Label=iredden\nepartington” will result in 2 meta values (user.names) containing “iredden” and “epartington”.

 

The template can be downloaded at the bottom of this article.  It is configured for a FireEye HX appliance but can be easily modified for anything!

 

 

Configuration

The main configuration of the script is in 2 variables.  These variables are t_keys_to_use and b_debug.

t_keys_to_use – Is a LUA key/value table containing which keys to parse.

 

b_debug – By default, this variable is set to true.  This means that no meta will be created.  Instead, output will be provided for debugging to logs.

 

You also need to configure the cefhelper:setKeys() section of the script.  It needs to contain all the same keys from the t_keys_to_use table.  For example:

cefhelper:setKeys({
   nwlanguagekey.create("vx.threatscore"),
   nwlanguagekey.create("vx.detection"),
   nwlanguagekey.create("virusname"),
   ...
   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("vx.filedesc")
})



You can access logs on a Log Decoder via the REST API:

http://de.co.der.ip:50102/logs?msg=pull&force-content-type=text/plain&expiry=600&count=50 

 

Example - Accellion File Transfer (FTA)

 

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

CEF Extension

Description

System Parser

NetWitness Key Name

msg

Message

Yes

msg

src

Source

Yes

src

deviceFacility

Device Facility

Yes

deviceFacility

fname

File Name

No

filename

fsize

File Size

No

n/a

location

Location

No

directory

type

Transfer Type

No

action

seconds

Transfer Time - # Seconds

No

n/a

suser

Source User

Yes

username

 

Helper Configuration:

local t_keys_to_use = {
        ["fname"] = "filename",
        ["location"] = "directory",

        ["type"] = "action"
}

cefhelper:setKeys({
   nwlanguagekey.create("filename"),
   nwlanguagekey.create("directory"),
   nwlanguagekey.create("action")
})

local b_debug = false

 

Example CEF Message:

CEF:0|Accellion|FTA|FTA_9_12_80|203|Download|1|msg=(172.16.20.45) test.txt downloaded by ian.redden@rsa.local (23 bytes, 3.5 sec) (Type: ssl_download, Location: ) src=172.16.20.45 deviceFacility=local5 fname=test.txt fsize=23 cs2Label=location cs2= cs1Label=type cs1=ssl_download cn1Label=seconds cn1=3.5 suser=ian.redden@rsa.local

 

Example - Payload Security's VxStream

 

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

CEF Extension

Description

System Parser

NetWitness Key Name

Threat Score

Threat Score

No

vx.threatscore

AV Detection Rate

Detection Rate

No

vx.detection

Malware Family

Malware Family Name

No

virusname

fileHash

File Hash Value

No

vx.filehash

fname

File Name

No

vx.fname

fsize

File Size

No

vx.fsize

fileType

File Type

No

vx.filetype

flexString1

Comments

No

vx.comments

Client

Client

No

vx.client

Indicators

Source User

No

vx.indicators

Contacted Domains

Contacted Domains

No

alias.host

Contacted Hosts

Contacted Hosts

No

alias.host

Compromised Hosts

Compromised Hosts

No

alias.host

File Description

File Description

No

vx.filedesc

 

Helper Configuration:

local t_keys_to_use = {
   ["Threat Score"] = "vx.threatscore",
   ["AV Detection Rate"] = "vx.detection",
   ["Malware Family"] = "virusname",
   ["fileHash"] = "vx.filehash",
   ["fname"] = "vx.fname",
   ["fsize"] = "vx.fsize",
   ["fileType"] = "vx.filetype",
   ["flexString1"] = "vx.comments",
   ["Client"] = "vx.client",
   ["Indicators"] = "vx.indicators",
   ["Contacted Domains"] = "alias.host",
   ["Contacted Hosts"] = "alias.host",
   ["Compromised Hosts"] = "alias.host",
   ["File Description"] = "vx.filedesc"
}

cefhelper:setKeys({
   nwlanguagekey.create("vx.threatscore"),
   nwlanguagekey.create("vx.detection"),
   nwlanguagekey.create("virusname"),
   nwlanguagekey.create("vx.filehash"),
   nwlanguagekey.create("vx.fname"),
   nwlanguagekey.create("vx.fsize"),
   nwlanguagekey.create("vx.filetype"),
   nwlanguagekey.create("vx.comments"),
   nwlanguagekey.create("vx.client"),
   nwlanguagekey.create("vx.indicators"),
   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("vx.filedesc")
})

local b_debug = false

 

Example CEF Message:

Mar 29 15:42:59 192.168.1.100 CEF:0|Payload Security|VxStream|6.20|Sample Analysis Result - Malicious|Sample Analysis Result - Malicious|100|end=03/29/2017 21:28:55 cn1=100 cn1Label=Threat Score cn2=88 cn2Label=AV Detection Rate cs1=Trojan.Generic cs1Label=Malware Family cs2=100 cs2Label=EnvironmentID cs3=Windows 7 32 bit cs3Label=Environment Description fileHash=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fname=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fsize=357888 fileType=exe request=https://www.hybrid-analysis.com/sample/8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572/?environmentId\=100 msg=Malicious flexString1= flexString1Label=Uploader Comment cs4=zpr5huq4bgmutfnf.tor2web.org \nzpr5huq4bgmutfnf.onion.to \ncrl2.alphassl.com \nipinfo.io \nkosdfnure75.op1gifsd05mllk.com \ngfdkotriam.fo4j4wnq51hepa.com cs4Label=Contacted Domains cs5=84.200.69.80:53 \n34.196.176.140:80 \n185.100.85.150:443 \n192.36.27.5:443 cs5Label=Contacted Hosts cs6=84.200.69.80 \n185.100.85.150 \n192.36.27.5 cs6Label=Compromised Hosts cs8=2812134 \n2015576 \n2812134 \n2020116 \n2020716 \n2016810 \n2016810 cs8Label=ET Alerts cs9=api-12:1:4 \napi-75:1:10 \nsuricata-2:2:10 \nnetwork-21:2:10 \napi-10:1:6 \ntarget-4:2:10 \nnetwork-0:0:1 \nnetwork-2:0:5 \ntarget-25:0:3 \nnetwork-22:1:10 \nregistry-25:1:3 \nstatic-3:1:10 \ntarget-58:2:10 \nstatic-6:1:10 \navtest-3:2:10 \nsuricata-1:1:10 \napi-25:1:7 \nnetwork-27:2:10 \nregistry-27:1:10 \nnetwork-14:2:9 \napi-16:0:1 \nregistry-36:1:8 \nstring-14:1:3 \napi-51:1:5 \nregistry-35:1:5 \napi-37:0:10 \napi-76:1:10 \nnetwork-24:2:6 \nstatic-17:1:10 \nregistry-1:1:10 \nnetwork-5:1:7 \ntarget-14:2:8 \nhandle-0:1:7 \nstatic-0:1:10 \ntarget-3:0:5 \napi-27:1:10 \napi-77:1:10 \navtest-0:2:8 \nnetwork-1:0:1 \napi-55:0:7 \napi-9:1:4 \napi-42:2:7 \nstatic-60:1:10 \napi-39:0:8 \nregistry-61:1:10 \nstring-13:1:7 \nnetwork-15:2:5 \napi-26:1:10 \navtest-5:2:5 \nmutant-0:0:3 \nregistry-55:0:10 \napi-11:0:2 \napi-6:1:4 \nstatic-1:1:1 cs9Label=Indicators priority=9 cs10=PE32 executable (GUI) Intel 80386 (stripped to external PDB) \n for MS Windows cs10Label=File Description

 

Example – FireEye HX

FireEye HX is FireEye’s endpoint protection appliance.  There are a few meta keys that we need to parse from FireEye HX:

 

CEF Extension

Description

NetWitness Key Name

IOC Name

IOC Name / Malware Family

risk.warning

Target OS

Target Operating System

OS

dhost

Destination Hostname

alias.host

dst

Destination IP

ip.dst

dntdom

Destination Domain

ad.domain.dst

 

Helper Configuration:

local t_keys_to_use = {
        ["IOC Name"] = "risk.warning",
        ["Target OS"] = "OS”,
        ["dhost"] = "alias.host",
        ["dst"] = "ip.dst",
        ["dntdom"] = "ad.domain.dst"
}

cefhelper:setKeys({
   nwlanguagekey.create("risk.warning"),
   nwlanguagekey.create("OS"),

   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("ip.dst"),
   nwlanguagekey.create("ad.domain.dst")
})

local b_debug = false

Example CEF Message:

CEF:0|fireeye|hx|3.1.3|IOC Hit Found|IOC Hit Found|10|rt=Sep 29 2016 02:39:54 UTC dvchost=lab.rsa.local categoryDeviceGroup=/IDS categoryDeviceType=Forensic Investigation categoryObject=/Host cs1Label=Host Agent Cert Hash cs1=YUGh0fvlBG5ewBBahhbEZH dst=172.16.10.50 dmac=aa-bb-7a-fa-75-d8 dhost=victim-a3c696c8 dntdom=WORKGROUP deviceCustomDate1Label=Agent Last Audit deviceCustomDate1=Sep 29 2016 02:39:49 UTC cs2Label=FireEye Agent Version cs2=21.33.0 cs5Label=Target GMT Offset cs5=PT0H cs6Label=Target OS cs6=Windows 7 Professional 7601 Service Pack 1 externalId=1049717 start=Sep 29 2016 02:39:28 UTC categoryOutcome=/Success categorySignificance=/Compromise categoryBehavior=/Execute categoryTechnique=Exploit act=Detection IOC Hit msg=Host victim-a3c696c8 IOC compromise alert categoryTupleDescription=A Detection IOC found a compromise indication. cs4Label=IOC Name cs4=SANDSTORM (FAMILY)

 

DISCLAIMER: As always, this script is provided as is.  If you have any questions, feel free to reach out to me at ian.redden@rsa.com.

Happy Hunting!

 

Update 5/26/2017 - Updated CEF template fixing several bugs.

Update 6/14/2017 - Added sample for VxStream

Attachments

Outcomes