David Waugh

LUA Parser to convert meta separated by commas into single meta values

Discussion created by David Waugh Employee on Jan 25, 2016
Latest reply on Feb 24, 2016 by David Waugh

If you are using the CEF parser, some event sources may return meta in the form of comma separated lists.


For example in the following log message:


Jan 25 08:43:09 rsareNsa CEF: 0|RSA|Security Analytics Malware||Suspicious Event|Detected suspicious network event|2|nextgen=100.0 event.type=NEXTGEN event.id=34954 country.dst.code=US city.dst=Dublin org.dst=Amazon.com payload=31779809 packets=32690 country.dst=Ireland,Internal time=Mon Jan 25 08:37:29 UTC 2016 threat.source=snort rule,netwitness filetype=x86 pe,windows_executable,windows executable latdec.dst=53.3331 eth.src=00:50:56:03:01:C8 ip.proto=6 tcp.flags=24 ip.src= tcp.dstport=80 threat.category=spectrum,suspicious,malware,shellcode-detect,data leakage,attempted-user,informational eth.dst=00:50:56:03:08:2B lifetime=41 did=rsadecoder alert.id=nw05130,nw20045,nw25130,nw30035,nw30040,nw32505,nw32765,nw110060,\rO,.U,3y,4y,cA,jQ,xR,ˆ.,¥L,Ã6,Ä6 sessionid=46822006 medium=1 size=33557045 ad.username.src=Administrator rpackets=97 action=get ad.domain.src=WAUGH eth.src.vendor=VMware, Inc. rpayload=97 content=spectrum.analyze,application/octet-stream,spectrum.consume extension=exe eth.dst.vendor=VMware, Inc. rid=329515862 directory=/offlineupdate/avg_v26141/ risk.suspicious=abnormal exe,escalation multiple informational,packer armadillo,fake antivirus malware indicators eth.type=2048 ip.dst= service=80 filename=Metascan_Offline_Updater_avg_v26141_1453659907.exe streams=2 risk.info=flags_ack,flags_psh,FILE-MULTIMEDIA Microsoft Windows DirectX directshow wav file overflow attempt,INDICATOR-SHELLCODE x86 NOOP,FILE-MULTIMEDIA MultiMedia Jukebox playlist file handling heap overflow attempt,high risk filetypes,http1.1 without referer header,FILE-MULTIMEDIA MultiMedia Soft Components AdjMmsEng.dll PLS file processing buffer overflow attempt,FILE-OTHER Adobe Acrobat EMF conversion heap buffer overflow attempt,outbound_traffic,large outbound data transfer,FILE-OTHER Interactive Data eSignal stack buffer overflow attempt,FILE-MULTIMEDIA CyberLink PowerDVD playlist file handling stack overflow attempt,FILE-MULTIMEDIA RealNetworks RealPlayer mpeg width integer memory under


The values of content, alert.id, risk.info,risk.suspicious threat.source are all comma separated lists.


This makes investigation on these values harder, as you would have to use an expression such as risk.info contains <some value> in order to match this meta across events.


The following LUA parser takes care of this for you as it splits comma separated lists of meta, into individual meta values.


Currently the parser only works on events where the device.type is netwitnessspectrum or rsa_security_analytics_esa.


It also works on the meta keys:

  • filename
  • threat.source
  • threat.category
  • content.type
  • risk.info
  • risk.suspicious
  • action
  • alert.id
  • filetype


The parser though can be easily modified to handle other device types or metakeys.