Eric Partington

Log - Parsing Cisco Umbrella

Blog Post created by Eric Partington Employee on Jan 12, 2018

Interesting question from an internal resource about how to parse CSV files that contain information pulled from Cisco Umbrella S3 Buckets to a local filestore and how to get them into RSA NetWitness Logs.  As a learning process I have documented the steps that were used to get this working, hopefully you find it useful. 

Thanks to Dave Glover for the help with the parser framework and Nir Oz for the original question.


At high level we will assume that the logs are pulled down by an external script to a location that either has the SFTP agent installed or can be placed in the correct directory on the log collector for the parsing pipeline to take over in NetWitness.


Review the typespec framework as we will create a new one for the file collection method.


Create the typespec file for file collection


<?xml version="1.0" encoding="UTF-8"?>
   <description>FileCollection specification for eventsource type "Cisco Umbrella" using file handler type "cisco_umbrella"</description>


Upload the typespec file to the log collector


set the name as cisco_umbrella.xml

make sure permissions are set right (same as the other files in the directory)


Restart the log collector service


Create Event Source

In the log collector UI Set up collection

Log collector > Config > Event Sources > File > New Event Category

call it cisco_umbrella

Create a new source

File Directory is cisco_umbrella (this is the directory inside the uploads directory where files will be placed for this collection)

Address - this set the device.ip of this collection so set it to the IP of the cloud system potentially so you have a good record in device.ip of where logs 'came from' originally


Review filesystem for uploads directory created for this collection



Start file collection mechanism on log collector


Upload test file

review that the collection mechanism works

place the csv in this directory for collection to take place



Review Logs

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkGrp[1]:64008] [getWork:651] [cisco_umbrella.cisco_umbrella] [idle] Work Unit Given to Work Manager: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1073] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File processed successfully: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1104] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File deleted: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv


View investigator


Default Parsing with no parser


New features added in NW11.0 are now included in the log decoders which are the lua parsers below...these help in the best effort parsing of messages to try to locate useful information from logs and parse them out.



With no parser created pulls out some information from these logs by default (new feature in NW11.0)

ip.addr,, filename, sld and tld.



Create Log Parser

Category for the logs will be Web Logs

Define the header with the LPT1.0 tool (or notepad++)

this is the raw log data as the decoder sees it

%cisco_umbrella-4: "2017-12-13 00:08:01","DC


Define the additional fields to parse out according to the columns in the CSV and the need for the data.


<?xml version="1.0" encoding="UTF-8"?>
                 displayname="Cisco Umbrella:custom"
                 group="Web Logs"

     eventcategory="1612000000"                content="&quot;&lt;event_time_string&gt;&quot;,&quot;&lt;rulename&gt;&quot;,&quot;&lt;saddr&gt;&quot;,&quot;&lt;stransaddr&gt;&quot;,&quot;&lt;daddr&gt;&quot;,&quot;&lt;content_type&gt;&quot;,&quot;&lt;action&gt;&quot;,&quot;&lt;url&gt;&quot;,&quot;&lt;web_referer&gt;&quot;,&quot;&lt;user_agent&gt;&quot;,&quot;&lt;resultcode&gt;&quot;,&quot;&lt;fld1&gt;&quot;,&quot;&lt;fld2&gt;&quot;,&quot;&lt;fld3&gt;&quot;,&quot;&lt;uid&gt;&quot;,&quot;&lt;fld4&gt;&quot;,&quot;&lt;fld5&gt;&quot;,&quot;&lt;fld6&gt;&quot;,&quot;&lt;fld7&gt;&quot;,&quot;&lt;fld8&gt;&quot;,&quot;&lt;fld9&gt;&quot;,&quot;&lt;group&gt;&quot;"/>


Verified in the LPT1.0 tool


save the file as cisco_umbrellamsg.xml

create the cisco_umbrella.ini for his device as well


DeviceGroup=Web Logs


save it in this directory structure for easy upload to the log decoder


place the xml and ini in this directory


zip archive the structure and rename the as cisco_umbrella.envision


Upload the parser

Log Decoder > Config > Parsers


check the filesystem to make sure the permissions are set right on the folder and files (same as the other files in the directory structure



Reload the Parsers

Log Decoder > Explore menu

Decoder > Parsers - right click - select properties

reload - submit

Review the logs to ensure the parser was loaded with no errors

Cat /var/log/messages | grep –i cisco_umbrella

Should show up in the log decoder parsers list when enabled correctly


Upload new file to test collection and new parsing

place in same folder location as before


Review Parsing


parsing looks good!

review the data that was parsed and what is visible on the decoder/concentrator ( keys like referer and url may not be indexed by default depending on what other customizations have been done those may need to be added to table-map.-custom.xml and index-concentrator-custom.xml)