Skip navigation
All Places > Products > RSA NetWitness Platform > Blog > Author: Josh Randall
1 2 Previous Next

RSA NetWitness Platform

22 Posts authored by: Josh Randall Employee

I helped one of my customers implement a use case last year that entailed sending email alerts to specific users when those users logged into legacy applications within their environment.


Creating the alerts for this activity with the ESA was rather trivial - we knew which event source would generate the logs and the meta to trigger against - but sending the alert via email to the specific user that was ID'd in the alert itself added a bit of complexity.


Fortunately, others have had similar-ish requirements in the past and there are guides on the community that cover how to generate custom emails for ESA alerts through the script notification option, such as Custom ESA email template with raw event payload and 000031690 - How to send customized subjects in an RSA Security Analytics ESA alert email.


This meant that all we had to do was map the usernames from the log events to the appropriate email addresses, enrich the events and/or alerts with those email addresses, and then customize the email notification using that information.  Mapping the usernames to email addresses and adding this information to events/alerts could have been accomplished in a couple different ways - either a custom Feed (Live: Create a Custom Feed) or an In-Memory Table (Alerting: Configure In-Memory Table as Enrichment Source) - for this customer the In-Memory Table was the preferred option because it would not create unnecessary meta in their environment.


We added the CSV containing the usernames and email addresses as an enrichment source:


....then added that enrichment to the ESA alert:


With these steps done, we triggered a couple alerts to see exactly what the raw output looked like, specifically how the enrichment data was included.  The easiest way to find raw alert output is within the respond module by clicking into the alert and looking for  the "Raw Alert" pane:


Armed with this information, we were then able to write the script (copy/pasting from the articles linked above and modifying the details) to extract the email address and use that as the "to_addr" for the email script (also attached at the bottom of this post):

#!/usr/bin/env python
from smtplib import SMTP
import datetime
import json
import sys

def dispatch(alert):
    The default dispatch just prints the 'last' alert to /tmp/esa_alert.json. Alert details
    are available in the Python hash passed to this method e.g. alert['id'], alert['severity'],
    alert['module_name'], alert['events'][0], etc.
    These can be used to implement the external integration required.

    with open("/tmp/esa_alert.json", mode='w') as alert_file:
        alert_file.write(json.dumps(alert, indent=True))

def read():
    smtp_server = "<your_mail_relay_server>"
    smtp_port = "25"
    # "smtp_user" and "smtp_pass" are necessary
    # if your SMTP server requires authentication
    # used in "smtp.login()" below
    #smtp_user = "<your_smtp_user_name>"
    #smtp_pass = "<your_smtp_user_password>"
    from_addr = "<your_mail_sending_address>"
    missing_msg = ""
    to_addr = ""  #defined from enrichment table

    # Get data from JSON
    esa_alert = json.loads(open('/tmp/esa_alert.json').read())
    #Extract Variables (Add as required)
        module_name = esa_alert["module_name"]
    except KeyError:
        module_name = "null"
         to_addr = esa_alert["events"][0]["user_emails"][0]["email"]
    except KeyError:
         missing_msg = "ATTN:Unable to retrieve from enrich table"
         to_addr = "<address_to_send_to_when_enrichment_fails>"
        device_host = esa_alert["events"][0]["device_host"]
    except KeyError:
        device_host = "null"
        service_name = esa_alert["events"][0]["service_name"]
    except KeyError:
        host_dst = "null"
        user_dst = esa_alert["events"][0]["user_dst"]
    except KeyError:
        user_dst = "null"
    # Sends Email
    smtp = SMTP()

    date = "%m/%d/%Y %H:%M" ) + " GMT"
    subj = "Login Attempt on " + ( device_host )
    message_text = ("Alert Name: \t\t%s\n" % ( module_name ) +
        " \t\t%s\n" % ( missing_msg ) +
        "Date/Time : \t%s\n" % ( date  )  +
        "Host: \t%s\n" % ( device_host ) +
        "Service: \t%s\n" % ( service_name ) +
        "User: \t%s\n" % ( user_dst )

    msg = "From: %s\nTo: %s\nSubject: %s\nDate: %s\n\n%s\n" % ( from_addr, to_addr, subj, date, message_text )
    # "smtp.login()" is necessary if your
    # SMTP server requires authentication
    smtp.sendmail(from_addr, to_addr, msg)

if __name__ == "__main__":


And the result, after adding the script as a notification option within the ESA alert:



Of course, all of this can and should be modified to include whatever information you might want/need for your use case.

The RSA NetWitness Platform has multiple new enhancements as to how it handles Lists and Feeds in v11.x.  One of the enhancements introduced in the v11.1 release was the ability to use Context Hub Lists as Blacklist and/or Whitelist enrichment sources in ESA alerts.  This feature allows analysts and administrators a much easier path to tuning and updating ESA alerts than was previously available.


In this post, I'll be explaining how you can take that one step further and create ESA alerts that automatically update Context Hub Lists that can in turn be used as blacklist/whitelist enrichment sources in other ESA alerts.  The capabilities you'll use to accomplish this will be the ESA's script notifications, the ESA's Enrichment Sources and the Context Hub's List Data Source.


Your first step is to determine what kind of data you want to put into the Context Hub List.  For my test case I chose source and destination IP addresses.  Your next step is to determine where this List should live so that the Context Hub can access it.  The Context Hub can pull Lists either via HTTP, HTTPS, or from its local file system on the ESA appliance - for my test case I chose the local filesystem.


With that decided, your next step is to create the file that will become the List - the Context Hub looks within the /var/netwitness/contexthub-server/data directory on the ESA, so you'll create a CSV file in this location and add headers to help you (and others) know what data the List contains:


**NOTE** Be sure to make this CSV writeable for all users, e.g.:

# chmod 666 esaList.csv


Next, add this CSV to the CH as a Data Source.  In Admin / Services / Contexthub Server / Config --> Data Sources, choose List:


Select "Local File Store," then give your List a name and description and choose the CSV from the dropdown:


If you created headers in the CSV, select "With Column Headers" and then validate that the Context Hub can see and read your file.  After validation is successful, tell the Context Hub what types of meta are in each column, whether to Append to or Overwrite values in the List when it updates, and also whether to automatically expire (delete) values once they reach a certain age (maximum value here is 30 days):


For my test case, I chose not to map the date_added and source_alert columns from the CSV to any meta keys, because I only want them for my own awareness to know where each value came from (i.e.: what ESA alert) and when it was added.  Also, I chose to Append new values rather than Overwrite, because the Context Hub List has built in functionality that identifies new and unique values within the CSV and adds only those to the List.  Append will also enable the List Value Expiration feature to automatically remove old values.


Once you have selected your options, save your settings to close the wizard.  Before moving on, there are a few additional configuration options to point out which are accessible through the gear icon on the right side of the page.  These settings will allow you to modify the existing meta mapping or add new ones, adjust the Expiration, enable or disable whether the List's values are loaded into cache, and most importantly - the List's update schedule, or Recurrence:


**NOTE** At the time of this writing, the Schedule Recurrence has a bug that causes the Context Hub to ignore any user-defined schedule, which means it will revert to the default setting and only automatically update every 12 hours.


With the Context Hub List created, you can move on to the script and notification template that you will use to auto-update the CSV (both are attached to this blog - you can upload/import them as is, or feel free to modify them however you like for your use cases / environment).  You can refer to the documentation (System Configuration Guide for RSA NetWitness Platform 11.x - Table of Contents) to add notification outputs, servers, and templates.


To test that all of this works and writes what you want to the CSV file (for my test case, IP source and destination values), create an ESA alert that will fire with the data points you want capture, and then add the script notification, server, and template to the alert:


After deploying your alert and generating the traffic (or waiting) for it to fire, verify that your CSV auto-updates with the values from the alert by keeping an eye on the CSV file.  Additionally, you can force your Context Hub List to update by re-opening your List's settings (the gear icon mentioned above), re-saving your existing settings, and then checking its values within the Lists tab:



You'll notice that in my test case, my CSV file has 5 entries in it while my Context Hub List only has 3 - this is a result of the automatic de-duplication mentioned above; the List is only going to be Appending new and unique entries from the CSV.


Next up, add this List as an Enrichment Source to your ESA.  Navigate to Configure / ESA Rules --> Setting tab / Enrichment Sources, and add a new Context Hub source:


In the wizard, select the List you created at the start of this process and the columns that you will want to use within ESA alerts:


With that complete, save and exit the wizard, and then move on to the last step - creating or modifying an ESA alert to use this Context Hub List as a whitelist or blacklist.


Unless your ESA alert requires advanced logic and functionality, you can use the ESA Rule Builder to create the alert.  Within your alert statement, build out the alert logic and add a Meta Whitelist or Meta Blacklist Condition, depending on your use case:


Select the Context Hub List you just added as an Enrichment Source:


Select the column from the Context Hub List that you want to match against within your alert:


Lastly, select the NetWitness meta key that you want to match against it:


You can add additional Statements and additional blacklists or whitelists to your alert as your use case dictates.  Once complete, save and deploy your alert, and then verify that your alerts are firing as expected:


And finally, give yourself a pat on the back.

With all the recent blogs from Christopher Ahearn about creating custom lua parsers, some folks who try their hand at it may find themselves wondering how to easily and efficiently deploy their new, custom parsers across their RSA NetWitness environment.


Manually browsing to each Decoder's Config/Parsers tab to upload there will quickly become frustrating in larger or distributed environments with more than one Decoder.


Manually uploading to a single Decoder and then using the Config/Files tab's Push option would help eliminate the need to upload to every single Decoder, but you would still need to reload each Decoder's parsers.  While this could, of course, be scripted, I believe there is a simpler, easier, and more efficient option available.


Not coincidentally, that option is the title of this blog. We can leverage the Live module within the NetWitness UI to deploy custom parsers across entire environments and automatically reload each Decoder's parsers in the process.  To do this, we will need to create a custom resource bundle that mimics an OOTB Live resource.


First, lets take a look at one of the newer lua parsers from Live to see how it's being packaged.  We'll select one parser and then choose Package --> Create to generate a resource bundle.


In this ZIP's top-level directory, we see a LUAPARSER folder and a resourceBundleInfo.xml file.


Navigating down through the LUAPARSER folder, we eventually come to another ZIP file:


This contains an encrypted lua parser and a token to allow NetWitness Decoders to decrypt it (FYI - you do not need to encrypt your custom lua parsers).


The main takeaway from this is that when we create our custom resource bundle, we now know to create a directory structure like in the above screenshot, and that our custom lua parser will need to be packaged into a ZIP file at the bottom of this directory tree.


Next, lets take a look at the resourceBundleInfo.xml file in the top-level directory of the resource bundle.  This XML is the key to getting Live to properly identify and deploy our custom lua parser.


Everything that we really need to know about this XML is in the <resourceInfo> section.


A common or friendly name for our parser:



The name of the ZIP file at the bottom of the directory tree:



The full path of this ZIP file:



The version number (which can really be anything you want, as long as it's reflected accurately in the filePath):



The resourceType line is the name of the top-level folder in the resource bundle (you shouldn't need to change this):



The typeTitle (which you also shouldn't change):

            <typeTitle>Lua Parser</typeTitle>


And lastly the uuid, which is how Live and the NetWitness platform identify Live resources:



Modifying everything in this file should be pretty straightforward - you'll simply want to modify each line to reflect your parser's information. And for the uuid, we can simply create our own - but don't worry, it doesn't need to be anywhere near as long or complex as a Live resource uuid.


Now that we know what the structure of the resource bundle should look like, and what information the XML needs to contain, we can go ahead and create our own custom resource bundle.


Here's what a completed custom resource bundle looks like, using one of  Chris Ahearn's parsers as an example: What's on your wire: Detect Linux ELF files:





With the custom bundle packaged and ready to go, we can go into Live, select Package --> Deploy, browse to our bundle, and simply step through the process, deploying to as many or as few of our Decoders as we want:





For confirmation, we can broswe to any of our Decoders at Admin --> Services and see our custom parser deployed and enabled in the Config/General tab:


Lastly, for those who might have multiple custom resources they want to deploy at once in a single resource bundle, it's just a matter of adjusting the resourceBundleInfo.xml file to reflect each resource's name, version, path, and making sure each uuid is unique within the resource bundle, e.g.: uuid1, uuid2, uuid3, etc:



You can find a resource bundle template attached to this blog.


Happy customizing, everybody!

The Respond Engine in 11.x contains several useful pivot points and capabilities that allow analysts and responders to quickly navigate from incidents and alerts to the events that interest them.


In this blog post, I'll be discussing how to further enable and improve those pivot options within alert details to provide both more pivot links as well as more easily usable links.


During the incident aggregation process, the scripts that control the alert normalizations create several links (under Related Links) that appear within each alert's Event Details page.


These links allow analysts to copy/paste the URI into a browser and pivot directly to the events/session that caused the alert, or to an investigation query against the target host. 


What we'll we doing here is adding additional links to this Related Links section to allow for more pivot options, as well as adding the protocol and web server components to the existing URI in order to form a complete URL.


The files that we will be customizing for the first step are located on the Node0 (Admin) Server in the "/var/netwitness/respond-server/scripts" directory:

  • normalize_core_alerts.js
  • normalize_ma_alerts.js


(We will not be modifying the normalize_ecat_alerts.js or normalize_wtd_alerts.js scripts because the Related Links for those pivot you outside of the NetWitness UI.)


As always, back up these files before committing any changes and be sure to double-check your changes for any errors.


Within each of these files, there is a exports.normalizeAlert function:


At the end of this function, just above the "return normalized;" statement, you will add the following lines of


//copying additional links created by the utils.js script to the event's related_links
for(var j = 0; j <; j++){

if (normalized.related_links) {[j].related_links =[j].related_links.concat([normalized.related_links]);





So the end of the exports.normalizeAlert function now looks like this:


Once you have done this, you can now move on to the next step in this process.  This step will require modification of 3 files - the two we have already changed plus the utils.js script - all still located in the "/var/netwitness/respond-server/scripts" directory:

  • normalize_core_alerts.js
  • normalize_ma_alerts.js
  • utils.js


Within each of these files search for "url:" to locate the statements that generate the URIs in Related Links.  You will be modifying these URIs into complete URLs by adding "https://<your_UI_IP_or_Hostname>/" to the beginning of the statement.


For example, this: 


...becomes this:


Do this for all of the "url:" statements, except this one in "normalize_core_alerts.js," as this pulls its URI / URL from a function in the script that we are already modifying:


Once you have finished modifying these files and double-checking your work for syntax (or other) errors, restart the Respond Server (systemctl restart rsa-nw-respond-server) and begin reaping your rewards:


I've seen and heard a fair bit of discussion recently about whether it's possible to create custom matchCondition and groupBy fields within the new 11.x Respond Server.  "We have the capability within 10.x," the question goes, "but can we do this in 11.x?"


The answer is "Yes," but the process is slightly different, hence the reason for this blog post.


First, I think it will be useful to lay some groundwork and establish a common understanding of the incident creation process within Respond.


When the Respond Server consumes alerts off the message bus, those original, raw alerts can have many different meta fields.  The Respond Server needs to create a common schema for these alerts so that it knows where and how to store each piece of incoming data.  To do this, the Respond Server relies on a group of scripts to extract, normalize, and group meta.


With this common schema formed, the Respond Server can then begin to aggregate these alerts into incidents.  The initial aggregation process relies on matchCondition values within the Incident Rule.  For example, the OOTB User Behavior incident aggregation rule:


After aggregating incoming alerts based on these matchCondition values, the Respond Server then attempts to group them into separate Incidents (or suppress them) according to the groupBy values:


The common use case that we will be discussing here is in response to the need for aggregation and grouping using non-default options.  For instance, if we want to group incidents according to email subject, or threat description, or any other arbitrary or custom metakey, how to add those so that they appear as options within the UI, AND so that the aggregation and grouping works?


****Before getting into any of the details, I strongly recommend that you try these procedures on a lab or test system first, both to familiarize yourself with the process and to ensure it works, before making any changes to a production system.****


Now then, on to the good stuff.


First thing we need to do is identify the locations and names of the specific files that we will be modifying.  This is one area where the process in 11.x is slightly different compared to 10.x, as these files are in a different location.


To modify the available groupBy and matchCondition fields, we need these two files on the Node0 Server (aka Head Server; aka Admin Server):

  • /var/netwitness/respond-server/scripts/normalize_alerts.js
  • /var/netwitness/respond-server/data/aggregation_rule_schema.json


AND, depending on the source(s) of the alert(s), we will ALSO need to modify (at least) one of the following:

  • /var/netwitness/respond-server/scripts/normalize_core_alerts.js
    • for alert sources:
      • ESA
      • Reporting Engine
      • NetWitness Investigate
  • /var/netwitness/respond-server/scripts/normalize_ecat_alerts.js
    • for alert source NetWitness Endpoint (aka ECAT)
  • /var/netwitness/respond-server/scripts/normalize_ma_alerts.js
    • for alert source Malware Analysis
  • /var/netwitness/respond-server/scripts/normalize_wtd_alerts.js
    • for alert source Web Threat Detection


Once we know the source of the incoming alerts, we will then need to identify the key(s) and/or value(s) within those raw alerts that we want to match and group against.  At this point, you will most likely need to examine the raw alert within the Respond Server.


Browse to Respond --> Alerts and select your specific alert.  The Raw Alert will be visible in the window on the right (or, if you clicked on the hyperlink Alert Name, it will be on the left in the newly-opened window), allowing you to scroll through the raw data and identify the key or value specific to your use case.


For my test case, I generated alerts from the ESA with different "event_type" values:


...which meant I first needed to modify the "/var/netwitness/respond-server/scripts/normalize_core_alerts.js" file and add the "event_type" key.


Within "normalize_core_alerts.js" there is a "generateEventInfo" function, which is where we can define additional keys to be normalized by the Respond Server, and where I added my "event_type" key.

****NOTE: it is VERY important that you pay close attention to the formatting and syntax within this file when you add a new key, especially where trailing commas are needed/not needed.****


Next I modified the "/var/netwitness/respond-server/scripts/normalize_alerts.js" file and added a new line for my "event_type" key to the "normalizeAlert" function.

****Again, it is very important that you pay close attention to the formatting and syntax when you add your keys to this function block.****


Then I modified the "/var/netwitness/respond-server/data/aggregation_rule_schema.json" file and added a new schema for the "event_type" key.

****And yet again, pay very close attention to formatting and syntax when modifying this file, especially where commas are needed/not needed.****


****I recommend saving copies of each of these modified files, as they get over-written during the upgrade process.****


And finally, restart the Respond Server service, either from within the UI:


...or via command line from the Node0 Server:

# systemctl restart rsa-nw-respond-server


Give it a minute or two for the service to fully restart, refresh your browser, and you can now select your custom matchCondition and groupBy keys from the drop down menus:


...and view the fruits of your labor:


**Bonus note for anyone still reading: if you're like me and it bugs you if something is not in alphabetical order, you can adjust where your custom keys appear within the dropdown menus by inserting your custom schema within the "aggregation_rule_schema.json" file in a different location.


In the example shown above, I added my custom schema to the very end of the file, which is why my key appeared at the very bottom of each dropdown menu.


But if I place my custom schema in alphabetical order within the JSON file, it will appear within the dropdown menu in its new location:




Happy customizing, everybody!

While the release of the Unified Data Model (UDM) has given us a unified meta key foundation on which to build moving forward (awesome!), it has also opened an administrative can of worms (not so awesome...).


With these new and/or modified meta keys comes the challenge of combing through your NetWitness architecture to find all the places that the discontinued meta exist, identifying the discontinued keys that you want to change, and then actually changing them. We can’t automate this entire process yet, but we can still automate some to make our lives easier.


One of the primary places that meta keys live within NetWitness is the custom XML file that allows for tuning and adding to the default out-of-the-box meta. In the UI, these files are accessible at Admin (or Administration) → Services → <serviceName> → Config → Files:

Custom XMLs in the UI


And on disk at /etc/netwitness/ng/index-<serviceName>-custom.xml, (Log Decoders have an additional custom XML at /etc/netwitness/ng/envision/etc/table-map-custom.xml):

Custom XMLs in the Filesystem

Custom XMLs in the Filesystem (Table Map)


We could search through and update these files manually for every discontinued meta key...but frankly, that would be an enormous headache and a waste of time, which is why I put together this script to do it instead.


Before running the script, go to the UDM page on RSA Link ( and check out the table of Discontinued Meta ( Copy the contents of this table (with or without the header – the script will omit that line if you do include it) into a text file. No modification of this copied table is necessary – again, the script will take of that for us.

Discontinued Meta - UDM


Any discontinued meta keys from this table that do not have a specific 1-to-1 replacement meta key, such as orig_ip or any of the risk.* keys, will also be omitted when the script runs.


Next, copy this text file and the script to the filesystem of the appliance that you want to run it on (Log/Decoder, Log/Concentrator, Log/Hybrid, Archiver, or Broker), and make the script executable.


The script will require two arguments – the name of the text file that you copied the Discontinued Meta table into, and the name of the custom XML that you want to modify:


# python <> <text_file_with_copied_table.txt> <target_custom_file.xml>


For example:


# python filename.txt index-concentrator-custom.xml

# python filename.txt index-archiver-custom.xml

# python filename.txt table-map-custom.xml


The script will ask whether to perform a dry run replacement or to do it for real. If run as a dry-run, you will get an output of all the discontinued meta keys that were identified within the target custom XML, as well as the new meta key that replaces it in the UDM.


If you do not choose the dry-run option, the script will give you the option to view each discontinued meta key and the corresponding new meta key and accept or deny its replacement, or to simply replace everything without any further prompts.

Script Options

Script Options 2

If the actual replacement(s) are accepted, the script will backup the original custom XML before making any changes.


Once complete, I recommend that you compare the new and original files using your diff tool or utility of choice to verify that everything proceeded without error. And as a reminder, you will need to restart the service for these changes to take effect.


Happy UDM'ing!

If you have some custom alert templates that you've been using in NetWitness 10.6.x, you may find that certain expressions no longer work in 11.x, specifically the "@value_of" function we use to iterate through variables that are stored as arrays.


For reference, the out-of-the-box string arrays in the Event Stream Analysis are:


The statements we use in10.6.x to include these string arrays in our alerts looked like this:


CEF:0|RSA|NetWitness ESA|11.0|${statement}|${moduleName}|${severity}|rt=${time?datetime} id=${id} source=${eventSourceId} <#list events as x> sessionid=${x.sessionid!" "} service=${x.service!" "} hostname=<#if x.alias_host?has_content><@value_of x.alias_host /></#if> </#list>


To achieve the same functionality in 11.x for string arrays such as alias_host, we need to use a different expression in order to tell the freemarker template to iterate through each value in the array.  We also need the new expression to be able to handle null values, for example if there is no alias_host meta within the alert.


<#if x.alias_host?has_content><#list (x.alias_host) as alias_host> hostname=${alias_host!" "} </#list></#if>


It is important to take note of the spaces included within this expression, as these ensure each value of "" is delimited from subsequent "" values.  Otherwise, we would end up with this:


The template as a whole would look like this:



CEF:0|RSA|NetWitness ESA|11.0|${statement}|${moduleName}|${severity}|rt=${time?datetime} id=${id} source=${eventSourceId} <#list events as x> sessionid=${x.sessionid!" "} service=${x.service!" "} <#if x.alias_host?has_content><#list (x.alias_host) as alias_host> hostname=${alias_host!" "} </#list></#if> </#list>



And we can add additional expressions within the template as necessary:


CEF:0|RSA|NetWitness ESA|11.0|${statement}|${moduleName}|${severity}|rt=${time?datetime} id=${id} source=${eventSourceId} <#list events as x> sessionid=${x.sessionid!" "} service=${x.service!" "} <#if x.alias_host?has_content><#list (x.alias_host) as alias_host> hostname=${alias_host!" "} </#list></#if> <#if x.action?has_content><#list (x.action) as action> action=${action!" "} </#list></#if> </#list>

Filter Blog

By date: By tag: